It all boils down to how you define “fit” and “best adapted to a specific environment”.
Some would argue that being poor means you are unfit and unable to adapt to life in a capitalist environment and therefore should not survive or reproduce. Of course, those would be the ones who believe that Darwinism applies to social aspects rather than purely to biology.
Others will debate ad nauseum on points such as whether being able to make shelter and fire is a valid adaptation or whether we should just say, “Man was not meant to live in cold climates because we have no fur or blubber.” and leave it at that. And that is why we really need to figure out where to draw the line as to how far Darwinism truly applies. How can we discuss the ethics of something if we can’t even agree on what it is?
We also need to settle on what “Darwinism” is. To my mind, “Pure” Darwinism lacks (or rather, transcends) ethics because facts are simply truths without regard to social constraints. It’s akin to pondering the moral implications of π = 3.14159…. While others may sling Darwin’s name around to try and legitimize themselves, those are actually just philosophies with some foundation (however tenuous) in Darwin’s theories on genetics. The ethics of the followers of such philosophies are diverse, but also beyond the scope of the question as-asked since, in my opinion, they aren’t “pure” Darwinists.