The point of that argument is never "This is perfect." It's exactly as you say - "This is not bad enough to cause serious harm and therefore" (they're saying) "not bad enough to get worked up over it."

"It survived evolution" = "It won't kill you" = "It's good enough"

They're setting the bar a bit lower than optimum.