You're not at all wrong, I grew up with a lot of illegal friends. Good number of them have since gone on to become American citizens through hard work and determination and have built good lives for themselves.
Maternally, I'm a second generation American. My grandparents came to America post-WWII with nothing but an offer of employment, and ended up retired with the classic condo beach life. The classic story of rags to middle class American riches. There's a reason that's been the American tale, coming to a faraway land in search of prosperity takes drive. We filter for hard workers.
The problem is the black-market / underbelly aspects of this pool, and that very desperation and drive that drove them here has them put up with conditions we should be ashamed of be it in California, Iowa, or Appalachia. Businesses exploiting that cheap source of labor drive down the ability for those trying to live a basic American life.
Though I would add that doing it via a more realistic temporary work visa program would be preferable to encouraging an illegal crossing industry to flourish.