Yes. But there's a strong enough correlation for it to be beneficial. Most processed foods are processed to add sugars, fats, and sodium to make them more tasty, and to remove fiber to make them easier to eat.

You're right that a person shouldn't pre-judge a good based solely on its having been manufactured. But it's also true that someone who generally avoids processed foods will end up avoiding the most unhealthy foods available.