I think the worry is that you get the biased against reproduction genes that make a lot of sense for a for profit GMO plant that hop to normal plants and reduce their ability to reproduce.
Not saying that's the concern just my impression.
If it reduced the plants' ability to reproduce, surely that would not spread in a population. So that seems no different from any other deleterious mutation. Such mutations happen all the time in any population.
Your argument there does apply to non-Mendelian inheritance ("gene drive"), so banning modifications to crops that would be engineered to cause that seems prudent (although I'm not sure why anyone would want to do that). Modifications to pests, on the other hand...
So, the scenario you seem to be thinking of is that GMOs will drive down the price of agricultural commodities, so that farmers who don't avail themselves of them will be unable to compete. But this argument "proves too much": it's an argument not against GMOs, but against ANY agricultural technology that boosts yields and reduces costs. Presumably the Green Revolution is not something we should object to because food prices declined.
I think you are on to something in understanding the anti-GMO motivation in Europe, though. Farmers don't want to have to compete against the more efficient large scale farmers in the US. Anti-GMOism is a kind of camouflaged protectionism.