If it reduced the plants' ability to reproduce, surely that would not spread in a population. So that seems no different from any other deleterious mutation. Such mutations happen all the time in any population.
Your argument there does apply to non-Mendelian inheritance ("gene drive"), so banning modifications to crops that would be engineered to cause that seems prudent (although I'm not sure why anyone would want to do that). Modifications to pests, on the other hand...
Not saying that's the concern just my impression.