Obviously similar arguments apply to our lack of real knowledge of the side-effects of pharmaceuticals and vaccination programs even after substantial testing, but there's an argument that the rewards for developing cures for disease are worth risking the side effects in a way that increased crop yields aren't. Moreover drugs tend to directly affect the individuals taking them, whereas 'enhanced' crops and their non-sterile wind-blown pollen affect an entire ecosystem.
Then again the alternatives seem to be less-than-risk-free pesticides anyway...
Folks cross-breed, look for, and induce mutants (via radiation and other means) for exactly the same purposes as they use gene-splicing.
It would take longer to get round-up resist corn via cross-breeding, but folks would be trying to do it if they didn't have gene techniques.
I'll do my best. Is "GM Bananas or starving children" a false dichotomy? If the world had no bananas, couldn't we use the plantations to grow something else?
There's no shortage of food or food-bearing land in the world today. The remaining problems are purely political problems of distribution and corruption and economic problems of perverted incentives (food aid crowding out regional and local farming).
You would think that "this GM crop will allow us to feed millions of otherwise starving children" would be a good enough argument for most people.