What I don't get is why the focus on fertilizers? There may be some wisdom on not spraying random poisons on the stuff you are going to eat¹, but the worst you can get with fertilizers is soil salinization, and this is perfectly avoidable on any place that isn't a desert already, and there are ways to avoid it on deserts too, just a bit more expensive.
1 - So there's a reason for a poison by poison discussion, the overall label of "organic" just poisons the discussion.
How do the crops you mention “gather phosphorous”? From where?
If you learn about the soil food web you will learn that most things we dump on farmland are made available to plants by bacteria and fungi. Unfortunately, those are the things that we kill by farming the way we do.