I think water loss (especially from aquifers but also just drought and river diversion in California), soil erosion, habitat loss, and homogenisation are big worries. We’ve seen big problems from soil loss before in America where the prairie grass with its deep root systems holding the soil together was replaced by cereal crops with short roots and plowing exposing the soil. The result was the dust bowl.
The path you say has "no cause for panic" is a path to cyclical famine in 50-100 years.
https://en.wikipedia.org/wiki/Haber_process
The nitrogen is easily separated from air. The hydrogen is the energetically expensive input. Most hydrogen is presently made by steam reforming fossil fuels. Hydrogen can also be produced from water and electricity via electrolysis:
https://en.wikipedia.org/wiki/Electrolysis_of_water
In the 20th century, surplus hydroelectric power from dams was used to electrolyze water for making ammonia. More direct demand for electricity and cheaper processes for making hydrogen from fossil fuels gradually eliminated it. Now people are planning to bring back large scale electrolytic hydrogen for fossil-free hydrogen from renewables:
https://www.cnbc.com/2021/01/19/canada-is-set-to-have-one-th...
https://www.chemengonline.com/uniper-unveils-plans-for-germa...
And there are a few plants in operation that produce without any fossil inputs, but most of them aren't switched over yet, mostly because green hydrogen production isn't at the necessary scale yet and will probably be used for steel making first.
Rapeseed oil maybe isn't a perfect solution. But whining about everything that makes headway just because it isn't perfect yet also holds us back.
The most important oil consumption is for driving farm implements, but that can easily be changed to rapeseed oil in the short term and other green energy in the long term.
So no cause for panic.