This is incorrect, ammonium nitrate is intrinsically high-explosive. It is mixed with other ingredients to improve performance and/or sensitize it but this is not required. There are several famous examples of stored ammonium nitrate exploding e.g. https://en.wikipedia.org/wiki/2020_Beirut_explosion
Most of those other blasts, including Beirut, involved lots of contamination of the AN + large fires + very large quantities of ammonium nitrate.
In truckload or less quantities, and when not contaminated with specific substances, ammonium nitrate is pretty much impossible to detonate without a sizable quantity of some other high explosive.
If you’re going to assume they have access to significant explosives to actually detonate pure ammonium nitrate, then you also need to consider flour and other ‘safe’ compounds due to the potential for even larger fuel air explosions.
There is occasionally discussions if we should ban trucks that carries dangerous materials from driving near large cities. The main counter argument is that the number of people who has access to trucks with dangerous goods in them is so few compared to the number of people with access to a car. Thus the more logical solution to mitigate risks is to address car access to areas with crowds.