Industrial fertilizer can (and is) used to make bombs. Sometimes accidentally, see Beruit.

Since it's so dangerous and rife for abuse, we shouldn't allow anyone to use it despite the obvious benefits, right?

This is fear mongering and nothing more.