Terrorists are going to break into a government facility, go a mile deep, retrieve massive concrete cylinders, and transport them back to base. Just to get access to low level nuclear material? If they want to poison people en-masse there are way better options. Uranium is a relatively slow killer. And we test our water supply for it because it's a heavy metal like lead. So any real attempt to poison people with it will get detected pretty promptly.

Groups that have the expertise to covertly retrieve spent fuel could probably just buy guns and shoot up a stadium. And that would cause more damage. The relative risk presented by nuclear waste is trivial.