That's not quite right. Yes natural uranium and its daughters are a bit radioactive (this is partially why the center of the Earth is warm). But when you go ahead and fission nuclear fuel, it breaks into smaller atoms that are less stable. This means they give off their excess energy faster, or in other words are more radioactive. The higher rate of energy release can lead to higher amounts of biological damage.

You can hold fresh nuclear fuel in your hand and be fine. Once it goes in the reactor to be fissioned, you'd get a fatal radiation dose in seconds if you stood next to it unshielded.