I'd guess the latter. The way half-life works, the stuff that's dangerous to you as you walk around quickly decays and stops being dangerous.

The problem is with long half-life (and thus low-radioactivity) elements that sit on the trees, get mixed with soil, and thus eventually end up in ground water and local food chain. You might be able to walk around a disaster site safely in relatively short time after the event, but I wouldn't try growing anything there.