> If it was radiation wouldn't it be enough to destroy us living in such close proximity?

Different kinds of radiation differ by how deep they penetrate matter. Simplifying a bit you can imagine that for every atom the radiation passes, there's a probability that it interacts / gets absorbed.

Radiation that passes thousands of kilometres of rock without interaction will have a rather low probability of interacting with a mere metre (or so) of human.