Yes, sure you could worry about all of those things too, and in some cases you should. It depends what/who you're guarding against.

Your attacker could hire a private army and drive tanks into the datacentre and steal your data. But that's unlikely to happen.

Security is about layers, and I think as a security agency, the NSA did a pretty good job. Choose how many layers you want, and which are appropriate.