{SWAT, pizza orders, etc} assume that the phone number that shows up on caller ID is authentication of the identity of the phone line on the other end. They could call back the number on caller ID to verify the original caller matched the person who picked up, but they don't.

Having knowledge of a Social Security number was assumed to be authentication, but it's increasingly obvious that such an authentication scheme is antiquated and was destined to fail from the beginning. When an identity thief can get a mortgage under my name with little more than credit bureau data on me, it costs only a little more than $15 to destroy my credit, my time, and my future because transactions don't have sufficient authentication.

These awfully designed authentication schemes will only magnify the problems as more companies (especially credit bureaus and data marketers) pass around data on me and make it easier for someone to buy it on demand.

No the people that designed and implemented Social Security knew it was not secure for identification purposes, the first few decades of the program even had "Not to be used for Identification" on the card.

Then the government, and financial industry got lazy and said "well since the majority of people already have these numbers assigned to them lets just use them for Identification as well" and made it a defacto National ID. Something it was never designed for, nor secure enough to be,

I'm pretty much a hair away from blocking all calls without caller ID at my house so I can reliably lock out the remaining spam callers.

This is getting to the point of paranoia at this point. You're already at significant risk if you ever say anything that gets attention by virtue of living in a society. But it comes with benefits, too...