The aspect to me, that doesn't involve the traditional alien race destroys humanity, is an idea I heard in an astrophysics podcast. It is still pretty terrifying although I guess more sad.

An advanced aliens visited the earth hundreds, thousands, or even millions of years ago, extracted all of the rare compound that allows for their ability to have such an advanced society capable of traveling such distances and just left. We are purposefully being ignored since it is known to the galaxy/universe that we have no way of effectively reaching or communicating with other advanced civilizations.

All of the useful material was extracted so no advanced alien civilizations even have a reason to come and visit.