Because we've been listening, rather half-assedly, for less than 100 years to a thing that is orders of magnitude larger than that (in light years) and haven't heard anything ... there must be nothing there?!

That's pretty much like knocking on the door of a house with all the lights on, and when it isn't answered three milliseconds later, concluding no-one is home.