You're not assuming that it is, you're assuming that if it can be, it must be.

Which assumes that every part of the milky way is equally amenable to colonization, as if the galaxy were merely points laid out on a Cartesian grid, and that any species capable of colonizing one star must continue to colonize them all.

You must consider the case of technologically advanced species incapable of colonizing other stars, and technologically advanced species capable of colonizing other stars, but unwilling to do so on a large scale, or attempts at large scale colonization which fail across a scale of millions of years or even attempts to hide the obvious presence of such colonization from observers.

>Presume that we are not special at all and there exist more technological advanced civilizations that predate us by at least a couple million years

Ok.

>Presume such a large scale colonization would be easy to observe

By humans, given that we only recently became aware that extrasolar planets are even a thing, and have only been studying the universe with anything but optical telescopes for less than a century, and don't even know what the signs of "large scale colonization" would look like. No. Why?

>Or you can choose to attack any of the presuppositions, but that's also very hard in my humble oppinion.

Any argument can seem airtight when you refuse to consider more than a single case to be valid.