In many cases there is evidence that the palaces and cities were not just abandoned, but they were burned.

So this makes plausible that the abandoning was not something voluntary.