They explain that part by using the idea of 'traitors'. They are often the minorities of that land(whom they currently hate, and serve as a nice punching bag in the whole process, and in the overall scheme of hate process).

The 'traitor' part is like a constant theme in these stories. Super advanced ancestors, but also very nice, accommodating to everyone. Some savages from the outside invade, traitors help them from inside. Savages win, they are colonised. Traitors benefit. The monuments, symbols and anything to do with the traitors is now demonised and the 'treason' is used as a casus belli for a genocide/ethnic cleansing.