It’s often because of political influence campaigns that history is actually destroyed, and in such periods there is no appetite for objective historical research. If you are a Muslim ruler trying to convert your subjects and the Sun Worshippers are leading a resistance then destroying their 700 year old temple that reinforces their identity is a no brainer.

This type of thing is widespread throughout the world [1] so there’s no reason to doubt it has a much longer history than records can follow.

[1] https://en.m.wikipedia.org/wiki/List_of_destroyed_heritage