The Pantheon is a temple dedicated to all the ancient Roman gods. It cannot be both that and the Church of St Mary and the Martyrs. When the Church took it over, it ceased to be the Pantheon.
There's no doubt the building is the same, and we can argue about how 'true to form' it is. But that's the point. When it stopped being what it originally was, its 'form' is now a goal to never deviate from.
Whereas, the various depictions of the martyrs present within the Church of St Mary and the Martyrs, for example, has no such ideal yet. The Church can (and does) shift that around. However, if it were to stop being the church of st Mary and the Martyrs, then it would have an ideal form that any restorer may want to restore it towards.
It's difficult, if not impossible, for people to see architectural spaces in the way an animal or alien might see them. Even when the culture which built it is totally alien or unknown, that basic fact, plus the context in which it is seen by the observer (including how they arrived there), as well as typical architectural features (e.g. steps) imbue it with meaning that color everything about it. Our whole world is like that. Take, for example, a stop sign--we simply can't perceive that object, at least without effort, without the culturally imbued meaning. It's not just a red & white octagon with shapes in the middle; the meaning vastly overshadows its raw physical properties.
This is demonstrably wrong, as we have many examples of old buildings and we know what people actually feel about them.
Consider two very famous examples of buildings that completely changed their purpose, but are still the same building: the Pantheon and the Hagia Sophia in Istanbul. The Pantheon was a Roman temple to all of the Gods when it was built in 126, and became a Christian church ~400 years later, but everyone recognizes it as the same building. The Hagia Sophia was a Christian church built in 537, and 1100 years later became an Islamic mosque, but it is still the same building to everyone.
Then for the opposite example: St Peter's Basillica at the Vatican. This was built in 1506 on the site of a much older church, also called St Peter's Basillica (commissioned by emperor Constantine), which also served as a Christian church, was also used the parish of the Pope, was also the location of papal coronation and so on. And still, no one would say "St Peter's Basilica was built by Saint Constantine", because the current basilica is universally recognized as a new building, despite serving the same function.
A building is recognized for the beauty and specifics of its structural and painterly elements, not for its role. If it is destroyed and rebuilt in almost the same size and shape, then it is recognized as the same building; regardless of entirely changing its purpose.
Conversely, if a building is rebuilt into a completely different shape, even a universally admired as beautiful one, it is considered a different building, even if keeps serving the same purpose.