History, yes. The actual obsolete building housing humans, no.

So is every building history, then? Are we not allowed to knock down any buildings, since they are all part of history?

You don't need to preserve everything to remember history. If we never destroy old buildings, we'd have to abandon all our cities and find new places to build new cities, because we can't just keep living like it's the middle ages (or whatever time period you feel we need to preserve).