Castles that stay above ground, yes, but there likely would be some buried structures, too.

Pompeii looked remarkably fresh when dug out.

Pompeii was literally buried in a matter of hours, so that’s kind of a unique case. Any underground structures would be turned into a cave from water seeping in. Most people don’t dig into the side of caves, so it’s likely we’d never know. A great example is the millennium old water passages in Afghanistan. You can’t tell they are man made any more.

The point stands, most things don’t survive all that long (in the grand scheme of things). Occasionally, the conditions are exactly right to preserve something, but there is a limit.

Structures / cities might also get buried within a few hundred years due to overgrowth.

Sufficient sized meteorite impact would melt the entire surface of the planet.

That did happen — the Moon is theorised to have formed from the ejecta of a planet-sized impactor.

However, that happened very early in the history of the solar system, likely before life had a chance to start.