Regarding the Pont du Gard, it's intact "due to the importance of its secondary function, as a toll bridge. For centuries the local lords and bishops were responsible for its upkeep, in exchange for the right to levy tolls on travellers using it to cross the river."

I believe bigiain's point is that the structures needed active upkeep to last thousands of years.

Your point is true about people robbing the masonry, but that's closely related to bigiain's comment about "keeping curious humans away from long half life radioactive trash".