By "still standing" do you include ruins where, say, only part of a wall remains?
Regarding the Pont du Gard, it's intact "due to the importance of its secondary function, as a toll bridge. For centuries the local lords and bishops were responsible for its upkeep, in exchange for the right to levy tolls on travellers using it to cross the river."
I believe bigiain's point is that the structures needed active upkeep to last thousands of years.
Your point is true about people robbing the masonry, but that's closely related to bigiain's comment about "keeping curious humans away from long half life radioactive trash".
I think the supply line from port to the facility would be a softer target and is something the jury and stakeholders would be considering in serious detail.
Here's some info from the royal commission: http://nuclearrc.sa.gov.au/
And from the Citizens' Jury: http://nuclear.yoursay.sa.gov.au/
As I understand it, being paid to store nuclear waste is being viewed as an income stream for the state to backup or replace income from mining operations.