There is such a proper mechanism: robots.txt. Whenever I put all my users' credit card numbers on a public web server I make sure the URL is properly protected by robots.txt. If I want to be absolutely sure everything is safe I use CSS to make the numbers invisible on top of that! ;-)

Why on earth would you let credit card numbers be publicly accessible; furthermore, one can simply view source to see those numbers. Security through obscurity is not a very good approach - you should read up on PCI compliance by starting here: http://en.wikipedia.org/wiki/Payment_Card_Industry_Data_Secu....

Obscurity? Pffft. Simplicity! Nothing beats an HTML page right there in htdocs. You can store anything there and it's totally NoSQL as well. Anything else would be premature optimization. Do the simplest thing that could possibly work! Agile! I've got it all covered by unit tests so there can't be a problem. And if there's a breach I'll just rewrite it all in Haskell. Who are you anyway to tell a seasoned professional how to deal with credit card numbers securely? :)

Google shouldn't bear responsibility for indexing something sensitive, as the sensitive information should not be reachable via any link anywhere, let alone on the public Internet. It's about as sane as expecting the government to clean up after one's own mistakes, and just as damaging to the concept of personal or corporate responsibility.