News for hackers. Stuff that matters.
Re:Coinbase Merchant Data Leak? by enoptix on Friday April 05, 2013 @09:58AM (#5498420)
You can also specify a meta robots tag inside the page HTML. If you want to block a lot of pages, your best bet would be to add it to your master layout or template. You get the same effect of blocking on robots.txt but without exposing a list of blocked pages.

The downside is that Google will still crawl the page and use your bandwidth, but the page won't be indexed.

There are no comments on this comment yet.