Re:More than 1MM Facebook accounts exposed
by wesley on Friday November 02, 2012 @05:13AM
(#4731827)
Wouldn't a proper robots.txt rule fix this?
[ Parent
| on HN ]
Re:More than 1MM Facebook accounts exposed
by mkjones on Friday November 02, 2012 @05:17AM
(#4731837)
A robots.txt file disallowing crawling on the sites that display the contents of user email would help fix this.
However, as some of the discussion below points out, I don't believe that disallowing crawling of these URLs in our robots.txt would keep them from the index if a search engine finds reference to them elsewhere; I think it simply keeps them from being crawled.