News for hackers. Stuff that matters.
Re:Ask HN: Why is Google purging all search results for Syri... by diziet on Saturday September 14, 2013 @08:03PM (#6387612)
Not having robots.txt just results in Google defaulting to their reasonable crawl settings. For most sites, a hastily configured robots.txt usually results in problems rather than SEO help.

The indexed pages are also missing most of the 'important' landing pages and other big pages, so most likely some sort of automated spam detection was triggered.

My ISP seems to time out on the domain, sshing and curling from various servers returns the page.

Re:Ask HN: Why is Google purging all search results for Syri... by lcedp on Sunday September 15, 2013 @05:17AM (#6388541)
Just saying, the absence of `robots.txt` can be sign of not really caring about SEO.