Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on How I Ruined My SEO. Parent comment
Likely not the cause but not having a robots.txt can cause weird crawling problems.
Since it takes two minutes to set up, why not. And, here's a tip, don't try and be clever, keep it simple:
User-agent: \* Disallow:
There are no comments on this comment yet.
Likely not the cause but not having a robots.txt can cause weird crawling problems.
Since it takes two minutes to set up, why not. And, here's a tip, don't try and be clever, keep it simple: