News for hackers. Stuff that matters.
Re:Fewer Than Half of Google Searches Now Result in a Click by erik_seaberg on Thursday August 15, 2019 @11:08AM (#20705882)
If you have a calendar where every month links to the previous and next months, a crawler can get stuck and hammer the server. That's the kind of thing robots.txt is for.

There are no comments on this comment yet.