Re:Fewer Than Half of Google Searches Now Result in a Click
by erik_seaberg on Thursday August 15, 2019 @11:08AM
(#20705882)
If you have a calendar where every month links to the previous and next months, a crawler can get stuck and hammer the server. That's the kind of thing robots.txt is for.
[ Parent
| on HN ]
There are no comments on this comment yet.