Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Google and Facebook are monopolies. Does it matter? [audio]. Parent comment
Most, if not all, websites will block your bot after a couple of request even if it respects robots.txt.
Also they seem to have different rules for GoogleBot and different for yours.
There are no comments on this comment yet.
Most, if not all, websites will block your bot after a couple of request even if it respects robots.txt.
Also they seem to have different rules for GoogleBot and different for yours.