Re:Notes on Writing Web Scrapers
by buffet_overflow on Thursday December 02, 2021 @04:24PM
(#29422831)
This is a nice approach. I generally leave a project specific email in the request headers with a similar short summary of my goals.
[ Parent
| on HN ]
Re:Notes on Writing Web Scrapers
by marginalia_nu on Thursday December 02, 2021 @05:11PM
(#29423476)
Yeah, my User-Agent is "search.marginalia.nu", and my contact information is not hard to find on that site. Misbehaving bots are very annoying and it's incredibly frustrating when you can't get hold of the owner to tell them about it.