Re:Breaking the 4Chan CAPTCHA
by brookst on Saturday November 30, 2024 @10:35AM
(#42282120)
Wouldn’t a well-intentioned bot follow robots.txt anyway?
[ Parent
| on HN ]
Curl and libcurl doesn't. Wget doesn't, I think.
CURL isn't a bot, it's a tool. It can be part of a bot (which may or may not respect robots.txt) but it can also just act as a user agent directly for a human operator in which case it SHOULD just do what asked. Chrome doesn't follow robots.txt either for the same reason.