Re:I was wrong about robots.txt
by EPendragon on Thursday July 17, 2025 @09:47AM
(#44593390)
There are tools like curl-impersonate: https://github.com/lwthiker/curl-impersonate out there that allow you to pretend to be any browser you like. Might take a bit of trial and error, but this mechanism could be bypassed with some persistence in identifying what is it that the resource is trying to block.
[ Parent
| on HN ]
There are no comments on this comment yet.