News for hackers. Stuff that matters.
Re:I was wrong about robots.txt by EPendragon on Thursday July 17, 2025 @09:47AM (#44593390)
There are tools like curl-impersonate: https://github.com/lwthiker/curl-impersonate out there that allow you to pretend to be any browser you like. Might take a bit of trial and error, but this mechanism could be bypassed with some persistence in identifying what is it that the resource is trying to block.

There are no comments on this comment yet.