News for hackers. Stuff that matters.
Re:Nike.com's robots.txt by wizzwizz4 on Friday February 11, 2022 @08:41AM (#30301339)
Why does marginalia.nu use Cloudflare? Have you been hit by DDOSs? I can't think how a search engine would benefit from Cloudflare caching things.
Re:Nike.com's robots.txt by marginalia_nu on Friday February 11, 2022 @08:43AM (#30301363)
Yeah, It's 100% about bot mitigation. I got hit pretty bad by some botnet a while back, saw 10-15 searches/second. Can't have nice things on the Internet I guess.

Don't really understand their motive either. Maybe they thought I was cloud hosted or used some expensive API to do searches and were attempting to rack up big bills or something.

Re:Nike.com's robots.txt by wizzwizz4 on Friday February 11, 2022 @08:47AM (#30301415)
:-|

I wish I had a good solution to this. Cloudflare to mitigate DDOS attacks has somewhat of a “baby with the bathwater” vibe (considering how much of a pain it is if you can't pass the CAPTCHA, or if you're on Tor), but I can't think of an alternative.

Re:Nike.com's robots.txt by marginalia_nu on Friday February 11, 2022 @08:50AM (#30301439)
:-| indeed.

I've considered having a naked endpoint with a rate limiter that, when it hits some ceiling (dunno, sustained load of 2 RPS or so), offers the alternatives of going through an unlimited cloudflared domain, or waiting until the bots give up. But that might be annoying too.

Re:Nike.com's robots.txt by wizzwizz4 on Friday February 11, 2022 @09:08AM (#30301631)
You could try to push forward the frontier of accessible CAPTCHAs. You just need some tasks that basically all humans can do, but barely and bots can, that don't rely on vision, hearing, emotional response or cultural knowledge. Should be easy, right?