News for hackers. Stuff that matters.
Re:Why are anime catgirls blocking my access to the Linux ke... by kldg on Wednesday August 20, 2025 @11:29PM (#44968777)
does it work well? I run chromium controlled by playwright for scraping and typically make Gemini implement the script for it because it's not worth my time otherwise. -but I'm not crawling the Internet generally (which I think there is very little financial incentive to do; it's a very expensive process even ignoring Anubis et al); it's always that I want something specific and am sufficiently annoyed by lack of API.

regarding authentication mentioned elsewhere, passing cookies is no big deal.

Re:Why are anime catgirls blocking my access to the Linux ke... by eaglefield on Thursday August 21, 2025 @05:36AM (#44970786)
Anubis is not meant to stop single endpoints from scraping. It's meant to make it harder for massive AI scrapers. The problematic ones evade rate limiting by using many different ip addresses, and make scraping cheaper on themselves by running headless. Anubis is specifically built to make that kind of scraping harder as i understand it.