News for hackers. Stuff that matters.
Re:Panic at the Job Market by ceejayoz on Wednesday July 17, 2024 @12:59PM (#40987899)
AI scrapers are pretty widely ignoring robots.txt, and plenty lie about their user agents. https://rknight.me/blog/perplexity-ai-is-lying-about-its-use...

I'd fully expect OpenAI to do some checks that their bot isn't getting different responses than a seemingly real request.

There are no comments on this comment yet.