News for hackers. Stuff that matters.
Re:Panic at the Job Market by Etheryte on Wednesday July 17, 2024 @12:23PM (#40987540)
Nearly all browsers, scrapers, etc use the same user agent these days. Tools such as curl and wget are the only ones that come to mind off the top of the head that don't do that out of the box.
Re:Panic at the Job Market by nojs on Wednesday July 17, 2024 @12:36PM (#40987680)
Re:Panic at the Job Market by mananaysiempre on Wednesday July 17, 2024 @01:49PM (#40988448)
There was a discussion some days ago about one of the AI companies using a very characteristic user agent string for web crawling, but a more browser-like one for web browsing performed at the behest of the user. And there were some pertinent points there—if the AI bot is acting on an explicit request of a user, it does deserve to get treated like any other user agent more or less.
Re:Panic at the Job Market by PeterisP on Thursday July 18, 2024 @05:11AM (#40993830)
It also would make some sense for various real-time-request-augmented bots to not only use the user agent string of the user's browser, but actually use the user's browser to make the request.
Re:Panic at the Job Market by ceejayoz on Wednesday July 17, 2024 @12:59PM (#40987899)
AI scrapers are pretty widely ignoring robots.txt, and plenty lie about their user agents. https://rknight.me/blog/perplexity-ai-is-lying-about-its-use...

I'd fully expect OpenAI to do some checks that their bot isn't getting different responses than a seemingly real request.

Re:Panic at the Job Market by odo1242 on Wednesday July 17, 2024 @01:31PM (#40988232)
OpenAI managed to add this after a lot of complaining, but most AI scrapers lie about their user agent and ignore robots.txt. Plus, OpenAI gets to keep all the data from before they added this.
Re:Panic at the Job Market by tonetegeatinst on Wednesday July 17, 2024 @01:29PM (#40988199)
Yeah this does work as long as the scraper respects robot.text

But dosnt openai and other companies use third party datasets? Like sure they do plenty of scraping but I'd bet for some stuff its cheaper to buy the dataset and then cleanup the data.

Re:Panic at the Job Market by twelve40 on Thursday July 18, 2024 @12:13AM (#40992453)
This is great, but what about the Common Crawl data they've used (or still get?), data that Bing might share with them, and other ways they acquire data past present and future? All of a sudden not so nicely labelled as GPTBot, is it?