News for hackers. Stuff that matters.
Re:Detecting Chrome headless, the game goes on by floatingatoll on Friday July 19, 2019 @05:38PM (#20482665)
We found at one job that approximate one quarter of well-known search engines blatantly use robots.txt noindex declarations as a list of URLs to index, and one openly mocked us for asking them to stop.

Voluntary honor systems don’t work, because there’s no way to compel non-compliers to stop other than standard “anti-attacker arms race” approaches, such as the obstacle described at the head of this thread.

Re:Detecting Chrome headless, the game goes on by jdc on Friday July 19, 2019 @07:08PM (#20483173)
It sounds like scraping is a big problem for you guys. What kind of outfit is it, if you don't mind me asking?
Re:Detecting Chrome headless, the game goes on by floatingatoll on Friday July 19, 2019 @07:46PM (#20483326)
Drop me an email and I’m happy to describe further.