Please make this tool “opt-in” by default

Re:Please make this tool “opt-in” by default by wmf on Sunday April 23, 2023 @06:39PM (#35681410)

This is a scraping tool that can download a list of URLs very fast. It's not an AI tool per se but the advertised use is to download corpora for AI training. Traditionally, polite Web spiders tried not to hit any one server too hard but when your goal is to download a bunch of data from a small number of sites fast you're inevitably going to be putting real load on each site. I'm not sure there's a solution that can satisfy everyone.

[ Parent | on HN ]

Re:Please make this tool “opt-in” by default by rolph on Sunday April 23, 2023 @06:55PM (#35681549)

ask first, or respect robots.txt no semantic malarchy

[ Parent | on HN ]

Re:Please make this tool “opt-in” by default by gus_massa on Sunday April 23, 2023 @06:59PM (#35681583)

I guess the "solution" is that sites detect abuse and ban the ip for 24 hours. When enough users of this tool get banned, they will fill the github issues tracker with complain and the maintainers will have to learn how to write a polite spider.

We have the inverse problem in the university. We are all (many thousands) behind a proxy, so some popular sites detect the avalanche of request and add a captcha or other anti abuse tool to everyone.

[ Parent | on HN ]

Re:Please make this tool “opt-in” by default by rolph on Sunday April 23, 2023 @07:29PM (#35681817)

it seems like what would be reqired is to whitelist, known acceptable user agents, then review requests from green agents and whiten them based on X criteria, otherwise leave green agents green and put them in a hole.

[ Parent | on HN ]

Re:Please make this tool “opt-in” by default by version_five on Sunday April 23, 2023 @07:05PM (#35681624)

The tool has instructions for how to opt out. They could also ban the user agent. This is just someone venting about something that's currently popular.

[ Parent | on HN ]

Re:Please make this tool “opt-in” by default by rolph on Sunday April 23, 2023 @07:21PM (#35681750)

the tool also has settings that ignore opt out.

>> Websites can pass the http headers X-Robots-Tag: noai, X-Robots-Tag: noindex , X-Robots-Tag: noimageai and X-Robots-Tag: noimageindex By default img2dataset will ignore images with such headers.<<

>>To disable this behavior and download all images, you may pass --disallowed_header_directives '[]'<<

[ Parent | on HN ]

Re:Please make this tool “opt-in” by default by version_five on Sunday April 23, 2023 @07:34PM (#35681857)

Fair enough, though it's open source right, so if it didn't have the setting it could be trivially changed anyway. In either case, it's the user ignoring the expressed will of the site as opposed to the maintainers of the tool

[ Parent | on HN ]

Re:Please make this tool “opt-in” by default by gus_massa on Sunday April 23, 2023 @10:01PM (#35682885)

There are two solutions

1) Keep a list of all scrapping tools, versions, user-agent string, and which headers opt-out of each one and how can user misconfigure them to avoid the opt-out.

2) Add a general rate limit per ip, Bonus points for a te4mporal ban in case they don't understand the polite version.

[ Parent | on HN ]

Re:Please make this tool “opt-in” by default by adultSwim on Monday April 24, 2023 @11:48AM (#35689149)

The tool came up with it's own solution to an already solved problem. It should respect robots.txt

[ Parent | on HN ]

Re:Please make this tool “opt-in” by default by yarg on Monday April 24, 2023 @04:50AM (#35685149)

I once accidentally got my company's office blocked from w3.org automatically downloading the schemas for some XML documents.

What then happens is all requests to w3.org then redirect to a page explaining how anyone automatically downloading those schemas from those URLs became part of an accidental DDOS attack.

[ Parent | on HN ]