Please make this tool “opt-in” by default

Re:Please make this tool “opt-in” by default by rolph on Sunday April 23, 2023 @07:21PM (#35681750)

the tool also has settings that ignore opt out.

>> Websites can pass the http headers X-Robots-Tag: noai, X-Robots-Tag: noindex , X-Robots-Tag: noimageai and X-Robots-Tag: noimageindex By default img2dataset will ignore images with such headers.<<

>>To disable this behavior and download all images, you may pass --disallowed_header_directives '[]'<<

[ Parent | on HN ]

Re:Please make this tool “opt-in” by default by version_five on Sunday April 23, 2023 @07:34PM (#35681857)

Fair enough, though it's open source right, so if it didn't have the setting it could be trivially changed anyway. In either case, it's the user ignoring the expressed will of the site as opposed to the maintainers of the tool

[ Parent | on HN ]

Re:Please make this tool “opt-in” by default by gus_massa on Sunday April 23, 2023 @10:01PM (#35682885)

There are two solutions

1) Keep a list of all scrapping tools, versions, user-agent string, and which headers opt-out of each one and how can user misconfigure them to avoid the opt-out.

2) Add a general rate limit per ip, Bonus points for a te4mporal ban in case they don't understand the polite version.

[ Parent | on HN ]