Re:Please make this tool “opt-in” by default
by gus_massa on Sunday April 23, 2023 @10:01PM
(#35682885)
There are two solutions
1) Keep a list of all scrapping tools, versions, user-agent string, and which headers opt-out of each one and how can user misconfigure them to avoid the opt-out.
2) Add a general rate limit per ip, Bonus points for a te4mporal ban in case they don't understand the polite version.
1) Keep a list of all scrapping tools, versions, user-agent string, and which headers opt-out of each one and how can user misconfigure them to avoid the opt-out.
2) Add a general rate limit per ip, Bonus points for a te4mporal ban in case they don't understand the polite version.