Re:Chrome phasing out support for User-Agent
by adwww on Wednesday March 25, 2020 @03:28PM
(#22687693)
> Any legit person building a legit bot would know to fake the agent string.
What, that's totally backwards. Anyone using a bot to do things that might get blocked by publishers fakes the string, legit purposes should really show who / what they are.
Re:Chrome phasing out support for User-Agent
by xiongchiamiov on Wednesday March 25, 2020 @05:47PM
(#22688800)
It actually is encouraging people to have useful user agents. By default most people end up with a user agent that's something like "libcurl version foo.bar.baz", which isn't actually a description of who or what they are; given the prevalence of curl, it really just tells you that it's a program that uses http.
Re:Chrome phasing out support for User-Agent
by jedberg on Wednesday March 25, 2020 @03:32PM
(#22687713)
We only blocked agent strings with "lib" in them. You could change the agent to "WebScraperSupreme.com" and it would have been fine (and in fact some people did do that).
Re:Chrome phasing out support for User-Agent
by NotSammyHagar on Wednesday March 25, 2020 @03:11PM
(#22687534)
Yes perhaps. But it caused problems for regular users like this fellow. I also have tried various 'download via script' for web pages for offline use. I thought I had a problem on my end, I never realized I could have been getting blocked.
Re:Chrome phasing out support for User-Agent
by naasking on Thursday March 26, 2020 @01:13PM
(#22695703)
Hard to argue with the economics of that mitigation though. The abuse:legitimate use ratio is probably pretty high. Getting rid of user agent strings will bring back the scaling problems, as they should probably be addressed directly.
That is exactly what they are doing, and it works really well.
We blocked user agents with lib in them at reddit for a long time.
Any legit person building a legit bot would know to fake the agent string.
The script kiddies would just go away. It drastically reduced bot traffic when we did that. Obviously some of the malicious bot writers know to fake their agent string too, and we had other mitigations for that.
But sometimes the simplest solutions solve the majority of issues.