Re:Fighting cryptojacking and doing good things with content...
by viraptor on Wednesday March 31, 2021 @10:58PM
(#26656147)
That will depend massively on your site structure. (With enough pages Bingbot will do a magnitude more requests than other crawlers) Overall "lower % than your tooling tells you" - for example cookiebot runs crawlers with both proper user agents and "totally a normal Chrome user" from another network. (Not sure if misconfigured or on purpose)
[ Parent
| on HN ]
Re:Fighting cryptojacking and doing good things with content...
by Semaphor on Thursday April 01, 2021 @03:26AM
(#26657547)
When doing some stuff with automated Google Admanager Ads (that required frequent updates), we realized that google is checking every link. At least once per update. Usually more often. With no proper user agent identifying it as a bot. Our solution was to redirect all links through our server and completely block every known google IP-range during the redirect.
Re:Fighting cryptojacking and doing good things with content...
by chrisweekly on Friday April 02, 2021 @10:46PM
(#26678038)
I meant net-wide, not for their particular endpoints.