Re:Update to Kagi Search Pricing
by Atlas22 on Friday March 10, 2023 @05:23PM
(#35102057)
Good to know they are working on this.
Is there a legal issue with spoofing user agent to be the google crawler? Spoofing is certainly enough to get rid of article paywalls for 99% of sites Ive encountered. At least last I heard you can also work around cloudflare captcha by just routing requests through a worker on their service.
Is there a legal issue with spoofing user agent to be the google crawler? Spoofing is certainly enough to get rid of article paywalls for 99% of sites Ive encountered. At least last I heard you can also work around cloudflare captcha by just routing requests through a worker on their service.