Re:13ft – A site similar to 12ft.io but is self hosted
by selcuka on Monday August 19, 2024 @09:21PM
(#41295962)
> Synchronously verifying it, would probably be too slow.
Why would it be slow? There is a JSON documenbt that lists all IP ranges on the same page you linked to:
https://developers.google.com/static/search/apis/ipranges/go...
Re:13ft – A site similar to 12ft.io but is self hosted
by WatchDog on Monday August 19, 2024 @09:40PM
(#41296048)
Sure, that's one option, and I don't have any insight into what nyt actually does with regards to it's handling of googlebot traffic.
But if I were implementing filtering, I might prefer a solution that doesn't require keeping a whitelist up to date.
Re:13ft – A site similar to 12ft.io but is self hosted
by Gooblebrai on Tuesday August 20, 2024 @02:09AM
(#41297243)
The whitelist can be updated asynchronously
Re:13ft – A site similar to 12ft.io but is self hosted
by bomewish on Tuesday August 20, 2024 @02:47AM
(#41297443)
Maybe we could use GCP infra and the trick will work better ?
Re:13ft – A site similar to 12ft.io but is self hosted
by katzgrau on Monday August 19, 2024 @08:50PM
(#41295785)
There are easily installable databases of IP block info, super easy to do it synchronously, especially if it’s stored in memory. I run a small group of servers that each have to do it thousands of times per second.
Re:13ft – A site similar to 12ft.io but is self hosted
by immibis on Tuesday August 20, 2024 @11:10AM
(#41300793)
Which leads to the possibility of triggering a self-inflicted DoS. I am behind a CGNAT right now. You reckon that if I set myself to Googlebot and loaded NYT, they'd ban the entire o2 mobile network in Germany? (or possibly shared infrastructure with Deutsche Telekom - not sure)
Not to mention the possibility of just filling up the banned IP table.
Re:13ft – A site similar to 12ft.io but is self hosted
by xp84 on Tuesday August 20, 2024 @11:50PM
(#41306617)
Hypothetically if they were doing that, they’d only be ‘banning’ that mobile network in the ‘paywall-relaxing-for-Googlebot’ code - not banning the IP traffic or serving a 403 or anything. They ordinarily throw paywalls at those users anyway.
Synchronously verifying it, would probably be too slow.
You can verify googlebot authenticity by doing a reverse dns lookup, then checking that reverse dns name resolves correctly to the expected IP address[0].
[0]: https://developers.google.com/search/docs/crawling-indexing/...