News for hackers. Stuff that matters.
Re:Tell HN: Server error (5xx) in Google Search Console may ... by agpl3141592 on Wednesday August 14, 2024 @04:53PM (#41250399)
Sounds more like an issue on cloud flare or whatever server Google is hitting.

It should return 429 so Google can reduce the requests.

I'm not even sure why you would rate limit in the first place? IPS are not unique. One company gateway or university for example has plenty of users.

Rate limit requests from users you know and make sure every public API is properly cached.

Re:Tell HN: Server error (5xx) in Google Search Console may ... by renewiltord on Wednesday August 14, 2024 @05:04PM (#41250477)
Why do you say that? The 429 edge status code implies quite strongly that the client gets 429s as well.
Re:Tell HN: Server error (5xx) in Google Search Console may ... by ComputerGuru on Wednesday August 14, 2024 @05:53PM (#41250902)
Not hard to test it from a vpn connection and verify.
Re:Tell HN: Server error (5xx) in Google Search Console may ... by andersa on Wednesday August 14, 2024 @05:44PM (#41250824)
Cloudflare doesn't offer rate limiting by user.

Caching is completely useless as a replacement for rate limiting. Simply add a bunch of nonsense to the url, or change query parameters, and you will bypass the cache.

Re:Tell HN: Server error (5xx) in Google Search Console may ... by jhy on Wednesday August 14, 2024 @06:20PM (#41251137)
Cloudflare does offer rate limiting - it's in Security / WAF / Rate limiting rules.

On the free plan it is perhaps a bit limited in the rules you can create (match based on bot or URL regex, limit by # requests per IP per 10 seconds). But still pretty useful for slowing stupid requests.

https://developers.cloudflare.com/waf/rate-limiting-rules/cr...

And / or you can use Transform Rules to better normalize the Origin URLs so that the query string or other path info doesn't create new origin requests. Or more simply, enable Ignore Query String in the Cache Settings.

Re:Tell HN: Server error (5xx) in Google Search Console may ... by christina97 on Wednesday August 14, 2024 @06:08PM (#41251022)
That’s not proper caching then.
Re:Tell HN: Server error (5xx) in Google Search Console may ... by santah on Thursday August 15, 2024 @12:44AM (#41253143)
I rate limit because of aggressive bots (scrapers and ones trying random injection attacks etc).

And yes, it's returning 429 ...