News for hackers. Stuff that matters.

A comment on Roblox downloaded 145GB models from us (including adult content) without asking. Parent comment

Re:Roblox downloaded 145GB models from us (including adult c... by DaSHacka on Tuesday October 27, 2026 @10:55AM (#50260339)
Depending on the service, you may not want it to be indexed anyway (IE login pages) or, if you do, most commercial crawlers have methods to allow site owners to verify that the traffic is a legitimate crawler and not an impersonator, see:

https://developers.google.com/crawling/docs/crawlers-fetcher...

(Though restricting the Google UA to only legitimate Google IPs seemingly would break Brave's crawler, though that's their own fault: https://safe.search.brave.com/help/brave-search-crawler/)

https://duckduckgo.com/duckduckgo-help-pages/results/duckduc...

https://www.bing.com/webmasters/help/how-to-verify-bingbot-3...

https://kagi.com/bot

[ Parent | on HN ]
Re:Roblox downloaded 145GB models from us (including adult c... by xmcp123 on Tuesday October 27, 2026 @02:51PM (#50263945)
It is trivial to forge a Google bot user agent and scrape through GCP.
[ Parent | on HN ]
Re:Roblox downloaded 145GB models from us (including adult c... by DaSHacka on Tuesday October 27, 2026 @03:53PM (#50264831)
That wouldn't work if a sysadmin implemented the rDNS lookup mentioned in TFA. The PTR specifically must end in `.googlebot.com`, not `.googleusercontent.com`.
[ Parent | on HN ]
Re:Roblox downloaded 145GB models from us (including adult c... by xmcp123 on Wednesday October 28, 2026 @02:32PM (#50278718)
No one does that though. The latency would be significant.
[ Parent | on HN ]