Re:Ask HN: Website with 6^16 subpages and 80k+ daily bots
by damir on Wednesday October 23, 2024 @02:54PM
(#41928199)
Hey, maybe you are right, maybe some stats on which bots from how many IPs have how many hits per hour/day/week etc...
Thank's for the idea!
Re:Ask HN: Website with 6^16 subpages and 80k+ daily bots
by eddd-ddde on Wednesday October 23, 2024 @06:08PM
(#41929683)
Huh, for some reason I assumed this was precompiled / statically generated. Not that fun once you see it as a single page.
Re:Ask HN: Website with 6^16 subpages and 80k+ daily bots
by TeMPOraL on Thursday October 24, 2024 @03:52AM
(#41933126)
FWIW, a billion static pages vs. single script with URL rewrite that makes it look like a billion static pages are effectively equivalent, once a cache gets involved.
Re:Ask HN: Website with 6^16 subpages and 80k+ daily bots
by eddd-ddde on Thursday October 24, 2024 @07:32AM
(#41934489)
Kinda true, but then the real "billion page site" is just cloudflare or something.
Re:Ask HN: Website with 6^16 subpages and 80k+ daily bots
by TeMPOraL on Thursday October 24, 2024 @10:46AM
(#41936025)
Cloudflare succeeded where AMP failed.
Re:Ask HN: Website with 6^16 subpages and 80k+ daily bots
by bigiain on Thursday October 24, 2024 @01:04AM
(#41932092)
> Which ones respect robots.txt
Add user agent specific disallow rules so different crawlers get blocked off from different R G or B values.
Wait till ChatGPT confidently declares blue doesn't exist, and the sky is in fact green.
Which is why I'd answer your question by recommending that you focus on the bots, not your content. What are they? How often do they hit the page? How deep do they crawl? Which ones respect robots.txt, and which do not?
Go create some bot-focused data. See if there is anything interesting in there.