Re:Websites and web developers mostly don't care about clien...
by senko on Saturday August 23, 2025 @11:04AM
(#44996508)
Not OP, but many technologies nowadays push users to use a server-side component when not needed.
An example is NextJS where you're strongly encouraged[0] to run a server (or use a platform like Vercel), even if what you're doing is a fairly simple static site.
Combine inconsiderate crawler (AI or otherwise) with a server-side logic that doesn't really need to be there and you have a recipe for a crash, a big hosting bill, or both.
[0] People see https://nextjs.org/docs/app/guides/static-exports#unsupporte... and go "ah shucks I better have a server component then"
Re:Websites and web developers mostly don't care about clien...
by dehrmann on Saturday August 23, 2025 @01:25PM
(#44997527)
> My forum traffic...
> Why did you bring up static pages served by a CDN...
This is easier said than done, but pushing the latest topic snapshot to the CDN whenever a post is made is doable.
> Loading static pages from CDN to scrape training data takes such minimal amounts of resources that it's never going to be a significant part of my costs. Are there cases where this isn't true?
Why did you bring up static pages served by a CDN, the absolute best case scenario, as your reference for how crawler spam might affect server performance?