Re:Dear AI Companies, instead of scraping OpenStreetMap, how...
by account42 on Wednesday July 31, 2024 @10:36AM
(#41119552)
Clearly not the case with most websites. And "milliseconds" are already a huge amount of time. Video games simulate huge worlds and render complex 3D graphics within 16ms or even much less with the >60 framerates that are expected these days.
[ Parent
| on HN ]
Re:Dear AI Companies, instead of scraping OpenStreetMap, how...
by jgalt212 on Wednesday July 31, 2024 @06:27PM
(#41124186)
You've gotten off topic here. The giant cache you speak of approaches the size of the content database when designing for the long tail of bots. A giant cache is non economic and thus not a solution unless you're an AWS salesman.
Re:Dear AI Companies, instead of scraping OpenStreetMap, how...
by account42 on Thursday August 01, 2024 @03:32AM
(#41126935)
Not at all. You can either design your website so that pages can be retrieved in sub-millisecond time (and that doesn't have to mean throwing money at cloud providers) or you can cry about bots.
Re:Dear AI Companies, instead of scraping OpenStreetMap, how...
by londons_explore on Thursday August 01, 2024 @05:58AM
(#41127559)
Yes, the cache ends up being bigger than the content database, but for text content that's typically not a problem. The human effort to type some text always hugely exceeds the cost of a few kilobytes of flash to store what they typed in a ready-to-serve form.
The generation process of taking the raw text and assembling the page around it is typically rather expensive for most CMS systems. Sure - it isn't theoretically expensive, but unless you want to engineer a CMS from scratch most people just pick one off the shelf and then end up having to pay the CPU time overhead of wordpress etc.