News for hackers. Stuff that matters.
Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by account42 on Wednesday July 31, 2024 @06:33AM (#41117900)
But you can still make sure that you save the data in a form where generating the served webpage takes the least amount of time. For most websites this means saving the HTML - in a giant cache or with a more deliberate pre-generation setup.
Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by jgalt212 on Wednesday July 31, 2024 @09:24AM (#41118978)
The data structure to html conversion takes milliseconds. That's distinction without a difference.
Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by account42 on Wednesday July 31, 2024 @10:36AM (#41119552)
Clearly not the case with most websites. And "milliseconds" are already a huge amount of time. Video games simulate huge worlds and render complex 3D graphics within 16ms or even much less with the >60 framerates that are expected these days.
Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by jgalt212 on Wednesday July 31, 2024 @06:27PM (#41124186)
You've gotten off topic here. The giant cache you speak of approaches the size of the content database when designing for the long tail of bots. A giant cache is non economic and thus not a solution unless you're an AWS salesman.
Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by account42 on Thursday August 01, 2024 @03:32AM (#41126935)
Not at all. You can either design your website so that pages can be retrieved in sub-millisecond time (and that doesn't have to mean throwing money at cloud providers) or you can cry about bots.
Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by londons_explore on Thursday August 01, 2024 @05:58AM (#41127559)
Yes, the cache ends up being bigger than the content database, but for text content that's typically not a problem. The human effort to type some text always hugely exceeds the cost of a few kilobytes of flash to store what they typed in a ready-to-serve form.

The generation process of taking the raw text and assembling the page around it is typically rather expensive for most CMS systems. Sure - it isn't theoretically expensive, but unless you want to engineer a CMS from scratch most people just pick one off the shelf and then end up having to pay the CPU time overhead of wordpress etc.