News for hackers. Stuff that matters.
Re:In Search of a Faster SQLite by rmbyrro on Monday December 16, 2024 @12:56PM (#42433448)
> Now my functions just downloaded the file from S3, if it was newer than the local copy

if you have strong consistency requirements, this doesn't work. synchronizing clocks reliably between different servers is surprisingly hard. you might end up working with stale data. might work for use cases that can accept eventual consistency.

Re:In Search of a Faster SQLite by Dylan16807 on Monday December 16, 2024 @02:27PM (#42434402)
If you have strong consistency requirements, then it doesn't work by the very nature of making multiple copies of the database. Even if the clocks are perfect. (Though the clocks are probably close enough that it doesn't matter.)
Re:In Search of a Faster SQLite by jedberg on Monday December 16, 2024 @02:11PM (#42434225)
One of the announcements from AWS this year at Re:invent is that they now can guarantee that the instances clocks are synced within microseconds of each other. Close enough that you can rely on it for distributed timekeeping.
Re:In Search of a Faster SQLite by efitz on Monday December 16, 2024 @02:32PM (#42434451)
I versioned the databases and my use case didn’t require strong consistency.
Re:In Search of a Faster SQLite by Spivak on Monday December 16, 2024 @01:05PM (#42433553)
I don't really know if that matters for this use case. Just by the very nature of source_data -> processing -> dest_data taking nonzero time anything consuming dest_data must already be tolerant of some amount of lag. And how it's coded guarantees you can never observe dest_data going new -> old -> new.
Re:In Search of a Faster SQLite by iskela on Monday December 16, 2024 @05:16PM (#42436074)
Wouldnt e-tag version numbers also work? Or just havkng .jsom with version metadata next to the db blob? No need to sync clocks. Just GET the small db-ver.json and compare version details?
Re:In Search of a Faster SQLite by 66yatman on Monday December 30, 2024 @12:21AM (#42546702)
This shouldn't depend on clocks, just tracking Etag is more consistency proof.