News for hackers. Stuff that matters.
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by BlackLotus89 on Monday May 20, 2024 @07:07AM (#40414215)
No it won't. sqlite "only" works with up to 281TB [0] [1]

[0] https://www.sqlite.org/releaselog/3_33_0.html

[1] https://www.sqlite.org/limits.html (#12)

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by sgt on Monday May 20, 2024 @07:09AM (#40414226)
You can split up into 10 SQLite DB's on this individual server.
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by mlnj on Monday May 20, 2024 @07:54AM (#40414467)
Just storing petabytes of data is not the issue. Managing and querying it reliably is.
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by zaphirplane on Monday May 20, 2024 @07:47AM (#40414436)
> You can split up into 10 SQLite DB's on this individual server.

1 is a scalable, managed, highly available service, with economies of scale the other is a fixed size, capital expenditure with fixed performance, limited DR, requiring a couple of SRE/DevOps and colo

There is also the will it always work question

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by cdchn on Monday May 20, 2024 @07:40AM (#40414402)
You've now implemented sharding on top of SQLite.

Eventually all programs will be able to read email.

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by mrbungie on Monday May 20, 2024 @08:10AM (#40414582)
Any non-trivial complexity codebase eventually implements a mediocre SQL/Lisp/etc.
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by Closi on Monday May 20, 2024 @10:18AM (#40415724)
Exactly! Why take a system not designed for this sort of scale and force it to scale, rather than use systems which are designed and tested for this scale and volume? All you will do is hackily re-invent all the other things that the other databases had to do to scale to this extent.

Plus size is only one limit, you would be limited to 1 write every few milliseconds. My napkin maths estimate is that there are at least 1-2m writes per hour going into this thing, so probably 300-600 writes / second (Average) and maybe over 1k writes/second peak. We are going to fall over here!

Not sure why some people seem to have a viwe of "There is no scaling problem that can't be solved with a sufficient enough number of SQLite databases".

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by chasil on Monday May 20, 2024 @11:57AM (#40416892)
Beware of WAL mode, as you sacrifice ACID in this configuration.

https://sqlite.org/lang_attach.html

'Transactions involving multiple attached databases are atomic, assuming that the main database is not ":memory:" and the journal_mode is not WAL. If the main database is ":memory:" or if the journal_mode is WAL, then transactions continue to be atomic within each individual database file. But if the host computer crashes in the middle of a COMMIT where two or more database files are updated, some of those files might get the changes where others might not.'

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by nemothekid on Monday May 20, 2024 @03:02PM (#40418918)
Once you are splitting up 10 sqlite dbs you have a bespoke distributed system anyways, and you will find yourself doing all the headache of LedgerStore anyways.

Most of the novel work in LedgerStore is probably around managing the headaches of distributed storage, not the persistence layer.