Uber Migrates 1T Records from DynamoDB to LedgerStore to Save $6M Annually

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by Closi on Monday May 20, 2024 @11:35AM (#40416676)

1.7 petabytes on Sqlite?

Sqlite's own advice:

> If your data will grow to a size that you are uncomfortable or unable to fit into a single disk file, then you should select a solution other than SQLite. SQLite supports databases up to 281 terabytes in size, assuming you can find a disk drive and filesystem that will support 281-terabyte files.

> Even so, when the size of the content looks like it might creep into the terabyte range, it would be good to consider a centralized client/server database [over SQLite].

[ Parent | on HN ]

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by cheeze on Monday May 20, 2024 @02:43PM (#40418666)

This is the worry IMO. It's fine to dump it on a server with SQLite, but once you start hitting scaling limits, you're in for a potentially rough migration.

[ Parent | on HN ]

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by faizshah on Tuesday May 21, 2024 @03:43AM (#40425255)

You just shard it into much smaller SQLite DBs by a partition key if you really wanted to do this.

Not really sure why you would subject yourself to this though, would be way easier just to use Cassandra/Scylla.

[ Parent | on HN ]