Uber Migrates 1T Records from DynamoDB to LedgerStore to Save $6M Annually

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by bayindirh on Monday May 20, 2024 @07:17AM (#40414273)

If you install a RAID controller and a couple of disk boxes, it's possible with 1:1 replication, or with backups. 60 disk 3.5" units already exist, so 2.5" SSD racks. It won't be cheap, but will be resilient and fast. Bloody fast if you have the budget.

[ Parent | on HN ]

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by zaphirplane on Monday May 20, 2024 @07:39AM (#40414390)

> or with backups take a while to restore a PB and a way to take a hot backup without impacting the service that by itself is a task or snapshots which is more disk

> 1:1 replication Depending on the amount of writes could be a ton of extra disk and a bucket for network cost

[ Parent | on HN ]

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by bayindirh on Monday May 20, 2024 @07:50AM (#40414445)

These systems support zero-downtime snapshots. You tell it to snapshot, it instantly snapshots, you can run a differential/incremental backup at great speeds. Your RAID controller is already caching the hot data, so the impact is minimal.

Except network cost there's no extra disk required. It's just broadcasted writes consumed on the other hand.

These boxes are not dumb JBODS. They support their own replication/backup subsystems, so everything is transparent.

[ Parent | on HN ]

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by Closi on Monday May 20, 2024 @11:39AM (#40416709)

Resilient and fast from a disk perspective, but in practice massively bottlenecked by the fact that Sqlite can only have 1 writer at a time.

[ Parent | on HN ]