News for hackers. Stuff that matters.
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by crmd on Monday May 20, 2024 @04:22PM (#40419850)
The primary use case is not analytics. This data store is the system of record in their credit card authorization and billing pipeline, and so it has extreme consistency requirements. The lion’s share of its engineering is to provide consistency across a large spectrum of failure modes.

Old data could probably live at lower cost in a data warehouse, but then developers would have multiple systems and namespaces to deal with in order to query on transactions.

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by amluto on Monday May 20, 2024 @07:16PM (#40421798)
> This data store is the system of record in their credit card authorization and billing pipeline, and so it has extreme consistency requirements.

This seems like an odd requirement to me. What are the use cases beyond a few minutes past the end of the ride?

- Reconciliation of Uber's finances. This definitely needs to be correct and consistent and maybe even auditable, but it's also done at the end of any given billing period. A daily roll-up would be sufficient.

- Showing each driver and each rider their own history. This ought to be fully correct (especially for drivers), but it's a massive double-edged sword. Uber's app, for example, appears to be entirely missing the "please forget my ride history" button. And this isn't surprising: Uber apparently stores it in a ledger, fully indexed, complete since at least 2017, that is cryptographically immutable. Is this actually a good thing? What happens when a privacy regulator tells Uber to expunge, anonymize, or pseudonymize old data?

- Handling chargebacks? Having a high quality record going back 180-365 days seems useful. But the cost to Uber of occasionally losing a record is very low and is barely more than proportional to the probability of loss of a given record.

So I don't get it. If I were running an operation like this, I would prefer not to have a fully immutable ledger full of personally identifiable location, usage, and transactional data.

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by nikolajan on Monday May 20, 2024 @08:16PM (#40422383)
>This seems like an odd requirement to me. What are the use cases beyond a few minutes past the end of the ride?

Exactly what he the OP said, authorization. Card networks cap latency of authorization requests (the time the merchant submits a request, to when it makes its make through the merchant processor, networks, issuer processor and back.)

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by kikimora on Tuesday May 21, 2024 @10:13AM (#40428753)
Capture can happen days after successful authorization.
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by 8ytecoder on Monday May 20, 2024 @06:34PM (#40421287)
Which is …normal. Different types of data access have different SLA requirements. Any company that cares about cost will warehouse data after x time frame. It’s done even in banking. Uber has a much more lenient need to make this data instantly available.
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by nikolajan on Monday May 20, 2024 @08:20PM (#40422421)
> It’s done even in banking

What are you basing this on, banks have less real time constraints when it comes to certain data but thats rare. You will almost never see a bank, processor, etc. archive their system of record data.

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by jiggawatts on Monday May 20, 2024 @06:40PM (#40421352)
Immutable data is always consistent. This is almost certainly an append-only ledger, a well-established solution for a simple problem.
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by lozenge on Tuesday May 21, 2024 @03:55AM (#40425322)
"then developers would have multiple systems and namespaces to deal with in order to query on transactions."

Yes, which they already had as DynamoDB only held 8 weeks of data. Presumably all payments to drivers were calculated monthly and turned into invoices. Any corrections would end up as adjustments on future invoices. Seems pretty normal.