News for hackers. Stuff that matters.
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by xyst on Monday May 20, 2024 @11:22AM (#40416550)
Uber must have picked up some Google rejects. This type of homegrown project was seen at Google all the time.

Usually to aim for a significant promotion.

“Designed and built homegrown system to save $Xm! Give me promo, bro?”

Just so happened to ignore that it took X+Y additional to build. Also it will probably be going to the G graveyard in a few years.

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by cj on Monday May 20, 2024 @12:01PM (#40416930)
$6m in annual cost savings is truly unremarkable, if we are to believe levels.fyi [0] [1]

If you're truly paying engineers, project managers, etc $500k a head, it dramatically undermines the financial cost savings.

It very well might be the case that "We spent $25m of engineering resources to save $6m annually".

[0] https://www.levels.fyi/companies/uber/salaries/software-engi...

[1] https://www.levels.fyi/companies/uber/salaries/software-engi...

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by booi on Monday May 20, 2024 @12:05PM (#40416971)
That’s what I was thinking and fully loaded cost at least 35% more than their salary.

Imagine trading 5 headcount full-time to manage the 1T+ fully custom database on an ongoing basis when they could have just used DynamoDB and have been done with it.

Or better, having to engineer a new feature that already existed in DynamoDB and just losing money at that point.

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by dangus on Monday May 20, 2024 @12:16PM (#40417059)
Of course now we are assuming that the existing solution didn’t also require engineering salaries to maintain.
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by leoqa on Monday May 20, 2024 @12:24PM (#40417150)
Bitter story time:

I made a config change to our AWS instances and projected approximately $10MM/year in AWS costs savings (pre-savings).

My boss asked me "Who told you to do this? We need to focus on $project instead". I found another team and transferred out. 3 months later there was a big fire drill about AWS costs and they took my 1-pager and executed it. Didn't get any credit in the shipped email nor did the manager reach out to apologize.

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by cj on Monday May 20, 2024 @12:46PM (#40417372)
The organization still got the end result, though (to play devil's advocate). That sounds like a win for the company. They got the cost savings, plus they redirected attention back to a project that was higher priority than saving $10m 3 months earlier than they could have.
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by klodolph on Monday May 20, 2024 @11:26PM (#40423761)
It may be a win for the company in some categories, but a loss in others.
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by heavenlyblue on Monday May 20, 2024 @01:43PM (#40417999)
Of course you didn't. You used your time to promote yourself instead of doing what you were asked to do instead. That could have cost a promotion for your manager who could have promoted you.
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by noncoml on Monday May 20, 2024 @03:41PM (#40419379)
I don’t know you are downvoted. Aligning the personal interest vector with the companies interest vector is a huge problem that is usually underrepresented in NH comments.

Usually we only complain about the short sighting of the CEOs that prioritize short term stock gains over long term prosperity, but that also is just a specialized case of the success vector misalignment

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by nostrebored on Monday May 20, 2024 @10:16PM (#40423327)
This is a poor way to view salaried work.

If OP is meeting deliverables and did this on the side, this is a great example of a story that would make me inclined to hire.

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by ckdarby on Tuesday May 21, 2024 @07:54AM (#40427142)
The problem is OP didn't meet deliverables and the team was behind schedule.

Helpful in the right light, but equally not helpful in another.

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by shermantanktop on Tuesday May 21, 2024 @11:01AM (#40429348)
“Team behind schedule” usually means that some manager promised an arbitrary date based on an imaginary timeline.

Every team needs slack. Bad managers hate slack. Good managers hide slack.

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by noncoml on Tuesday May 21, 2024 @02:06PM (#40431727)
I think maybe my point was misunderstood.

I was trying to say that the employee tried to do what’s good for the company. However that backfired as his personal, and his manager’s, success vector are not aligned with the company’s. Hence although he did something that benefited the company it hurt his and his manager’s performance (exactly because they are not aligned with that of the company’s).

Hope it clears things up.

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by leoqa on Monday May 20, 2024 @10:38PM (#40423452)
We were migrating to Kubernetes from fixed hosts and had to buy the capacity. I just spent a day looking at our utilization and made a better decision about what instance type to use.
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by RobRivera on Monday May 20, 2024 @09:56PM (#40423219)
Sounds like a fantastic way to maneuver and get credit for someone else's proactive engenuity
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by shawabawa3 on Monday May 20, 2024 @02:33PM (#40418546)
> we spent $25m of engineering resources to save $6m annually

This is a huge ROI. Borrowing $25m costs about $1.25m/yr so you're winning even with no upfront costs

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by valiant55 on Monday May 20, 2024 @03:07PM (#40418984)
I mean, ideally you are still employing those people into the future. Plus was there other opportunities to drive value that would've been better spent?
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by vineyardmike on Monday May 20, 2024 @05:00PM (#40420328)
Ideally for the people, but not a requirement. I doubt they won't be conducting more layoffs either.
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by sokoloff on Tuesday May 21, 2024 @12:38AM (#40424189)
That math assumes the on-going maintenance of LedgerStore is no higher than that of using DynamoDB was.

I suspect there's now a team of technologists maintaining, securing, and operating LedgerStore.

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by shawabawa3 on Tuesday May 21, 2024 @04:49AM (#40425706)
I suspect they already had a team maintaining/debugging/operating dynamodb

Hosted services doesn't reduce maintenance costs to zero

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by sokoloff on Tuesday May 21, 2024 @05:29AM (#40426077)
They have teams maintaining/debugging/operating their use/consumption of dynamodb; they don't have teams maintaining/debugging/operating/securing dynamodb itself.

With LedgerDB, they now have to do both.

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by shermantanktop on Tuesday May 21, 2024 @12:13PM (#40430373)
Optimizing to address high percentile issues (latency/error/correctness) on someone else’s code follows a bimodal pattern: it’s easy until it is impossible. Doing it on your own code is progressively more difficult and the complexity may cause you to give up, but you normally don’t hit the same wall.
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by johannes1234321 on Tuesday May 21, 2024 @04:39AM (#40425637)
Question is: Where else could they have spent those $25m? Could they have built something with more value?

Opportunity cost is a cost, too

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by shawabawa3 on Tuesday May 21, 2024 @04:48AM (#40425699)
if at uber's scale they still have enough opportunities of that scale for every one of their thousands of engineers i need to buy some of their stock
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by choppaface on Monday May 20, 2024 @03:48PM (#40419451)
Yes but since revenue is growing at over 70% due to squeezing out the drivers, there's more money to spend on fighting Amazon over the DynamoDB contract https://www.forbes.com/sites/lensherman/2023/01/16/ubers-new...
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by amanda99 on Monday May 20, 2024 @04:53PM (#40420250)
Right, but that's $25m in R&D investment. Much better than $6m in cost of good/services delivered! Former is great innovation and will be ignored by investors because it's just a fixed cost on the way to becoming profitable. The latter is going to appear in the marginal cost of services calculation.
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by chii on Tuesday May 21, 2024 @02:18AM (#40424764)
$25m spend to save $6m annually is a return on investment of 4 (and a bit) years to break even, then it's positive afterwards. So as long as the project runs for more than 4 years, they'd be ahead.

The only problem is that the database code might need maintenance, specs change or situations change, and the cost is _not_ 25m, but way more per year than anticipated. So much so that after a few years, a new batch of engineers start eyeing writing a new database system to purportedly save money...

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by password4321 on Monday May 20, 2024 @01:01PM (#40417535)
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by amluto on Monday May 20, 2024 @03:34PM (#40419304)
The thing I find odd about this is that the headline figure is about old immutable records. Almost all of that 1.7PB is ancient by what seems to be to by any practical standard. Uber is not likely to care about the credit card authorization flow for a ride two years ago, except maybe for analytics.

If I were doing this, I would be looking at data warehousing systems. 1.7PB of, say, Parquet files in S3 is not terribly expensive. 1.7PB of Parquet files in on-prem or collocated object storage, even replicated a zillion times, is quite cheap. And quite a few companies and open-source projects are currently competing aggressively to provide awesome tools for querying that data.

The hot data would fit on basically anything — the choice should be about robustness and barely even consider cost per TiB. Datomic got written up recently and seems credible for this type of application. FoundationDB is bulletproof. Postgres could probably handle it without breaking a sweat, although active/active replication isn’t free. Heck, writes straight to a warehouse with a cache in front to help with reads seems credible — Uber rides rarely go for longer than a couple hours, and back-of-the-envelope math suggests that the total data rate is maybe 50GB/hour. An entire day of data for an entire country would fit on a single very ordinary commodity server, and the live data for the entire world would fit on one mildly beefy server. The indexes involved sound straightforward.

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by crmd on Monday May 20, 2024 @04:22PM (#40419850)
The primary use case is not analytics. This data store is the system of record in their credit card authorization and billing pipeline, and so it has extreme consistency requirements. The lion’s share of its engineering is to provide consistency across a large spectrum of failure modes.

Old data could probably live at lower cost in a data warehouse, but then developers would have multiple systems and namespaces to deal with in order to query on transactions.

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by amluto on Monday May 20, 2024 @07:16PM (#40421798)
> This data store is the system of record in their credit card authorization and billing pipeline, and so it has extreme consistency requirements.

This seems like an odd requirement to me. What are the use cases beyond a few minutes past the end of the ride?

- Reconciliation of Uber's finances. This definitely needs to be correct and consistent and maybe even auditable, but it's also done at the end of any given billing period. A daily roll-up would be sufficient.

- Showing each driver and each rider their own history. This ought to be fully correct (especially for drivers), but it's a massive double-edged sword. Uber's app, for example, appears to be entirely missing the "please forget my ride history" button. And this isn't surprising: Uber apparently stores it in a ledger, fully indexed, complete since at least 2017, that is cryptographically immutable. Is this actually a good thing? What happens when a privacy regulator tells Uber to expunge, anonymize, or pseudonymize old data?

- Handling chargebacks? Having a high quality record going back 180-365 days seems useful. But the cost to Uber of occasionally losing a record is very low and is barely more than proportional to the probability of loss of a given record.

So I don't get it. If I were running an operation like this, I would prefer not to have a fully immutable ledger full of personally identifiable location, usage, and transactional data.

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by nikolajan on Monday May 20, 2024 @08:16PM (#40422383)
>This seems like an odd requirement to me. What are the use cases beyond a few minutes past the end of the ride?

Exactly what he the OP said, authorization. Card networks cap latency of authorization requests (the time the merchant submits a request, to when it makes its make through the merchant processor, networks, issuer processor and back.)

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by kikimora on Tuesday May 21, 2024 @10:13AM (#40428753)
Capture can happen days after successful authorization.
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by 8ytecoder on Monday May 20, 2024 @06:34PM (#40421287)
Which is …normal. Different types of data access have different SLA requirements. Any company that cares about cost will warehouse data after x time frame. It’s done even in banking. Uber has a much more lenient need to make this data instantly available.
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by nikolajan on Monday May 20, 2024 @08:20PM (#40422421)
> It’s done even in banking

What are you basing this on, banks have less real time constraints when it comes to certain data but thats rare. You will almost never see a bank, processor, etc. archive their system of record data.

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by jiggawatts on Monday May 20, 2024 @06:40PM (#40421352)
Immutable data is always consistent. This is almost certainly an append-only ledger, a well-established solution for a simple problem.
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by lozenge on Tuesday May 21, 2024 @03:55AM (#40425322)
"then developers would have multiple systems and namespaces to deal with in order to query on transactions."

Yes, which they already had as DynamoDB only held 8 weeks of data. Presumably all payments to drivers were calculated monthly and turned into invoices. Any corrections would end up as adjustments on future invoices. Seems pretty normal.

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by smileysteve on Monday May 20, 2024 @11:20PM (#40423725)
This. DynamoDB will export to parquet (or csv); s3 select becomes possible for data. More importantly, Athena can query it all
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by marcyb5st on Tuesday May 21, 2024 @10:49AM (#40429217)
In some countries (e.g. Italy, France) you are required by law to keep records of transactions and other data for 10y. But I agree with the data tiering part. Anything older than X months should go to storage (parquet files in some cloud storage).
Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by MarkMarine on Monday May 20, 2024 @03:35PM (#40419310)
I heard from some X-Uber people that you could call Uber a database company as much as you could call it a transportation company. Something like 80+ databases invented there in one form or another.

Promotion-driven development. I suppose better than blog post driven development, but marginally so.

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by vineyardmike on Monday May 20, 2024 @04:56PM (#40420281)
> I suppose better than blog post driven development, but marginally so.

I find DB development quite interesting, and I find Uber's core product quite not interesting (from an engineering perspective).

So as an outsider without any financial stake in the company, please keep writing about databases!

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by surfingdino on Tuesday May 21, 2024 @03:37AM (#40425217)
> I heard from some X-Uber people that you could call Uber a database company

Someone once called Airbus planes "Sun servers with wings". All businesses today have an IT department and those build around the idea of real-time scheduling done by computers are compute/storage businesses first. That's why Uber was able to launch Uber Eats or e-bikes. I am not surprised they have a bunch of busy devs building cool stuff. Not all of it is ego-driven. I liked their logger for Go last time I used it.

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by MarkMarine on Tuesday May 21, 2024 @11:38AM (#40429832)
nih-syndrome is a real thing. I'm not saying they don't make cool stuff but the engineering culture that prioritizes creating a new database over using something that already exists, then promoting the people who develop the (70th) new DB over the people who were focused on actually delivering value for the customer, it's not my choice in working environment.

The architecture where all the magic lives in the database layer and the services treat the DB like this magical thing that synchronizes across multiple DCs for you, and takes care of every complicated part of engineering a distributed system is a "now you have two problems" situation. It's a massive complicated system that will fail in novel ways, and you don't have a large community of other users that you can fall back on for expertise.

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by surfingdino on Tuesday May 21, 2024 @01:35PM (#40431388)
> you don't have a large community of other users that you can fall back on for expertise.

That is one of the most important points of consideration when I am being asked to asses new tech. If the project is not actively supported by the community, I don't recommend it. I just can't expose the client to risk just because there's a new shiny thing.

Re:Uber Migrates 1T Records from DynamoDB to LedgerStore to ... by danpalmer on Monday May 20, 2024 @08:50PM (#40422718)
I'm not sure if this is true of modern Google. Certainly there are a ton of custom things around, but it also seems like in the last 5 years there has been a huge push towards not re-inventing more databases/etc, and normalising everything onto a few core technologies. A good example is Spanner, many products have been moved off other home-grown systems onto Spanner, basically to reduce complexity and cost. Sure Spanner is a Google technology, but at least it's one, instead of having all sorts of different esoteric systems, and Spanner is just one example, this is happening in a number of different domains.