News for hackers. Stuff that matters.
Re:Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by NeedMoreTea on Wednesday October 17, 2018 @08:30PM (#18245016)
Which is about 99.7kb more than I would hope and expect as a customer.
Re:Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by Jedd on Wednesday October 17, 2018 @10:59PM (#18245588)
That's a fairly naive expectation.

I had a friend head off in 2011 to work with Hailo in London, and designed much of the data ingest & storage architecture there. That was built on Cassandra, and some of the numbers were staggering, even for a (initially) single-city, < 2,000 driver system.

You're not just tracking start and end point of successful rides -- you're tracking users when their app is open (both drivers and users move around before and after initiating a ride, and you need to share that data between parties), you're also tracking empty vehicles so you can identify over / under serviced areas ... doubtless lots of other real time stuff you want to know. But the big thing is you want to then have that data available for mining, improving your service, regulatory obligations, and so on.

Re:Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by NeedMoreTea on Thursday October 18, 2018 @01:39AM (#18246155)
Only if you believe in retaining all data in perpetuity. I suppose it demonstrates the difference in US and European perspectives on data.

Most of what you describe has little meaning or use at a few months old. Sure you might want to collect detailed awareness of individual drivers over say a month, generic traffic patterns over six months, and little or nothing beyond.

Once the data has outlived that direct, primary, purpose you dispose of it or it's a famous, excessive, data breach waiting to happen. There is absolutely nothing wrong with only keeping billing data beyond say six months max.

Re:Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by samontar on Thursday October 18, 2018 @03:30AM (#18246585)
In California, you soon won’t be able to keep 6 months max. You have to hold a year.
Re:Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by arbie on Thursday October 18, 2018 @02:34AM (#18246388)
> Most of what you describe has little meaning or use at a few months old. Sure you might want to collect detailed awareness of individual drivers over say a month, generic traffic patterns over six months, and little or nothing beyond.

The really invasive pattern-matches happen best retroactively at that scale.

Age of the data does not typically impact many such analyses.

Re:Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by Jedd on Friday October 19, 2018 @10:23PM (#18261372)
> Only if you believe in retaining all data in perpetuity.

There's a huge spectrum between your 'at most 30d' claim in another reply, and 'all data in perpetuity'.

30d is way too short. Perpetuity is probably the idealistic intent, and depending on storage costs over time this may be achievable.

> I suppose it demonstrates the difference in US and European perspectives on data.

How so? I described a the case of one UK-based company, and TFA is about one US-based company, both doing similar things.

> Most of what you describe has little meaning or use at a few months old.

Clearly this is not the case, as evinced by the actions of the people designing, building, operating, and maintaining these types of systems.

Re:Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by singingboyo on Wednesday October 17, 2018 @09:11PM (#18245172)
I think there are tradeoffs, for example, they probably store the route, which is good (refunds for really terrible routes) but bad (tracking). That's quite a bit of space, probably the big one. Rating and possibly comments, again could be pretty big. Multiple UUIDs or whatever (cust/driver IDs, CC txn IDs) will take up some space as well. 300B is probably a fair bit too low.

That said, 100kB does seem high. 10kB or so would seem more appropriate, as a very rough ballpark.

Re:Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by btown on Wednesday October 17, 2018 @10:10PM (#18245386)
Beyond the route itself, consider also: time spent in traffic (speed per geo-coordinate) for use in predicting surges, deviations between planned and taken routes, etc. The degree to which Uber has a pulse on urban traffic is fascinating and probably exceeds even city planners with access to ubiquitous traffic cameras.
Re:Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by rasen58 on Wednesday October 17, 2018 @09:43PM (#18245306)
They probably need to store a ton of things (like maps) that aren't data though, so probably not fair to say each ride is 100kb.
Re:Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by Invictus0 on Wednesday October 17, 2018 @10:38PM (#18245501)
Even if they have 20PB of maps, which is probably way too high, it's still 80kb.
Re:Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by NeedMoreTea on Wednesday October 17, 2018 @10:03PM (#18245363)
Almost none of which they need to keep. Route for 30 days perhaps, start and end point for longer. I imagine most ratings are just that with no comments.
Re:Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by MR4D on Wednesday October 17, 2018 @10:16PM (#18245407)
Routes determine pay to drivers, so they probably have to keep it much longer in case of a pay dispute.
Re:Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by p1necone on Wednesday October 17, 2018 @11:56PM (#18245786)
They could just give a digitally signed copy to the drivers and have the onus be on them to keep it.