News for hackers. Stuff that matters.

A comment on Uber’s Big Data Platform: 100+ Petabytes with Minute Latency. Parent comment

Re:Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by singingboyo on Wednesday October 17, 2018 @09:11PM (#18245172)
I think there are tradeoffs, for example, they probably store the route, which is good (refunds for really terrible routes) but bad (tracking). That's quite a bit of space, probably the big one. Rating and possibly comments, again could be pretty big. Multiple UUIDs or whatever (cust/driver IDs, CC txn IDs) will take up some space as well. 300B is probably a fair bit too low.

That said, 100kB does seem high. 10kB or so would seem more appropriate, as a very rough ballpark.

[ Parent | on HN ]
Re:Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by btown on Wednesday October 17, 2018 @10:10PM (#18245386)
Beyond the route itself, consider also: time spent in traffic (speed per geo-coordinate) for use in predicting surges, deviations between planned and taken routes, etc. The degree to which Uber has a pulse on urban traffic is fascinating and probably exceeds even city planners with access to ubiquitous traffic cameras.
[ Parent | on HN ]
Re:Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by rasen58 on Wednesday October 17, 2018 @09:43PM (#18245306)
They probably need to store a ton of things (like maps) that aren't data though, so probably not fair to say each ride is 100kb.
[ Parent | on HN ]
Re:Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by Invictus0 on Wednesday October 17, 2018 @10:38PM (#18245501)
Even if they have 20PB of maps, which is probably way too high, it's still 80kb.
[ Parent | on HN ]
Re:Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by NeedMoreTea on Wednesday October 17, 2018 @10:03PM (#18245363)
Almost none of which they need to keep. Route for 30 days perhaps, start and end point for longer. I imagine most ratings are just that with no comments.
[ Parent | on HN ]
Re:Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by MR4D on Wednesday October 17, 2018 @10:16PM (#18245407)
Routes determine pay to drivers, so they probably have to keep it much longer in case of a pay dispute.
[ Parent | on HN ]
Re:Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by p1necone on Wednesday October 17, 2018 @11:56PM (#18245786)
They could just give a digitally signed copy to the drivers and have the onus be on them to keep it.
[ Parent | on HN ]