News for hackers. Stuff that matters.
Re:Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by NeedMoreTea on Thursday October 18, 2018 @01:39AM (#18246155)
Only if you believe in retaining all data in perpetuity. I suppose it demonstrates the difference in US and European perspectives on data.

Most of what you describe has little meaning or use at a few months old. Sure you might want to collect detailed awareness of individual drivers over say a month, generic traffic patterns over six months, and little or nothing beyond.

Once the data has outlived that direct, primary, purpose you dispose of it or it's a famous, excessive, data breach waiting to happen. There is absolutely nothing wrong with only keeping billing data beyond say six months max.

Re:Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by samontar on Thursday October 18, 2018 @03:30AM (#18246585)
In California, you soon won’t be able to keep 6 months max. You have to hold a year.
Re:Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by arbie on Thursday October 18, 2018 @02:34AM (#18246388)
> Most of what you describe has little meaning or use at a few months old. Sure you might want to collect detailed awareness of individual drivers over say a month, generic traffic patterns over six months, and little or nothing beyond.

The really invasive pattern-matches happen best retroactively at that scale.

Age of the data does not typically impact many such analyses.

Re:Uber’s Big Data Platform: 100+ Petabytes with Minute Latency by Jedd on Friday October 19, 2018 @10:23PM (#18261372)
> Only if you believe in retaining all data in perpetuity.

There's a huge spectrum between your 'at most 30d' claim in another reply, and 'all data in perpetuity'.

30d is way too short. Perpetuity is probably the idealistic intent, and depending on storage costs over time this may be achievable.

> I suppose it demonstrates the difference in US and European perspectives on data.

How so? I described a the case of one UK-based company, and TFA is about one US-based company, both doing similar things.

> Most of what you describe has little meaning or use at a few months old.

Clearly this is not the case, as evinced by the actions of the people designing, building, operating, and maintaining these types of systems.