Re:Drug Agents Use Vast Phone Trove Eclipsing N.S.A.’s
by dredmorbius on Monday September 02, 2013 @01:09AM
(#6313536)
Because deleting stuff at scale is harder than you think.
If you've got an age-based retention policy, and you've built in the capacity to delete the data, not really. Periodic purge. I'm not saying this is trivial, but it's not hard. The system was clearly designed not to delete the data.
Your theory #2 holds much more water.
Re:Drug Agents Use Vast Phone Trove Eclipsing N.S.A.’s
by bigiain on Monday September 02, 2013 @02:13AM
(#6313680)
The optimist in me wants to argue that there may be OPEX vs CAPEX reasons to have not invested in "building in the capacity to delete", or possibly even tax implications of development budgets vs ongoing maintenance which might reasonably explain why you'd choose to build such a system as "append-only, archive indefinitely". The pessimist in me fears you're correct…
Re:Drug Agents Use Vast Phone Trove Eclipsing N.S.A.’s
by nohuck13 on Monday September 02, 2013 @01:35AM
(#6313585)
Re: theory #1- there are commodity solutions to the queriable data warehouse problem, e.g. Vertica or Greenplum. Looking at these can give you an idea of what might be hard. You've just described Vertica pretty accurately (column-oriented storage, materialised on disk for particular query patterns, ad-hoc deletion comparatively costly). And ageing out in Vertica is easy- you just drop a partition. I can't imagine a custom system would be built without so e provision for this.
Re:Drug Agents Use Vast Phone Trove Eclipsing N.S.A.’s
by bigiain on Monday September 02, 2013 @02:08AM
(#6313659)
Sure, if you were building it from scratch today, but Vertical was founded in 2005 - the article talks about data going back to 1987. Cynical-me suspects the NSA has been "requiring" them to retain all that data since before it became easy/practical to expire it properly.
Theory #2 – The NSA (or it's predecessor or a related agency) have been paying them to store whatever they can since whenever it became possible. If you're prepared to ignore the privacy implications, it's obvious that some _tiny_ percentage of that data will become useful for law enforcement purposes. Unfortunately – when the "privacy implications" are considered in terms of "Is _your_ privacy more important than _my_ career?" – it's pretty clear that a very powerful cohort of law enforcement and intelligence agency decision makers say "Hell no! I just need one or two more big successes and I'll get that promotion I want! Of course it's worth monitoring every single person on the planet to give me a shot at the executive bathroom and an office with a door!" (or, less cynically but equal in consequence "Should I listen to every phone conversation in America, if it might possibly mean we can stop the next 9-11? Yeah, I think it might be…").