Re:Drug Agents Use Vast Phone Trove Eclipsing N.S.A.’s
by nohuck13 on Monday September 02, 2013 @01:35AM
(#6313585)
Re: theory #1- there are commodity solutions to the queriable data warehouse problem, e.g. Vertica or Greenplum. Looking at these can give you an idea of what might be hard. You've just described Vertica pretty accurately (column-oriented storage, materialised on disk for particular query patterns, ad-hoc deletion comparatively costly). And ageing out in Vertica is easy- you just drop a partition. I can't imagine a custom system would be built without so e provision for this.
[ Parent
| on HN ]
Re:Drug Agents Use Vast Phone Trove Eclipsing N.S.A.’s
by bigiain on Monday September 02, 2013 @02:08AM
(#6313659)
Sure, if you were building it from scratch today, but Vertical was founded in 2005 - the article talks about data going back to 1987. Cynical-me suspects the NSA has been "requiring" them to retain all that data since before it became easy/practical to expire it properly.