Bahamas: https://firstlook.org/theintercept/article/2014/05/19/data-p...
Afghanistan: https://wikileaks.org/WikiLeaks-statement-on-the-mass.html
The average call length is 1.8 minutes right now [2]. So we've got 240 mil person * 5 calls / person-day * 365 days * 1.8 min / call. So, about 788 billion minutes of calls / year at a flow rate of about 2 billion calls / day.
At MP3 compression of 128kbps, vocal data takes about 0.94 MB / minute [3]. So, close to 688 petabytes of storage data at a rate of about 1.9 petabytes / day. Seems within the realm of doable.
The problem of analysing this is in the "ridiculous parallelism" category, so they'd just be constrained by server farm capacity. Lets say they had a system with a conservative million nodes. Each day, each node would have to process ~2 GB of audio data looking for patterns. Not even challenging. If I were clever, I'd probably run a brute force audio to text on each node, then a text to symbol pattern analyser. I'd also have higher level net processes that look for patterns in calls spatially and temporally, but with far less processors.
[1] http://www.pewinternet.org/2010/09/02/cell-phones-and-americ...
[2] http://www.statista.com/statistics/185828/average-local-mobi...
[3] http://iaudiophile.net/forums/archive/index.php/t-2081.html
So without any non-COTS tech and without sacrificing the lilt of grandma's sparkly voice, we're already at a 90% reduction of your numbers.
So let's go with 190 TB/day. Say we keep 30 days in hot disk storage, and spool it off to a digital tape robot afterward. We'd need a few thousand COTS hard drives on less than a thousand servers (triple it for decent RAID).
Then a few million-dollar LTO tape libraries, and a couple of guys to schlep tape cartridges all day long.
(Of course, since we don't require all that compute power, and because Gov't doesn't build datacenters like Google does, they'd just buy a few fat IBM z/OS boxes, and a few fat EMC cabinets for low $10's of millions total, and call it a day.)
Either way, it's well within the realm of eminently achievable, I'd say. Which means it's obviously happening. :)
On the final point, agree. This is in the range where its cost is round off error in some of the large security budgets. (+ probably far more for the labor, ops, ect...)
That was one of the main reasons I worked through it, because it sounds like a horrifically large task, but its really not even that crazy in terms of data rates and storage when you break it down.
And did you seriously make a link to cite that 128 kb * 60 = 0.94 MB?
US telecommunications is geographically disperse, and my original point anyway was that the volume of calls is enormous - storing and processing that much data is hardly a trivial problem since it's both storage AND CPU intensive. From the US black ops budget you could make some reasonable estimates as to the total size and capability of the NSA to do this.
Satellite calls have been intercepted worldwide for decades now by numerous intelligence agencies.
It definitely seems like he meant that 80% of calls are recorded and stored. Whether that's true or not will (hopefully) come out in time.
the libertarian in me is greatly annoyed they can do this.
Aren't cats notoriously good at surviving in the wild? Both feral and domestic cats (if they have to).
The NSA's own BLARNEY initiative for this stuff dates back to 1978, although I think GCHQ beat them to the punch in effective mass telephony intercept - the first one I'm aware of started in the early 1980s, although that is offline, and probably no longer archived as the media (digital tape with low-bitrate encoded audio) would probably have degraded beyond all usefulness by now (and it sounded pretty crunchy in the first place, I understand, kinda like a bad squelchy lower-side-band transmission!).
Remember, however, that they didn't have the same kind of analysis capability back then that they've got these days (but they can probably go back and analyse old stuff they still retain). As Snowden's disclosed, you're seeing the newer systems having full-take ring buffers in nearline storage, followed by offline selection for recent access and analysis from that using a huge amount of distributed processing, and in turn automatic selected archiving out of that. It's pretty much the difference between microfiche and Google in effectiveness.
http://blog.archive.org/2013/06/15/cost-to-store-all-us-phon...
The Utah facility gives them the ability to store calls for longer, it doesn't increase their scope.
The reason they don't record 20% of calls is, from my understanding, that they have internal rules against recording certain people important to the democratic process within the US (e.g. politicians (and their families), judges, election officials, etc).
Essentially the politicians were concerned that the NSA's resources would be utilised by whichever party was in office to gain an advantage, so they put themselves (both parties) onto a "no spy" list. But you won't find the law where it says that, because it is all internal-rules created by committee behind closed doors.
I'd love to hear how you came to your conclusion that only politicians, et. al. enjoy such protections.
What exactly used to be true that is no longer true?
That's what you said that is no longer true.
"The FISA Court (FISC) today released a heavily redacted version of its July ruling approving the renewal of the bulk metadata collection on all phone calls from US phone providers under Section 215 of the Patriot Act. "
Then there is the "3 hops" rule that allows full spying of individuals if you are "3 hops" from a suspected terrorist, which is most people alive.
I'm not saying whether or not I support any of this, but I do acknowledge that the NSA was not allowed to spy on US citizens before the patriot act and everything we know indicates that they strictly followed this rule, but things have changed.
I think he meant that the NSA had access to the 80% of calls that are routed through the US, not the the NSA is recording and storing literally every single one of them. I think he was misquoted or misspoke.
William Binney hasn't worked for the NSA since 2001. Were they recording all calls back then? Did someone still there leak new information to him?