jimlongley wrote:Even if they are, that is a mind boggling amount of data that they already, according to his testimony, don't have time to go through. By the time they get the ability to sort through it with any level of efficiency we will all be beyond caring.
A bunch of years ago I was the engineer of a network control center. The center monitored the condition (not the content) of a large (195 nodes) T1 network for the State of NY. We were required, per contract, to record and monitor ALL alarms and events on the network, and investigate each an every one, as well as keeping a database of those events, a raw record of those events, and a separate database of the trouble tickets generated and solutions. The amount of data gathered rapidly became so unmanageable that it was considered a joke to threaten someone on the staff with having to go look for a specific incident in the raw record. On top of that we kept a backup copy at a "geographically diverse location."
As the head engineer, I was charged with the responsibility for ensuring that all of the data was stored and accessible, and I hired a database "expert" to program the access to the raw data as well as sorting and storing it in a useable manner. It quickly became obvious that the state of the art was not up to the task, and the programmer was even behind that. (At one point, shortly before he was moved to another job we noticed that the trouble ticket database was taking huge amounts of time to load, and it turned out that what he had done to handle completed trouble tickets was set a "delete flag" on those tickets, so that when you searched for open tickets they got ignored, but every time you accessed the database ALL of it got loaded. One day my lead computer operator, who was not a programmer herself but had some programming ability, decided to see if she could improve the speed by "packing" the database, and the command she issued deleted all of the data with delete flags set. That was when it was decided that our programmer would be better off in a different job and that it was a good thing to have a separate copy of the database off site. Forgot to mention, one salient thing on his resume was his experience with NSA.
We got a new programmer who understood the troubleshooting process and trouble tickets and things got a little better.
All of this with the State of New York looking over our shoulders and nitpicking.
I eventually quit the job and went back to something more comfortable.
I don't think NSA has the ability, now or in the near future, to process that data, and the amount will continue to grow as they sit on it, so I don't much care what they are keeping of mine.
I think the key in your remarks is "a bunch of years ago." A bunch of years ago system modeling in my industry could take hours; now what took hours is done in a few seconds. What took about 30 minutes five years ago takes about 30 seconds now. And we just have PCs, not supercomputers. He's been out of the NSA for over 10 years and there have been major advances in computing ability during that time. And in the context he's talking about search time is not a critical factor. He's not talking about real-time monitoring but looking back through stored data for either legitimate criminal or illegitimate and nefarious politically motivated investigations. In either case it doesn't really matter if it takes an hour, a day, a week, or a month to pull from the database.
Also, this article doesn't address it, but the original court testimony he referred to also included testimony from an AT&T technician about the interface alluded to in the article. They're not just collecting emails, they're collecting everything...URL's visited, streaming audio and video (not the URL, the actual stream), online chats, online phone conversations, purchase data.....everything.