Be careful with those emails

[Rex B]

The NSA is collecting and storing all emails for the last 10 years (so far) in a $2 billion data facility. Originally intended to find terrorists, it may now be used to trap political enemies. The writer thinks that's what happened to Petraeus and Gen Allen.

http://rt.com/usa/news/surveillance-spy ... izens-178/" onclick="window.open(this.href);return false;

[jimlongley]

Even if they are, that is a mind boggling amount of data that they already, according to his testimony, don't have time to go through. By the time they get the ability to sort through it with any level of efficiency we will all be beyond caring.

A bunch of years ago I was the engineer of a network control center. The center monitored the condition (not the content) of a large (195 nodes) T1 network for the State of NY. We were required, per contract, to record and monitor ALL alarms and events on the network, and investigate each an every one, as well as keeping a database of those events, a raw record of those events, and a separate database of the trouble tickets generated and solutions. The amount of data gathered rapidly became so unmanageable that it was considered a joke to threaten someone on the staff with having to go look for a specific incident in the raw record. On top of that we kept a backup copy at a "geographically diverse location."

As the head engineer, I was charged with the responsibility for ensuring that all of the data was stored and accessible, and I hired a database "expert" to program the access to the raw data as well as sorting and storing it in a useable manner. It quickly became obvious that the state of the art was not up to the task, and the programmer was even behind that. (At one point, shortly before he was moved to another job we noticed that the trouble ticket database was taking huge amounts of time to load, and it turned out that what he had done to handle completed trouble tickets was set a "delete flag" on those tickets, so that when you searched for open tickets they got ignored, but every time you accessed the database ALL of it got loaded. One day my lead computer operator, who was not a programmer herself but had some programming ability, decided to see if she could improve the speed by "packing" the database, and the command she issued deleted all of the data with delete flags set. That was when it was decided that our programmer would be better off in a different job and that it was a good thing to have a separate copy of the database off site. Forgot to mention, one salient thing on his resume was his experience with NSA.

We got a new programmer who understood the troubleshooting process and trouble tickets and things got a little better.

All of this with the State of New York looking over our shoulders and nitpicking.

I eventually quit the job and went back to something more comfortable.

I don't think NSA has the ability, now or in the near future, to process that data, and the amount will continue to grow as they sit on it, so I don't much care what they are keeping of mine.

[n5wd]

The NSA is collecting and storing all emails for the last 10 years (so far) in a $2 billion data facility. Originally intended to find terrorists, it may now be used to trap political enemies. The writer thinks that's what happened to Petraeus and Gen Allen.

http://rt.com/usa/news/surveillance-spy ... izens-178/" onclick="window.open(this.href);return false;

Just wondering.. you DO realize that's an official media outlet of Putin's Russian government that you're quoting, don't you?

[RPB]

Even if they are, that is a mind boggling amount of data that they already, according to his testimony, don't have time to go through. By the time they get the ability to sort through it with any level of efficiency we will all be beyond caring.

A bunch of years ago I was the engineer of a network control center. The center monitored the condition (not the content) of a large (195 nodes) T1 network for the State of NY. We were required, per contract, to record and monitor ALL alarms and events on the network, and investigate each an every one, as well as keeping a database of those events, a raw record of those events, and a separate database of the trouble tickets generated and solutions. The amount of data gathered rapidly became so unmanageable that it was considered a joke to threaten someone on the staff with having to go look for a specific incident in the raw record. On top of that we kept a backup copy at a "geographically diverse location."

As the head engineer, I was charged with the responsibility for ensuring that all of the data was stored and accessible, and I hired a database "expert" to program the access to the raw data as well as sorting and storing it in a useable manner. It quickly became obvious that the state of the art was not up to the task, and the programmer was even behind that. (At one point, shortly before he was moved to another job we noticed that the trouble ticket database was taking huge amounts of time to load, and it turned out that what he had done to handle completed trouble tickets was set a "delete flag" on those tickets, so that when you searched for open tickets they got ignored, but every time you accessed the database ALL of it got loaded. One day my lead computer operator, who was not a programmer herself but had some programming ability, decided to see if she could improve the speed by "packing" the database, and the command she issued deleted all of the data with delete flags set. That was when it was decided that our programmer would be better off in a different job and that it was a good thing to have a separate copy of the database off site. Forgot to mention, one salient thing on his resume was his experience with NSA.

We got a new programmer who understood the troubleshooting process and trouble tickets and things got a little better.

All of this with the State of New York looking over our shoulders and nitpicking.

I eventually quit the job and went back to something more comfortable.

I don't think NSA has the ability, now or in the near future, to process that data, and the amount will continue to grow as they sit on it, so I don't much care what they are keeping of mine.

collecting and storing all emails for the last 10 years (so far)

I wish they'd sort out my Junk/Spam folder, I get tired of responding to all these E-mails promising to make me two inches taller, I do not need to grow two inches, I'm 6'0" tall already and that's plenty until the osteoporosis gets worse. I'm not looking for an online pharmacy either ...

(My sweet dad, before he passed on, at first used to reply to each and every Junk E-mail, telling them no thank you, not interested at this time etc)

[VMI77]

Even if they are, that is a mind boggling amount of data that they already, according to his testimony, don't have time to go through. By the time they get the ability to sort through it with any level of efficiency we will all be beyond caring.

A bunch of years ago I was the engineer of a network control center. The center monitored the condition (not the content) of a large (195 nodes) T1 network for the State of NY. We were required, per contract, to record and monitor ALL alarms and events on the network, and investigate each an every one, as well as keeping a database of those events, a raw record of those events, and a separate database of the trouble tickets generated and solutions. The amount of data gathered rapidly became so unmanageable that it was considered a joke to threaten someone on the staff with having to go look for a specific incident in the raw record. On top of that we kept a backup copy at a "geographically diverse location."

As the head engineer, I was charged with the responsibility for ensuring that all of the data was stored and accessible, and I hired a database "expert" to program the access to the raw data as well as sorting and storing it in a useable manner. It quickly became obvious that the state of the art was not up to the task, and the programmer was even behind that. (At one point, shortly before he was moved to another job we noticed that the trouble ticket database was taking huge amounts of time to load, and it turned out that what he had done to handle completed trouble tickets was set a "delete flag" on those tickets, so that when you searched for open tickets they got ignored, but every time you accessed the database ALL of it got loaded. One day my lead computer operator, who was not a programmer herself but had some programming ability, decided to see if she could improve the speed by "packing" the database, and the command she issued deleted all of the data with delete flags set. That was when it was decided that our programmer would be better off in a different job and that it was a good thing to have a separate copy of the database off site. Forgot to mention, one salient thing on his resume was his experience with NSA.

We got a new programmer who understood the troubleshooting process and trouble tickets and things got a little better.

All of this with the State of New York looking over our shoulders and nitpicking.

I eventually quit the job and went back to something more comfortable.

I don't think NSA has the ability, now or in the near future, to process that data, and the amount will continue to grow as they sit on it, so I don't much care what they are keeping of mine.

I think the key in your remarks is "a bunch of years ago." A bunch of years ago system modeling in my industry could take hours; now what took hours is done in a few seconds. What took about 30 minutes five years ago takes about 30 seconds now. And we just have PCs, not supercomputers. He's been out of the NSA for over 10 years and there have been major advances in computing ability during that time. And in the context he's talking about search time is not a critical factor. He's not talking about real-time monitoring but looking back through stored data for either legitimate criminal or illegitimate and nefarious politically motivated investigations. In either case it doesn't really matter if it takes an hour, a day, a week, or a month to pull from the database.

Also, this article doesn't address it, but the original court testimony he referred to also included testimony from an AT&T technician about the interface alluded to in the article. They're not just collecting emails, they're collecting everything...URL's visited, streaming audio and video (not the URL, the actual stream), online chats, online phone conversations, purchase data.....everything.

[mamabearCali]

Just confirms what I think we all knew in our gut. You know maybe a nice cut to the NSA budget is in order.

[Dragonfighter]

<SNIP>

I think the key in your remarks is "a bunch of years ago." A bunch of years ago system modeling in my industry could take hours; now what took hours is done in a few seconds. What took about 30 minutes five years ago takes about 30 seconds now. And we just have PCs, not supercomputers. He's been out of the NSA for over 10 years and there have been major advances in computing ability during that time. And in the context he's talking about search time is not a critical factor. He's not talking about real-time monitoring but looking back through stored data for either legitimate criminal or illegitimate and nefarious politically motivated investigations. In either case it doesn't really matter if it takes an hour, a day, a week, or a month to pull from the database.

Also, this article doesn't address it, but the original court testimony he referred to also included testimony from an AT&T technician about the interface alluded to in the article. They're not just collecting emails, they're collecting everything...URL's visited, streaming audio and video (not the URL, the actual stream), online chats, online phone conversations, purchase data.....everything.

For years, and I mean yeeeaars, phone traffic has been monitored to flag certain "keywords". All an agency has to do is set a combination of keywords they are looking for and the system will filter millions of emails in a matter of a few hours. So here is the scenario: You are looking for fundamentalist gun owners and you set your filter this particular day to "weapons, 2a, constitutional, (any number of makers' brand names), etc. So you get a few million emails either sent, forwarded or replied to that discuss the 2A in conjunction with weapons and bingo, a list of possible dissidents. The "agency" now simply reconfigures for each group of "inconvenient citizens". They have the data, all they need to do is sift it. And since they are allowed to gather the data without due process, then what prevents some Chicago mobster turned attorney general from sorting out and then setting about the squelching of anyone in their way. The only possible solution is to have that data base eradicated and cause for data collection very narrowly tailored to actual threat profiles.

[Jumping Frog]

There will come a day when this government, too, will have to be removed by force. Breaks my heart.

[jimlongley]

Even if they are, that is a mind boggling amount of data that they already, according to his testimony, don't have time to go through. By the time they get the ability to sort through it with any level of efficiency we will all be beyond caring.

A bunch of years ago I was the engineer of a network control center. The center monitored the condition (not the content) of a large (195 nodes) T1 network for the State of NY. We were required, per contract, to record and monitor ALL alarms and events on the network, and investigate each an every one, as well as keeping a database of those events, a raw record of those events, and a separate database of the trouble tickets generated and solutions. The amount of data gathered rapidly became so unmanageable that it was considered a joke to threaten someone on the staff with having to go look for a specific incident in the raw record. On top of that we kept a backup copy at a "geographically diverse location."

As the head engineer, I was charged with the responsibility for ensuring that all of the data was stored and accessible, and I hired a database "expert" to program the access to the raw data as well as sorting and storing it in a useable manner. It quickly became obvious that the state of the art was not up to the task, and the programmer was even behind that. (At one point, shortly before he was moved to another job we noticed that the trouble ticket database was taking huge amounts of time to load, and it turned out that what he had done to handle completed trouble tickets was set a "delete flag" on those tickets, so that when you searched for open tickets they got ignored, but every time you accessed the database ALL of it got loaded. One day my lead computer operator, who was not a programmer herself but had some programming ability, decided to see if she could improve the speed by "packing" the database, and the command she issued deleted all of the data with delete flags set. That was when it was decided that our programmer would be better off in a different job and that it was a good thing to have a separate copy of the database off site. Forgot to mention, one salient thing on his resume was his experience with NSA.

We got a new programmer who understood the troubleshooting process and trouble tickets and things got a little better.

All of this with the State of New York looking over our shoulders and nitpicking.

I eventually quit the job and went back to something more comfortable.

I don't think NSA has the ability, now or in the near future, to process that data, and the amount will continue to grow as they sit on it, so I don't much care what they are keeping of mine.

I think the key in your remarks is "a bunch of years ago." A bunch of years ago system modeling in my industry could take hours; now what took hours is done in a few seconds. What took about 30 minutes five years ago takes about 30 seconds now. And we just have PCs, not supercomputers. He's been out of the NSA for over 10 years and there have been major advances in computing ability during that time. And in the context he's talking about search time is not a critical factor. He's not talking about real-time monitoring but looking back through stored data for either legitimate criminal or illegitimate and nefarious politically motivated investigations. In either case it doesn't really matter if it takes an hour, a day, a week, or a month to pull from the database.

Also, this article doesn't address it, but the original court testimony he referred to also included testimony from an AT&T technician about the interface alluded to in the article. They're not just collecting emails, they're collecting everything...URL's visited, streaming audio and video (not the URL, the actual stream), online chats, online phone conversations, purchase data.....everything.

Actually, it was 20+ years ago and T1 was the "ne plus ultra" for networks, but the higher speeds have been offset by higher density of traffic and content. And I realize he's been out of NSA for more than 10 years, and I believe the same applies there.

For years, and I mean yeeeaars, phone traffic has been monitored to flag certain "keywords". All an agency has to do is set a combination of keywords they are looking for and the system will filter millions of emails in a matter of a few hours. So here is the scenario: You are looking for fundamentalist gun owners and you set your filter this particular day to "weapons, 2a, constitutional, (any number of makers' brand names), etc. So you get a few million emails either sent, forwarded or replied to that discuss the 2A in conjunction with weapons and bingo, a list of possible dissidents. The "agency" now simply reconfigures for each group of "inconvenient citizens". They have the data, all they need to do is sift it. And since they are allowed to gather the data without due process, then what prevents some Chicago mobster turned attorney general from sorting out and then setting about the squelching of anyone in their way. The only possible solution is to have that data base eradicated and cause for data collection very narrowly tailored to actual threat profiles.

Sorry, I don't believe that totally, there are just too many competing digital formats, not to mention analog, to do that in real time, which is what monitoring is. If it's being post processed, which using a database implies, then it is not monitoring.

I do believe that monitoring has taken place in some circumstances, I have even participated in both monitoring and failed attempts at monitoring. I actually had a lot of fun monitoring a lottery network back then, we were watching for errors in the lines, much easier to detect than specific words or word patterns, and my partner and myself actually figured out not only the polling sequences and addresses of stations on the network, but could even tell the difference, at a glance, between a quick pick ticket for one game and a selected ticket with choices penciled in, and even what the choices were, as well as which station originated the ticket.

We were sitting there running a test overseen by the State of NY and the computer company, and as Rich and I started saying what kind of ticket was being sent from which machine, the computer company person went ballistic on us, insisting on an immediate meeting with us, the State, and a variety of other players including a contracted security team. The security team basically laughed at them after they explained what the problem was, telling them that hours and hours of repetition watching those lines under test would have resulted in an easy education for anyone smart enough to be able to operate the sophisticated equipment we were using to monitor the lines to begin with. End of meeting, but then a "secrecy oath" to be signed so that we would not "expose" the secrets we had discovered.

------

But if I moved up a couple of levels, or down as the case may be, in the protocols stacks, I not only would have to be able to sort by station address, but which of several T1 "channels" that station was transmitting on, and then which of several available T1s in a T3, and so on, adding levels of complexity until the current state of the art. And that's just in the digital domain without any encryption, if it's analog, then you have to get it all the way back to analog in order to be able to tell what was said.

As a telephone engineer I participated in attempts to automatically monitor voice conversations, and the efforts were laughable. Look at voice recognition menus today, and how many errors are generated. An agency relying on real time monitoring of analog telephone traffic looking for specific keywords had better hope that the person they were monitoring spoke with no accent, at an even rate, and did not have a cold.

Do I believe "they" are trying to do it? Yes, most absolutely. But do I believe they are succeeding, not anywhere nearly to the extent that movies and TV shows would have us believe. And as I have said, the amount of digital data out there is just monstrously huge, and post processing that in all of its various formats is not a logistical problem I would like to even attempt, and I have troubleshot IBM SNA networks using a protocol analyzer that did not decode SNA.

[VMI77]

<SNIP>

I think the key in your remarks is "a bunch of years ago." A bunch of years ago system modeling in my industry could take hours; now what took hours is done in a few seconds. What took about 30 minutes five years ago takes about 30 seconds now. And we just have PCs, not supercomputers. He's been out of the NSA for over 10 years and there have been major advances in computing ability during that time. And in the context he's talking about search time is not a critical factor. He's not talking about real-time monitoring but looking back through stored data for either legitimate criminal or illegitimate and nefarious politically motivated investigations. In either case it doesn't really matter if it takes an hour, a day, a week, or a month to pull from the database.

Also, this article doesn't address it, but the original court testimony he referred to also included testimony from an AT&T technician about the interface alluded to in the article. They're not just collecting emails, they're collecting everything...URL's visited, streaming audio and video (not the URL, the actual stream), online chats, online phone conversations, purchase data.....everything.

For years, and I mean yeeeaars, phone traffic has been monitored to flag certain "keywords". All an agency has to do is set a combination of keywords they are looking for and the system will filter millions of emails in a matter of a few hours. So here is the scenario: You are looking for fundamentalist gun owners and you set your filter this particular day to "weapons, 2a, constitutional, (any number of makers' brand names), etc. So you get a few million emails either sent, forwarded or replied to that discuss the 2A in conjunction with weapons and bingo, a list of possible dissidents. The "agency" now simply reconfigures for each group of "inconvenient citizens". They have the data, all they need to do is sift it. And since they are allowed to gather the data without due process, then what prevents some Chicago mobster turned attorney general from sorting out and then setting about the squelching of anyone in their way. The only possible solution is to have that data base eradicated and cause for data collection very narrowly tailored to actual threat profiles.

It's worse than that....since with stored data they have the advantage of hindsight they can do things like pick a brand new target and diagram his entire social network; pick out messages or data made in one social context and read them into the current social context; and they can ferret out material for blackmail as was done with Petraeus and Allen. They don't need to filter anything in real time --which at least limits searches to whatever is under consideration at the time of the search-- and when they think of new search terms or connections they can keep going back to the database and pulling out new "enemies." And they can construct all kinds of lists from this data...not just identify "dissidents." For instance, how many people here post messages on their latest gun purchase, what they shot at the range last week, or tell friends and relatives about a new gun, or about any of their guns, bulk ammo buys, accessories like scopes and magazines, etc. I'd bet with ten years worth of email data they can determine pretty accurately what every person on this board owns in the way of guns, whether they have "full" capacity magazines, night vision equipment, and bulk quantities of ammo.

[VMI77]

Do I believe "they" are trying to do it? Yes, most absolutely. But do I believe they are succeeding, not anywhere nearly to the extent that movies and TV shows would have us believe. And as I have said, the amount of digital data out there is just monstrously huge, and post processing that in all of its various formats is not a logistical problem I would like to even attempt, and I have troubleshot IBM SNA networks using a protocol analyzer that did not decode SNA.

I'm not addressing real-time monitoring....I think that is fairly limited and specifically targeted. Also, I don't believe some of the capabilities displayed in movies and on TV exist yet --for instance, the ability to gather images from surveillance cameras all over the world and track targets in real time, or for that matter, just to instantaneously get individual video feeds feeds from surveillance cameras all over the world. But even free browser add ons have some amazing search capabilities today....for instance, searching the web for identical images --e.g. "who stole my photos" and the like. I've used them to find original images people are trying to pass off on their own. I don't know how comprehensive they are but they're pretty fast. I think search algorithms have become very fast and sophisticated. I see evidence of it in software available to me for free, so I can't help but think the NSA has software that is way better.

[ghostrider]

The NSA is collecting and storing all emails for the last 10 years (so far) in a $2 billion data facility

I'd be surprised if that's all that was spent. Canada spent that just trying to register long guns in a country with only 33 million people. Just think of how many email messages are sent daily in the US alone.

http://royal.pingdom.com/2011/01/12/int ... n-numbers/" onclick="window.open(this.href);return false;

[comp73]

Not trying to fan the conspiracy flames, but if some GUY can do this with 5 mid range servers....

http://www.zdnet.com/25-gpus-devour-pas ... 000008368/" onclick="window.open(this.href);return false;

Or you can consider the Air Force with their PS3's....

http://www.popsci.com/technology/articl ... station-3s" onclick="window.open(this.href);return false;

While I would agree that the NSA only has the ability to real time track specific targets, they are also reported to have the largest concentration of supercomputers in the world. I expect it is just a matter of time before they have the capability of real time tracking everything, which leads to the question, then what?

[sawdust]

There will come a day when this government, too, will have to be removed by force. Breaks my heart.

Methinks you have just become a priority

Also, how far-fetched is the "Person of Interest" tv show? Really.

Technological innovation is now the driving force in probably all aspects of societal evolution.
Example 1: There were helicopters- slow, cumbersome, limited capacity and limited utility. Then someone had the idea of putting guns on them and they became weapons platforms. And there were drones- radio-controlled aircraft, used primarily as targets. Then someone had the idea of creating a radio-controlled helicopter, small enough to put on Navy destroyers, and loading them with uhhhh... weapons of mass destruction to use against enemy subs. Then someone had the thought, "Well, really, there isn't that much of a sub threat in the Gulf of Tonkin, so what if we put video cameras on those things and use them for reconnaissance. Today, we have high-flying, long-range, gps-guided, missile-toting, hi-res camera-equipped, radio-controlled aircraft. We also have super-quiet, camera-equipped micro-drones that can linger un-noticed to monitor an unlimited array of activities, even inside of buildings.

Example 2: The first personal computer was a kit, of interest only to gadgeteering technophiles. Then someone had the idea of putting a keyboard on and making a whole bunch of computers in their garage. Then someone had the thought, "Hmmm, maybe something like this could be put to use in business- if only we could figure out how to do it." They figured it out. Then, someone else figured out that data could be sent from one computer to another, even over telephone lines, and something called Tel-Net was created, but its use was limited primarily to university regimes. Well, maybe there was a bit of government interest also. "Well," someone thought, "If professors can do it, why can't everyone else?" So, today, we have the Internet, which in its various aspects, controls, dictates, directs, monitors, and oversees almost all of our daily life.

What's next? So, is "Person of Interest" really so far-fetched?

Having said all of the fore-going, I do not fear technology (it has saved my life more than once), I do have serious doubts about the ******* that crave to control it, and by extension, me. Quote from my best-selling book: " It's not that I resist authority; however, I do tend to resist petty authority and tyrannical authority... I do not willingly concede to others the authority to tell me what color that I may paint my house." [Rant off?]

It's downright scary.

[bikerbill]

The issue to me isn't that they have all this stored and, lucky us, don't have the time to go through it ... the issue is that if they want to cause you trouble, they can FIND your stuff, find something in it they don't like, and use it against you ... storage of this kind of material is so completely against what this country USED to stand for ... Obama has to be the worst thing that's happened to this country since Pearl Harbor ...