This post (AC@Jun 10th, 2014 2:11am) almost certainly is incorrect in several respects.
The NSA apparently has access to a quantity of data, both IP and telephone, that exceeds by orders of magnitude what could be examined by intelligence agencies that collectively employ in the neighborhood of 100,000 people, quite a few of them managers and support staff. The data quantity also exceeds by orders of magnitude what it would be possible to store for any reasonable period, although techniques like deduplication could be used for mitigation. But beyond any doubt they use programmed filters to ignore or discard a great deal of the data to which the have access.
In the case of internet protocol traffic, such programmed filters might reject based on IP addresses, protocol, content type, email address, email body content, or possibly other characteristics. With appropriate equipment, something similar to Wireshark would be useful and automatic. Such devices would be imperfect and can be expected to be configured to err in the direction of overcollection subject to downstream capacity constraints. Manual review might begin at that point or be preceded by more complex automated filters, but eventually data would be available to intelligence analysts, who are required to eliminate "US Person" data that is not subject to legal collection. "US Persons" includes both foreign nationals legally in the US as and US citizens in other countries. Sorting these cases probably has a large manual component and deletions are legally required to be done immediately on recognition. Remaining data may be retained for varying periods specified in law, but remains subject to a legal and FISC-ordered requirement for immediate removal should it later be found to refer to US persons and its retention not authorized as part of an investigation. The same legal and court orders require it to be purged at the end of its lawful retention period.
As many have stated, it probably would be relatively straightforward to suspend the time based purging, but changes to upstream automated and manual minimization procedures might well be both complicated and time consuming to make.
Not quite. The current state appears to be that the government claimed that the temporary restraining order requiring retention of certain data, if applied to FISA Section 702 data, was inconsistent with standing court orders requiring that the data be deleted as soon as possible and retained no longer than a specified period. The court granted the request and scheduled a hearing for argument of the issue by both litigants.
No. Just no. The level of apparent paranoia on this site (more than anything the NSA or its Five Eyes associates have done) suggests a legitimacy problem in the US. (I assume most of the comments are from US residents.)
Some things to keep in mind about the NSA is that its programs were
- designed based on laws passed by the US Congress and signed by the contemporary President - several Congresses and Presidents. - developed by civilian and military government employees and US citizen contractors who, apart from being in the most intelligent tenth or so of the population, are mostly not distinguishable from the population as a whole, in particular in their concern for civil rights; - supervised by NSA management; the NSA legal office, the Inspector Generals of the NSA and DoD; the Attorney General; and a court composed of federal judges appointed by the President with the advice and consent of the Senate and designated for the Foreign Intelligence Surveillance Court by the Chief Justice of the Supreme Court (also appointed by the President with the advice and consent of the Senate). - reported upon regularly to the Intelligence Committees of the Senate and the House of Representatives.
One may take this either as evidence that the entire government is, or is becoming, a tyranny or as evidence that controls are in place that might or might not be considered adequate.
Contrary to often expressed opinion, it is not "obvious" that any of the reported activities violate either the Fourth Amendment or the law. Fourth Amendment law is both voluminous and complex, and skilled attorneys disagree about how to apply it in particular cases. That is what the case in hand is about.
Demonizing the NSA or others in the Executive branch, or judges who may be hearing cases that involve them, is not useful. It also goes against the basic agreement that underlies all representative democratic regimes: that we accept the output of the legislative and legal processes (including the laws against corruption in office) even when we disagree with them.
It is certain that the technical facilities available to the NSA and other intelligence agencies (and a large number of private sector organizations and individuals) have the potential for serious abuse. Depending on point of view there is little evidence for that at present, or none. It is certain as well that it is necessary to keep these intelligence programs and activities under close watch; review their operation often to identify actual, potential, or perceived misuse; and review their internal and external controls to identify potential areas for abuse (or where they might be limiting the legitimate use of a program).
This post (AC@Jun 10th, 2014 2:11am) almost certainly is incorrect in several respects.
The NSA apparently has access to a quantity of data, both IP and telephone, that exceeds by orders of magnitude what could be examined by intelligence agencies that collectively employ in the neighborhood of 100,000 people, quite a few of them managers and support staff. The data quantity also exceeds by orders of magnitude what it would be possible to store for any reasonable period, although techniques like deduplication could be used for mitigation. But beyond any doubt they use programmed filters to ignore or discard a great deal of the data to which the have access.
In the case of internet protocol traffic, such programmed filters might reject based on IP addresses, protocol, content type, email address, email body content, or possibly other characteristics. With appropriate equipment, something similar to Wireshark would be useful and automatic. Such devices would be imperfect and can be expected to be configured to err in the direction of overcollection subject to downstream capacity constraints. Manual review might begin at that point or be preceded by more complex automated filters, but eventually data would be available to intelligence analysts, who are required to eliminate "US Person" data that is not subject to legal collection. "US Persons" includes both foreign nationals legally in the US as and US citizens in other countries. Sorting these cases probably has a large manual component and deletions are legally required to be done immediately on recognition. Remaining data may be retained for varying periods specified in law, but remains subject to a legal and FISC-ordered requirement for immediate removal should it later be found to refer to US persons and its retention not authorized as part of an investigation. The same legal and court orders require it to be purged at the end of its lawful retention period.
As many have stated, it probably would be relatively straightforward to suspend the time based purging, but changes to upstream automated and manual minimization procedures might well be both complicated and time consuming to make.
Not quite. The current state appears to be that the government claimed that the temporary restraining order requiring retention of certain data, if applied to FISA Section 702 data, was inconsistent with standing court orders requiring that the data be deleted as soon as possible and retained no longer than a specified period. The court granted the request and scheduled a hearing for argument of the issue by both litigants.
One of the problems is that it has not been legally determined that the programs, as such, violate the law. Legislators who claim to have been kept in the dark might have an interest in taking that position, just as NSA and other intelligence agency managers may have an interest in justifying the programs. The text of the relevant section of 50 US Code 1861 is somewhat convoluted and grants a lot of authority to the FISC. Most of the legislators who approved it are attorneys, though, and should have been able to spot potential problems - if they or someone on their staffs actually had read it carefully. That some of them now claim to have been bamboozled is evidence enough to retire them at the next election for lack of intelligence or carelessness with the truth.
LOVINT is a particularly inappropriate example of NSA institutional misbehavior. The violators were identified based on NSA's internal controls and procedures or confessed when it appeared likely they would be caught, the cases were handled according to applicable military or civil service procedures and mostly resulted in loss of job, and they were reported to the FISC as required. There are better examples in the incorrect capture of domestic traffic due to incorrect filter implementation and capture of domestic telephone traffic due to incorrect entry of an area code. Like LOVINT violations, neither of these indicates systematic abuse, and both were reported to the FISC as required. All three, however, indicate that internal design and operational controls warrant attention.
The Constitutional right to mount a legal challenge to the predecessor of the Patriot Act Section 215 metadata program appears to be the substance of Jewel v. NSA, which is yet to be decided. The government's brief is part of the process, as is the plaintiff's brief seeking the temporary restraining order from which the government seeks relief.
a. Absent other selectors (such as the IP address associated previously with a known target), both IP addresses indicating a US location would indicate traffic of little interest to an intelligence agency. Spoofing the source address certainly could be useful to obscure the sender's true location, but spoofing the destination, or both, seems unlikely to be useful.
b. Again, unless other selectors apply, capturing the average YouTube stream probably is low priority.
c. The objection seems to presume that commercial search engines and web sites have a rather larger degree of corruption than seems likely.
A back of the envelope calculation suggests that NSA's Utah data center has enough floor space to accommodate storage for a couple of weeks of aggregate internet protocol data. They are pruning the data early and often. They may have a goal of scanning everything, but claims about plans to retain all of it do not compute, especially on an assumption that the intelligence agencies might be interested in something beyond IP traffic, like voice.
No. Just no. The level of apparent paranoia on this site (more than anything the NSA or its Five Eyes associates have done) suggests a legitimacy problem in the US. (I assume most of the comments are from US residents.)
Some things to keep in mind about the NSA is that its programs were
- designed based on laws passed by the US Congress and signed by the contemporary President - several Congresses and Presidents. - developed by civilian and military government employees and US citizen contractors who, apart from being in the most intelligent tenth or so of the population, are mostly not distinguishable from the population as a whole, in particular in their concern for civil rights; - supervised by NSA management; the NSA legal office, the Inspector Generals of the NSA and DoD; the Attorney General; and a court composed of federal judges appointed by the President with the advice and consent of the Senate and designated for the Foreign Intelligence Surveillance Court by the Chief Justice of the Supreme Court (also appointed by the President with the advice and consent of the Senate). - reported upon regularly to the Intelligence Committees of the Senate and the House of Representatives.
One may take this either as evidence that the entire government is, or is becoming, a tyranny or as evidence that controls are in place that might or might not be considered adequate.
Contrary to often expressed opinion, it is not "obvious" that any of the reported activities violate either the Fourth Amendment or the law. Fourth Amendment law is both voluminous and complex, and skilled attorneys disagree about how to apply it in particular cases. That is what the case in hand is about.
Demonizing the NSA or others in the Executive branch, or judges who may be hearing cases that involve them, is not useful. It also goes against the basic agreement that underlies all representative democratic regimes: that we accept the output of the legislative and legal processes (including the laws against corruption in office) even when we disagree with them.
It is certain that the technical facilities available to the NSA and other intelligence agencies (and a large number of private sector organizations and individuals) have the potential for serious abuse. Depending on point of view there is little evidence for that at present, or none. It is certain as well that it is necessary to keep these intelligence programs and activities under close watch; review their operation often to identify actual, potential, or perceived misuse; and review their internal and external controls to identify potential areas for abuse (or where they might be limiting the legitimate use of a program).
Many database managers have a timestamp facility, and it is often used in application databases. It is likely that NSA database designers are aware of this and use it where that makes application sense.
On the other hand, it is probable, and in agreement with the published NSA slides, that a large part of the data that is acquired, for some meaning of "acquired", is discarded almost immediately, for example:
a. because both source and destination IP addresses are in the U. S.;
b. because it clearly is noise (e. g., cat videos not to or from a targeted area);
c. because it is a search for, or search result from, a query of no intelligence interest (e. g., my search for information about domestic dishwashers).
Such information normally would not be placed in a database, and the claim in the governments brief that significant changes could be required to retain it is reasonably plausible, although possibly somewhat overstated.
"[A]ccording the constitution the NSA is not supposed to record communications without a warrant."
Different rules apply to different kinds of communication. Telephone content is legally different from telephone company business records (the "metadata"), for example. The issue in this article appears to be mostly or entirely Section 702 collections, which are foreign targeted and for which warrants may not be required under the existing minimization rules for handling incidental capture of domestic data and data pertaining to U. S. citizens abroad.
"They are clearly already operating in breach of the law." Actually, they are not. That is precisely the controversy in front of the court.
The poster plainly is unfamiliar with the history of the FISA and its various reauthorizations and amendments, or with the general contents of Executive Order 12333. As I noted earlier, it is possible that the laws and order are being violated, but no real evidence of that seems to be available.
This is one of the few technically informed comments in this thread, or indeed most of those I have seen on anything to do with signals intelligence in the last year. The published briefing charts seem to describe multi-tier filtering in which nearly all of the data is discarded in the initial collection phase, and the remainder is later matched and merged with other data one or more steps later.
All things considered, it might not be completely unreasonable to argue that this does not constitute "spying" or even "collection" to the extent this is done by machine, for example based on comparison of source/destination IP address with known network structure. It would also, of course, be reasonable to argue that any agency that has a sniffer on the internet trunks needs very careful supervision, and that certainly would include the NSA, GCHQ, CSEC, ASD, and GCSB, as well as all employees of the companies who maintain the communication facilities.
The NSA has had minimization rules very similar to the present ones in place for about 40 years, since before the Church Committee hearings in late 1975.
You might wish to argue that they did not follow them, but there seems to be no evidence for such a claim other than a few NSA employees who were punished or dismissed for personal use of some of the data.
What data in which Section 792 database can be identified by the search argument "Jewel" might not be a useful question. In fact, such a search might well return no results at all, any more than it would if applied to the Section 215 telephone metadata. In the latter case the relevant data probably would be found by searching the data for each telephone number that "Jewel" might have used during the period in question. That might not get them all (e. g., calls made from a public or other telephone) and might fetch back a few calls made by others. Most of the data collected initially will not contain much personal identification data, if any, and will not be useful to anyone until and unless merged with other data that does; that is likely to be nontrivial and not done on a uniform basis.
Certainly NSA has backups, but might have very limited or costly search mechanisms. They also could, as someone mentioned, snapshot the database; depending on the storage details that could present some difficulties, mostly a matter of resources. The technical justification is more than a tiny bit disingenuous.
However, the minimization rules I have seen in the publicly available documents require that information about U. S. persons be purged immediately when recognized as such. Changing existing procedures for that would, indeed, force that agency into noncompliance with its own internal controls, FISC orders, and the law under which the program is authorized, as the brief argues, in part.
Stipulating that the statement is true that the Section 702 collections that are the primary or exclusive target of the brief comprise multiple interrelated databases, and admitting the possibility that NSA has decent software development practices, several months may not be unreasonable for implementing the implied changes.
Techdirt has not posted any stories submitted by tddial.
Re: Re: Depends on their system design
The NSA apparently has access to a quantity of data, both IP and telephone, that exceeds by orders of magnitude what could be examined by intelligence agencies that collectively employ in the neighborhood of 100,000 people, quite a few of them managers and support staff. The data quantity also exceeds by orders of magnitude what it would be possible to store for any reasonable period, although techniques like deduplication could be used for mitigation. But beyond any doubt they use programmed filters to ignore or discard a great deal of the data to which the have access.
In the case of internet protocol traffic, such programmed filters might reject based on IP addresses, protocol, content type, email address, email body content, or possibly other characteristics. With appropriate equipment, something similar to Wireshark would be useful and automatic. Such devices would be imperfect and can be expected to be configured to err in the direction of overcollection subject to downstream capacity constraints. Manual review might begin at that point or be preceded by more complex automated filters, but eventually data would be available to intelligence analysts, who are required to eliminate "US Person" data that is not subject to legal collection. "US Persons" includes both foreign nationals legally in the US as and US citizens in other countries. Sorting these cases probably has a large manual component and deletions are legally required to be done immediately on recognition. Remaining data may be retained for varying periods specified in law, but remains subject to a legal and FISC-ordered requirement for immediate removal should it later be found to refer to US persons and its retention not authorized as part of an investigation. The same legal and court orders require it to be purged at the end of its lawful retention period.
As many have stated, it probably would be relatively straightforward to suspend the time based purging, but changes to upstream automated and manual minimization procedures might well be both complicated and time consuming to make.
Re: Re:
(untitled comment)
Some things to keep in mind about the NSA is that its programs were
- designed based on laws passed by the US Congress and signed by the contemporary President - several Congresses and Presidents.
- developed by civilian and military government employees and US citizen contractors who, apart from being in the most intelligent tenth or so of the population, are mostly not distinguishable from the population as a whole, in particular in their concern for civil rights;
- supervised by NSA management; the NSA legal office, the Inspector Generals of the NSA and DoD; the Attorney General; and a court composed of federal judges appointed by the President with the advice and consent of the Senate and designated for the Foreign Intelligence Surveillance Court by the Chief Justice of the Supreme Court (also appointed by the President with the advice and consent of the Senate).
- reported upon regularly to the Intelligence Committees of the Senate and the House of Representatives.
One may take this either as evidence that the entire government is, or is becoming, a tyranny or as evidence that controls are in place that might or might not be considered adequate.
Contrary to often expressed opinion, it is not "obvious" that any of the reported activities violate either the Fourth Amendment or the law. Fourth Amendment law is both voluminous and complex, and skilled attorneys disagree about how to apply it in particular cases. That is what the case in hand is about.
Demonizing the NSA or others in the Executive branch, or judges who may be hearing cases that involve them, is not useful. It also goes against the basic agreement that underlies all representative democratic regimes: that we accept the output of the legislative and legal processes (including the laws against corruption in office) even when we disagree with them.
It is certain that the technical facilities available to the NSA and other intelligence agencies (and a large number of private sector organizations and individuals) have the potential for serious abuse. Depending on point of view there is little evidence for that at present, or none. It is certain as well that it is necessary to keep these intelligence programs and activities under close watch; review their operation often to identify actual, potential, or perceived misuse; and review their internal and external controls to identify potential areas for abuse (or where they might be limiting the legitimate use of a program).
Re: Re: Depends on their system design
The NSA apparently has access to a quantity of data, both IP and telephone, that exceeds by orders of magnitude what could be examined by intelligence agencies that collectively employ in the neighborhood of 100,000 people, quite a few of them managers and support staff. The data quantity also exceeds by orders of magnitude what it would be possible to store for any reasonable period, although techniques like deduplication could be used for mitigation. But beyond any doubt they use programmed filters to ignore or discard a great deal of the data to which the have access.
In the case of internet protocol traffic, such programmed filters might reject based on IP addresses, protocol, content type, email address, email body content, or possibly other characteristics. With appropriate equipment, something similar to Wireshark would be useful and automatic. Such devices would be imperfect and can be expected to be configured to err in the direction of overcollection subject to downstream capacity constraints. Manual review might begin at that point or be preceded by more complex automated filters, but eventually data would be available to intelligence analysts, who are required to eliminate "US Person" data that is not subject to legal collection. "US Persons" includes both foreign nationals legally in the US as and US citizens in other countries. Sorting these cases probably has a large manual component and deletions are legally required to be done immediately on recognition. Remaining data may be retained for varying periods specified in law, but remains subject to a legal and FISC-ordered requirement for immediate removal should it later be found to refer to US persons and its retention not authorized as part of an investigation. The same legal and court orders require it to be purged at the end of its lawful retention period.
As many have stated, it probably would be relatively straightforward to suspend the time based purging, but changes to upstream automated and manual minimization procedures might well be both complicated and time consuming to make.
Re: Re:
(untitled comment)
LOVINT is a particularly inappropriate example of NSA institutional misbehavior. The violators were identified based on NSA's internal controls and procedures or confessed when it appeared likely they would be caught, the cases were handled according to applicable military or civil service procedures and mostly resulted in loss of job, and they were reported to the FISC as required. There are better examples in the incorrect capture of domestic traffic due to incorrect filter implementation and capture of domestic telephone traffic due to incorrect entry of an area code. Like LOVINT violations, neither of these indicates systematic abuse, and both were reported to the FISC as required. All three, however, indicate that internal design and operational controls warrant attention.
The Constitutional right to mount a legal challenge to the predecessor of the Patriot Act Section 215 metadata program appears to be the substance of Jewel v. NSA, which is yet to be decided. The government's brief is part of the process, as is the plaintiff's brief seeking the temporary restraining order from which the government seeks relief.
(untitled comment)
b. Again, unless other selectors apply, capturing the average YouTube stream probably is low priority.
c. The objection seems to presume that commercial search engines and web sites have a rather larger degree of corruption than seems likely.
A back of the envelope calculation suggests that NSA's Utah data center has enough floor space to accommodate storage for a couple of weeks of aggregate internet protocol data. They are pruning the data early and often. They may have a goal of scanning everything, but claims about plans to retain all of it do not compute, especially on an assumption that the intelligence agencies might be interested in something beyond IP traffic, like voice.
(untitled comment)
Some things to keep in mind about the NSA is that its programs were
- designed based on laws passed by the US Congress and signed by the contemporary President - several Congresses and Presidents.
- developed by civilian and military government employees and US citizen contractors who, apart from being in the most intelligent tenth or so of the population, are mostly not distinguishable from the population as a whole, in particular in their concern for civil rights;
- supervised by NSA management; the NSA legal office, the Inspector Generals of the NSA and DoD; the Attorney General; and a court composed of federal judges appointed by the President with the advice and consent of the Senate and designated for the Foreign Intelligence Surveillance Court by the Chief Justice of the Supreme Court (also appointed by the President with the advice and consent of the Senate).
- reported upon regularly to the Intelligence Committees of the Senate and the House of Representatives.
One may take this either as evidence that the entire government is, or is becoming, a tyranny or as evidence that controls are in place that might or might not be considered adequate.
Contrary to often expressed opinion, it is not "obvious" that any of the reported activities violate either the Fourth Amendment or the law. Fourth Amendment law is both voluminous and complex, and skilled attorneys disagree about how to apply it in particular cases. That is what the case in hand is about.
Demonizing the NSA or others in the Executive branch, or judges who may be hearing cases that involve them, is not useful. It also goes against the basic agreement that underlies all representative democratic regimes: that we accept the output of the legislative and legal processes (including the laws against corruption in office) even when we disagree with them.
It is certain that the technical facilities available to the NSA and other intelligence agencies (and a large number of private sector organizations and individuals) have the potential for serious abuse. Depending on point of view there is little evidence for that at present, or none. It is certain as well that it is necessary to keep these intelligence programs and activities under close watch; review their operation often to identify actual, potential, or perceived misuse; and review their internal and external controls to identify potential areas for abuse (or where they might be limiting the legitimate use of a program).
Re: Re: How do we know?
On the other hand, it is probable, and in agreement with the published NSA slides, that a large part of the data that is acquired, for some meaning of "acquired", is discarded almost immediately, for example:
a. because both source and destination IP addresses are in the U. S.;
b. because it clearly is noise (e. g., cat videos not to or from a targeted area);
c. because it is a search for, or search result from, a query of no intelligence interest (e. g., my search for information about domestic dishwashers).
Such information normally would not be placed in a database, and the claim in the governments brief that significant changes could be required to retain it is reasonably plausible, although possibly somewhat overstated.
Re: You swallowed the cool-aid
Different rules apply to different kinds of communication. Telephone content is legally different from telephone company business records (the "metadata"), for example. The issue in this article appears to be mostly or entirely Section 702 collections, which are foreign targeted and for which warrants may not be required under the existing minimization rules for handling incidental capture of domestic data and data pertaining to U. S. citizens abroad.
"They are clearly already operating in breach of the law."
Actually, they are not. That is precisely the controversy in front of the court.
The poster plainly is unfamiliar with the history of the FISA and its various reauthorizations and amendments, or with the general contents of Executive Order 12333. As I noted earlier, it is possible that the laws and order are being violated, but no real evidence of that seems to be available.
Re: Depends on their system design
All things considered, it might not be completely unreasonable to argue that this does not constitute "spying" or even "collection" to the extent this is done by machine, for example based on comparison of source/destination IP address with known network structure. It would also, of course, be reasonable to argue that any agency that has a sniffer on the internet trunks needs very careful supervision, and that certainly would include the NSA, GCHQ, CSEC, ASD, and GCSB, as well as all employees of the companies who maintain the communication facilities.
Re: The silver lining isn't that deep.
You might wish to argue that they did not follow them, but there seems to be no evidence for such a claim other than a few NSA employees who were punished or dismissed for personal use of some of the data.
Re: Re:
Re:
Re: Backups and Spiders
However, the minimization rules I have seen in the publicly available documents require that information about U. S. persons be purged immediately when recognized as such. Changing existing procedures for that would, indeed, force that agency into noncompliance with its own internal controls, FISC orders, and the law under which the program is authorized, as the brief argues, in part.
Stipulating that the statement is true that the Section 702 collections that are the primary or exclusive target of the brief comprise multiple interrelated databases, and admitting the possibility that NSA has decent software development practices, several months may not be unreasonable for implementing the implied changes.
Techdirt has not posted any stories submitted by tddial.
Submit a story now.
Tools & Services
TwitterFacebook
RSS
Podcast
Research & Reports
Company
About UsAdvertising Policies
Privacy
Contact
Help & FeedbackMedia Kit
Sponsor/Advertise
Submit a Story
More
Copia InstituteInsider Shop
Support Techdirt