FBI, DOJ And Their Forensic Scientists State They'll Continue Using Discredited Junk Science To Put People Behind Bars
from the if-it-ain't-working,-don't-fix-it dept
For dozens of years, criminal prosecutions have relied on junk science. Forensic science, properly applied, can actually provide matches that identify suspects. But it's not properly applied. In the hands of the DOJ, forensic evidence examination is a closed loop. Outside scientists have been granted access to the DOJ's DNA work, but everything else -- from fingerprints to hair samples -- has been locked away in the government's database.
Still, the DOJ insists its science is solid, something it bases on confirmation bias. The matches determined in its forensic labs are "scientifically certain" because the DOJ's expert witnesses have said so in court. Not only are outside scientists locked out of examining evidence and forensic processes, but defense lawyers are as well.
The DOJ has finally decided to dial back its "scientific certainty" a bit by issuing guidance instructing its experts to not make this claim in court. This follows years of bogus matches being presented as sure things by forensic experts in court, leading to an unknown number of false convictions. This step back is a step forward for an agency that is mostly unwilling to admit to any mistakes or wrongdoing.
This small change is likely due to a damning report [PDF] issued by the President's Council of Advisors on Science and Technology (PCAST) that asserts that the "scientifically certain" evidence prosecutors rely on is severely flawed.
The questions that DNA analysis had raised about the scientific validity of traditional forensic disciplines and testimony based on them led, naturally, to increased efforts to test empirically the reliability of the methods that those disciplines employed. Relevant studies that followed included:
• a 2002 FBI re-examination of microscopic hair comparisons the agency’s scientists had performed in criminal cases, in which DNA testing revealed that 11 percent of hair samples found to match microscopically actually came from different individuals;
• a 2004 National Research Council report, commissioned by the FBI, on bullet-lead evidence, which found that there was insufficient research and data to support drawing a definitive connection between two bullets based on compositional similarity of the lead they contain;
• a 2005 report of an international committee established by the FBI to review the use of latent fingerprint evidence in the case of a terrorist bombing in Spain, in which the committee found that “confirmation bias”—the inclination to confirm a suspicion based on other grounds—contributed to a misidentification and improper detention; and
• studies reported in 2009 and 2010 on bitemark evidence, which found that current procedures for comparing bitemarks are unable to reliably exclude or include a suspect as a potential biter.
Beyond these kinds of shortfalls with respect to “reliable methods” in forensic feature-comparison disciplines, reviews have found that expert witnesses have often overstated the probative value of their evidence, going far beyond what the relevant science can justify. Examiners have sometimes testified, for example, that their conclusions are “100 percent certain;” or have “zero,” “essentially zero,” or “negligible,” error rate. As many reviews—including the highly regarded 2009 National Research Council study—have noted, however, such statements are not scientifically defensible: all laboratory tests and feature-comparison analyses have non-zero error rates.
Despite these conclusions, law enforcement forensic scientists, along with the FBI and DOJ are promising to continue using junk science to convict people. The Attorney General's response to the report is basically, "Thanks for all the hard work, but we're not changing a thing."
In a statement reported by the Wall Street Journal, Attorney General Loretta Lynch said that the agency remains “confident that, when used properly, forensic science evidence helps juries identify the guilty and clear the innocent, and the department believes that the current legal standards regarding the admissibility of forensic evidence are based on sound science and sound legal reasoning.” As such, she said, while “we appreciate their contribution to the field of scientific inquiry, the department will not be adopting the recommendations related to the admissibility of forensic science evidence.”
The FBI's response [PDF] is no more enthusiastic. Its one-page blast claims the entire PCAST report is flawed. It also asserts there are "multiple studies" that back up its forensics work. However, the FBI turned down an invitation to participate in the PCAST study and one of the PCAST members points out that the studies the FBI claims contradict the PCAST findings actually do no such thing.
Asked about the FBI’s complaints, Eric Lander, co-chair of the presidential council and president and founding member of the Broad Institute of MIT and Harvard, a biomedical research group, told The Intercept that the FBI is mistaken. “Neither report says that proficiency testing be used to estimate the ‘error rate’ of forensic methods,” he wrote in an email, and both reports agree that examiners should be subject to proficiency tests. And Lander said he is “not aware” of what studies the FBI believes were ignored by the report. “We specifically received FBI’s input on studies to consider and we did so.”
[...]
Lander wrote that the presidential council did in fact review the six studies the FBI complains that it missed. “However, these studies are clearly not empirical studies ‘providing support for foundational validity,'” he wrote. “Indeed, only one of the papers even reports an empirical study of current forensic method at all!”
But the most defensive response [PDF] to the PCAST study has come from the American Congress of Forensic Science Laboratories. Unsurprisingly, the group isn't pleased that its conclusions and methodology have been questioned. Before addressing the shortcomings it feels the study contains, the ACFSL first tries to claim the whole thing is just a politically-motivated attack on the good people in law enforcement.
Interestingly, the PCAST report comes during a presidential administration that has demonstrated a deep sensitivity to the needs and demands of trial attorneys, criminal defendants, and advocates of sweeping criminal justice reform. Future administrations may take a different approach, tending to champion positions traditionally held by police and prosecutors. We have no opinion in these matters. But these swings in ideological perspective cause commensurate changes in how forensic science and its role in our criminal justice system are perceived. In the current political climate, forensic science is looked upon with far more suspicion and, in some cases, disdain than would be the case in other political circumstances. And because forensic science is both expected and apt to remain independent of these political currents, it is vulnerable to being misportrayed and even bullied in a way that compromises its occupational stability.
It's somewhat interesting that a field where evidence has been used to deprive innocent people of their freedom -- and whose courtroom assertions are treated as nearly-unassailable despite a long history of uncorrected errors -- can somehow be bullied by a report both the DOJ and FBI have already indicated they're going to ignore.
It also attacks some of the contributors for their work with the Innocence Project (Eric Lander) and their previous assertions about the intrusiveness of DNA collections (Tania Simoncelli) -- all the while claiming it has no interest in "disparaging" the authors of the study. Even in this, the forensic scientists' sample size is way too small. Attacking Lander for his Innocence Project work conveniently overlooks his considerable contributions to the scientific community.
[Innocence Project's Chris] Fabricant said the congress’ assertions were absurd. “To suggest that the leading scientists in the country would cash in their credibility to do – what? What possible agenda could they be pursuing except scientific validity?” he asked. “I’d like to know what agenda they propose is being driven, and how somebody like Eric Lander – who mapped the human genome — is going to preside over a process that is intended to undermine the criminal justice.”
These responses indicate business -- as lousy and inaccurate as it is -- will continue as usual until the DOJ is forced to confront these issues, either by litigation or legislation. Not exactly heartening news for citizens who don't like being locked up for crimes they didn't commit.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: doj, fbi, forensic science, junk science, prosecutors, science
Reader Comments
Subscribe: RSS
View by: Time | Thread
"You keep using that phrase..."
Justice requires that you make absolutely sure that you convict the right person, even to the point of tossing a potential conviction if the evidence isn't rock solid that they're guilty of the crime they are being tried for.
If he was really interested in 'justice' he would support wholeheartedly any attempt to make sure that the tools used to secure a conviction are as close to being 100% accurate as humanly possible. That he and several other groups instead choose to attack both the study and the people who wrote it make clear that they don't give a damn about justice, all they want is convictions.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
Re: Re: Re:
[ link to this | view in chronology ]
What a Charade
So long as the lowbrow troglodytes infesting FBI and DOJ (haha) are going to continue to use wholly discredited junk science as "evidence" they should bring back the long debunked pseudo-science of phrenology.
These disgraceful clowns have absolutely zero credibility left.
https://www.princeton.edu/~cggross/ency_neuro_phren.pdf
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Pay for play
all evidence studies should be double blind.
[ link to this | view in chronology ]
You know who else had double blind studies?
Oedipus.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
Not really true
I had to comment on this because I worked on NGI from 2009 to 2014 as a Lead Engineer.
Fingerprint technology is very useful in catching real criminals. Rarely is someone convicted on a fingerprint match alone - that is just not the way Criminal Justice works in my experience. The fingerprint leads to a person who then law enforcement/detectives will build a deeper case based on additional information. One example I was directly involved in: Upgrade to NGI - Got a hit on a murder after years of no clues. Able to substantiate that the person was at the location of the crime by video of that person's car. Arrested and Convicted. Before NGI and the enhanced fingerprint technology, the police had no clues that had been able to pinpoint the murderer much less any suspect.
My point is people are convicted every day on very circumstantial evidence. The problem is not with the technology but the Prosecutors willing to prosecute people with out building a full case.
[ link to this | view in chronology ]
Re: Not really true
However, the entire tree is being poisoned by the darkness. In court, that normally entitles the defendant to an adverse inference. Nerd harder man, nerd harder!
[ link to this | view in chronology ]
Re: Re: Not really true
Fingerprints may be unique, but the person who calls the match, and the protocol for calling a match are clearly in question. By refusing blind quality assurance studies, the FBI has demonstrated a disregard for truth. And a willingness to make evidence say what it wants it to say.
[ link to this | view in chronology ]
Re: Not really true
Above all, defendants must have the right to challenge any and all evidence presented against them. Too often errors, innocent or malicious hide behind a wall of secrecy, NDAs and trade secret claims. Many analytic devices and tests do not function as advertised, having error rates as high as 1/3 false positives or more. Knowing this (or should know this) police, DAs and Judges make life wrecking decisions about individuals who are quite often totally innocent.
Being a retired biochemist, I am totally disgusted by the way that the legal system abuses what it calls science.
In full disclosure, I have never been arrested, ticketed for a moving violation or been attacked by a LEO. My worst violation was failure to display a registration sticker that was present in my glove box.
[ link to this | view in chronology ]
Re: Not really true
I don't see a problem with using the forensic evidence to aid in investigations and to be used in court. The problem I have is with that evidence being misrepresented by expert witnesses as being infallible.
In the cases of NGI and EFT, these can be used as a great investigation tool. But when the case gets to court, I want to hear "we used NGI and EFT to narrow down the list of suspects. When we then investigated suspect X, we found..." and not "we used NGI and EFT, and they pointed to suspect X as the likely culprit. The tools provided a 100% match, and irrefutably linked the suspect to the crime."
The problem is, the latter is often the case. I've been lucky to be in court where a judge has actually said "you can't say that. I know how science works" to an expert witness, but cases like that aren't common.
[ link to this | view in chronology ]
Re: Not really true
commentary on the applicability of your anecdote and anecdotal evidence and your ignorance of it harming your point aside,
cops found a fingerprint, and with traditional sight based analysis you need a suspect to compare to, so fingerprint analysis was useless in finding the culprit. you introduced ngi and were able to search a database with your proprietary, unproven algorithim and found a match.
With a suspect in hand they were able to find video of his car in the area, that they didn't consider relevant at any time in the last few years. probably because the video didn't actually show the driver clearly? because location proximity wasn't as clear as you represent?
That was the case? kinda proved this article's point i think. and i wonder why i never get selected for jury duty.
[ link to this | view in chronology ]
Re: Re: Not really true
1. NGI is 99.6 percent accurate. Data tests were completed against very large data sets 1st by the private industry and then by the FBI and NIST.
2. The fingerprint that the murder (he is convicted now) left was ran against the FBI's IAFIS monthly for years. It was also ran against local IAFIS systems. With NGI enhanced imaging both on the latent (the finger print left behind) and on all criminal records (10 Finger or Slaps) the new system was able to identify the murderer. This most wanted person was found just days after testing in this region had started. Mind you his prints were ran a few months earlier after being arrested but did not hit on the older system.
3. All the Security footage for that day at that location was kept.
4. He admitted to it after being caught.
http://wtop.com/news/2012/12/forensic-link-leads-to-arrest-in-vanessa-pham-murder/
At the end of the day a family who will never get their daughter back did get Justice and closure. This guy had up until that point got away with it.
To be honest NGI can beat experts in forensic matching every time. A computer doesn't have the same biases and errors.
I no longer work in this industry but this is not a system to be freaked out about and it doesn't have any effects on citizens privacy. NGI is not the NSA its use is not hidden behind Non Disclosures. The technology is open and widely used. I honestly do not see a better system for background checking and forensic science. The worst thing I have ever heard about NGI and its child systems is some dumb local police officer doing unauthorized backgrounds.
Please, focus your efforts on real criminal justice reform. The innocents project and Black Lives Matter. Police accountability. Focus your privacy issues on Cell phone trackers (or at least making the technology more open in courts) and against our own government (and friends) spying on us. There is sooooo much to focus on.
NGI and its child (local government) systems are part of the solution not the problem. They reduce that bad "expert" in courts by using real science built in not opinion.
[ link to this | view in chronology ]
Re: Re: Re: Not really true
Point us to published, double-blind studies supporting your position, ones that we can read without paying huge amounts of money or having access to the right libraries/connections. Tell us how NGI works and what makes it so good without violating confidentiality.
Can't do it, can you? Didn't think so.
It's all well and good to assert things about $SOFTWARE or $SCIENTIFIC_METHOD. But at the end of the day, you have to prove it.
And you can't.
[ link to this | view in chronology ]
Re: Re: Re: Re: Not really true
I can provide open studies because NGI is not hidden in secrecy. It was probably one of the most tested systems the government uses for security and accuracy. Everyone I worked with knew what was at stake.
The study used in this post muddies the water as it reflects pre 2011 fingerprinting.
By the way: I am no longer paid by this industry. Its just some work I did on my linkedin today.
http://nvlpubs.nist.gov/nistpubs/ir/2014/NIST.IR.8034.pdf
https://pdfs.semanticscholar.org/9bff/943d3 68469d614fb860624ac6b2e6275bffe.pdf
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Not really true
I have also seen them abused and misused to produce results that were more fallacious than the abominations spoken of in the comments on this critique of the FBI.
You use the term computer science. That is not generally applied to the application of computers in use, but rather the theory of algorithms and mathematics as applied to computation.
"Computer science is no more about computers than astronomy is about telescopes."
--Edsger Dijkstra
[ link to this | view in chronology ]
Re: Re: Re: Not really true
This point doesn't really count for much considering how defendants are often coerced into plea bargains because they can't afford the kind of legal representation required to adequately compete against the government in court.
[ link to this | view in chronology ]
Evidence again the courts are a sham.
This news confirms that the innocent convicts in the penal system are uncountable and probably numerous.
This news confirms that convicts are really political prisoners, not there because they were actually determined to be guilty of a crime, but are cast away for the benefit of officials. At very least for those officials who are part of the justice system who benefit from high conviction numbers.
Ultimately, this indicts the United States Justice System as being a tribunal, once there is a system of oversight to give it commission. I wonder if at any point a challenge of habeus corpus could appeal to the failure of the courts to provide for a fair trial.
[ link to this | view in chronology ]
Statement re 2004 Bullet analysis
Anyone who passed middle school algebra or had a course in logic, had to know that this "analysis" was anathema to science and law.
[ link to this | view in chronology ]
Just like the DEA
[ link to this | view in chronology ]
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Imagine what would happen if they had to deal with the fallout from decades of 'science' being shown to be inaccurate, and the ensuing lawsuits to get compensation for people railroaded to jail.
All of these former prosecutors who were tough on crime, but now will fight to the death to protect their record at the cost of innocent people.
This is another moment where they have chosen to protect the image over the truth, chipping away at the foundation of the system that we all know is broken but can't seem to find the will to demand change until it turns its lazy eye to us as the next target to be screwed.
[ link to this | view in chronology ]
So is climate science settled?
[ link to this | view in chronology ]
Climate science
Of course not. But we have scientific models that get pretty accurate in their ability to predict. And we have a predilection for preferring simple, easy explanations for phenomena we cannot fully understand.
Regarding any of these fields, every day we're learning new things we didn't learn before. Bone fragments and tools come in that force paleontologists to adjust their estimations of when certain things were invented and where. Observations from galaxies away show us things we've never seen before which can confirm (or disprove) the models we understood to be true.
Do we know everything about the climate and how it affects mass extinctions? Obviously not.
But we do know a whole lot of stuff about how the temperature of the Earth can affect mass extinction events. There's a certain tiny temperature range that surface life enjoys, and if it gets much hotter or much colder (by even a couple of degrees) then things stop surviving. First delicate species go extinct and later hardier species. Massively. (Hint: It's already happening. Look up the Holocene Extinction Event.)
For instance, we know with strong certainty the atmospheric conditions that took place in the Permian–Triassic extinction event (what we call The Great Dying since more of the world went from alive to dead than ever or since) and we know how it happened, and we know it can happen again, say if the sub-tundra methane deposits are released into the atmosphere. And we know that we're only one ~2c average temperature increase away from that happening again and choking out the Millennials' grandchildren. Those that depend on outside air to breathe at least.
And that's one of the several models of how the human species can end (or go from 7 billion to -- generously -- 70,000). There are a few others, and we know they can happen because they've happened before.
Is climatology a done deal? No. Are the doomsayers absolutely certain that we're all going to die? No. So how sure are they that anthropogenic non-point-source pollution is going to end us? Really, pretty darned.
And that's to say if we make no changes, if we still consume fossil fuels at our current rate (all the while accidentally dumping a bit in the ocean), if we continue to produce beef at our current rate, if we continue to manufacture without capping our carbon emissions, then yes, we're probably going to see a massive population drop-off, and the ones that live will not be having a good time of things.
Now, of course, we could see a massive change in technologies in the next few decades that could slow all that down. We might even be able to reverse things a bit if we got ambitious about it. But considering how companies regard greening their products as a marketing point (they charge more for convincing you their name represents a cleaner conscience), considering how old industries litigate for decades to tie up new industries that might disrupt their business model, it means we're just not cooperative enough to see the problem without a whole bunch of contrary disinformation. People don't want to have to adapt, even if it's going to kill them.
So yeah, because we're handling the environmental risks of industry about as well as we handled the health risks of tobacco, it is likely that the population of the world is going to respond too little too late, and most of them are going to die. Soon. Tragically.
You, in the meantime, get the privilege of contemplating whether or not its a problem and, if you like, convincing yourself the risk to the future of your species is worth our lifestyle today. If we don't change at all and you're young enough you may even live to participate in the extinction event, whether you're suffocating (or starving to death or dying from plague) yourself, or you first get to watch most everyone else do it.
The question should not be Is climate change settled. It should be Should we regard climate change as a high-extinction risk.
The answer is fuck-us-all yes.
[ link to this | view in chronology ]
"trust us"
Sounds more like they ask for blind faith, which is the attributes of religion rather than science.
So, maybe we should start talking about "forensics religion" from now on.
[ link to this | view in chronology ]
If you read journals then you can provide an opinion with a solid basis. But if you have but if you have been trained in science and math enough to proffer such an opinion, then say so. Otherwise you are merely taking a side that is oriented to political opinion.
I have been trained in the physical sciences, know a great deal of math, and would not dream of offering an opinion on the scientific veracity of climate warming. I do know that the climate is warming. That the last day of frost in the spring is getting earlier, and the last day of frost in the fall is getting later. That temperatures are generally increasing. That respected scientists have offered not just the data, but also a functional model for the process. What is more, the publishable data is available to anyone, and the work is open to replication.
With fingerprints, the issue is interpretation. To the best of my knowledge, no proper studies have been performed. The data is hidden, people not on the payroll are not permitted to study or attempt to replicate the work.
The former is science, the latter does not meet the criteria of the scientific method. Whether climate science produces correct hypothesis or not, it is conducted as science. Whether or no fingerprint matching produces a correct hypothesis or not, it is not conducted as science.
[ link to this | view in chronology ]