PCAST issues report: Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods
This morning, the President’s Council of Advisors on Science and Technology (PCAST) released recommendations of actions to be taken to strengthen forensic science. The report was written in response to a request from the president to ascertain whether there were “additional steps on the scientific side, beyond those already taken by the Administration in the aftermath of the highly critical 2009 National Research Council report on the state of forensic sciences” that would assist in ensuring the validity of forensic evidence used in court.
During the course of a year, PCAST reviewed more than 2,000 papers, spoke with forensic scientists, attorneys and judges, researchers, and members of federal agencies. The report expressed concern about DNA analysis of complex mixtures, latent fingerprint analysis, bitemark, hair, and firearms analysis. The report also suggests the Attorney General should “direct attorneys appearing on behalf of the Department of Justice to ensure expert testimony in court. . . meets the standards of scientific validity.” It also recommends the Attorney General “revise and reissue for public comment the DOJ proposed ‘Uniform Language for Testimony and Reports’ and supporting documents to bring them into alignment with standards for scientific validity.”
Given that DOJ’s proposed original ‘uniform language’ document did not meet the standards of scientific validity, it is difficult to imagine how those same attorneys are capable of ensuring expert testimony meet the aforementioned standards.
The response to the report is already disappointing. According to the Washington Post, “the Justice Department had held a series of calls with prosecutors, law enforcement and lab officials, and that it . . . and would provide ‘a packet of information to federal prosecutors regarding how to dispute this report in court.’” The Post also quotes the National District Attorney’s Association, “Adopting any of their recommendations would have a devastating effect.” Finally, the Post quotes the FBI’s statement indicating the PCAST report contained “troubling generalized conclusions about all forensic science disciplines.”
This reaction is not unlike the reaction to the 2009 report by the NRC. Prosecutors and law enforcement wring their hands and declare it’s simply not possible to remove the use of forensic science from the courtroom, without regard to the science or data.
The report points out, “neither experience, nor judgment, nor good professional practices (such as certification programs and accreditation programs, standardized protocols, proficiency testing, and codes of ethics) can substitute for actual evidence of foundational validity and reliability.” It goes on to state, “Similarly, an expert’s expression of confidence based on personal professional experience or expressions of consensus among practitioners about the accuracy of their field is no substitute for error rates estimated from relevant studies.”
In assessing DNA, PCAST found, “at present, studies have established the foundational validity of some objective methods under limited circumstances (specifically, a three-person mixture in which the minor contributor constitutes at least 20 percent of the intact DNA in the mixture) but that substantially more evidence is needed to establish foundational validity across broader settings.” They noted that using computer programs to interpret complex mixtures also will require scientific scrutiny to determine whether the methods are scientifically valid, determining the limits of reliability, and whether the software correctly implements the methods.
Bitemarks also made the list of analysis. One might have thought bitemarks would have been tossed after the 2009 report, which stated,
The committee received no evidence of an existing scientific basis for identifying an individual to the exclusion of all others. That same finding was reported in a 2001 review, which ‘revealed a lack of valid evidence to support many of the assumptions made by forensic dentists during bite mark comparisons.’ Some research is warranted in order to identify the circumstances within which the methods of forensic odontology can provide probative value.
So, to set the stage, a 2001 review found bitemark evidence to be baseless. The 2009 report confirmed it. Why is PCAST reviewing it? Because it is still in use today! Courts continue to let it in at the request of prosecutors and over the objections of defense attorneys. The 2016 PCAST Report notes, “available scientific evidence strongly suggests that examiners not only cannot identify the source of bitemark with reasonable accuracy, they cannot even consistently agree on whether an injury is a human bitemark.”
Fingerprints and false positives
The PCAST report gives credit to the FBI for research done after the 2009 report, but also notes the following:
PCAST finds that latent fingerprint analysis is a foundationally valid subjective methodology—albeit with a false positive rate that is substantial and is likely to be higher than expected by many jurors based on longstanding claims about the infallibility of fingerprint analysis. The false-positive rate could be as high as 1 error in 306 cases based on the FBI study and 1 error in 18 cases based on a study by another crime laboratory.
A false positive rate of 1 in 306? At a minimum, defense lawyers should demand that this information be presented along with the admission of any fingerprint ‘match.’
The PCAST report notes the existence of one study, post 2009 report, that is an “appropriately designed black box study of firearms.” It also provides, perhaps for the first time publicly, a false positive rate for firearms. “The false-positive rate was estimated at 1 in 66, with a confidence bound indicating that the rate could be as high as 1 in 46. While the study is available as a report to the Federal government, it has not been published in a scientific journal.”
“PCAST focused on the reliability of conclusions that an impression was likely to have come from a specific piece of footwear. This is a much harder problem because it requires knowing how accurately examiners can identify specific features shared between a shoe and an impression, how often they fail to identify features that would distinguish them, and what probative value should be ascribed to a particular ‘randomly acquired characteristic.’”
There are no “appropriate black-box studies to support the foundational validity of footwear analysis to associate shoeprints with particular shoes. . . Such associations are unsupported by any meaningful evidence or estimates of their accuracy and thus are not scientifically valid.”
While the PCAST report was being prepared, the DOJ issued for comment proposed guidelines concerning testimony of hair examinations, and provided supporting documentation “addressing the validity and reliability of the discipline.” The DOJ documents stated, ““microscopic hair comparison has been demonstrated to be a valid and reliable scientific methodology.” The PCAST report notes the study relied upon by DOJ for this statement “does not support the conclusion that hair analysis is a ‘valid and reliable scientific methodology.’ The FBI authors actually found that, in 9 of 80 cases (11 percent) the FBI Laboratory had found the hairs to be microscopically indistinguishable, the DNA analysis showed that the hairs actually came from different individuals.”
Regarding DOJ’s improper citation of studies to support their position, PCAST also observes, “These shortcomings illustrate both the difficulty of these scientific evaluations and the reason they are best carried out by a science-based agency that is not itself involved in the application of forensic science within the legal system.”
In the bitemark section, PCAST made a comment that could apply to all disciplines discussed.
We note that some practitioners have expressed concern that the exclusion of bitemarks in court could hamper efforts to convict defendants in some cases. If so, the correct solution, from a scientific perspective, would not be to admit expert testimony based on invalid and unreliable methods but rather to attempt to develop scientifically valid methods.
Protests from prosecutors, law enforcement, and judges, who typically do not have scientific training, should not drown out repeated calls for putting “science” in forensic science. Judge Kozinski perhaps said it best in his editorial in the Wall Street Journal, “Among the more than 2.2 million inmates in U.S. prisons and jails, countless may have been convicted using unreliable or fabricated forensic science. The U.S. has an abiding and unfulfilled moral obligation to free citizens who were imprisoned by such questionable means. If your son or daughter, sibling or cousin, best friend or spouse, was the victim of voodoo science, you would expect no less.”
Christine Funk is a former criminal defense attorney with an interest in forensic science.