NO WAY TO PUT LIPSTICK ON THIS PIG: PANEL OF SCIENTIFIC EXPERTS FINDS THAT HIGHLY-PUBLICIZED CHICAGO POLICE STUDY OF ID PROCEDURES IS JUNK SCIENCE

Approximately a year ago, a highly-publicized study out of Illinois claimed to have raised questions about some of the procedural reforms being adopted around the country to improve the handling of eyewitness evidence. The study, taking the form of a pilot project spearheaded by Chicago police across three counties, purported to reveal that current identification procedures protected against mistaken identifications better than reforms (PDF) proposed by respected social scientists, based on extensive research on eyewitness memory. In other words, the Chicago police were happy to report that, notwithstanding the 19 wrongful convictions recorded in Illinois that resulted from faulty eyewitness evidence, everything was just fine and no pesky (scientific) reforms were needed.

Despite the fact that the "report" trumpeting the study was never subjected to peer review, and despite the fact that it was authored by a lawyer for by the very same Chicago police department that had fought reforms in Illinois for many years, these "findings" were trumpeted on the front page of the New York Times and have since served as serious impediments to reform in Legislatures around the country.

But something seemed rotten in Illinois, since the study's results were at odds with the recommendations of nearly every scientist who had studied the issue and the methodology underlying the pilot project appeared something short of scientific. According to one article on the topic, "the design of the project contained so many fundamental flaws that it is fair to wonder whether its sole purpose was to inject confusion into the debate about the efficacy of sequential double-blind procedures and to thereby prevent adoption of the reforms." Leading eyewitness researcher Gary Wells described the study's methodology as "extremely problematic."

Bearing out these suspicions, a new analysis was released yesterday, in which a "Blue Ribbon" panel of social science luminaries concluded that the study was "crippled by a design flaw that made the study's conclusions a dangerous basis for shaping public policy." This scholarly article does a fine job of showing that the Illinois pilot project on eyewitness lineup procedures was a sham, and a scandalous waste of taxpayer dollars. On that point, six out of six respected scholars agree.

The panel now weighing in includes Nobel Laureate Daniel Kahneman of Princeton and Harvard Professor and author Daniel Schachter, and other eminent scholars across the social sciences. In their own words, (PDF) "the design [of the Illinois study] guaranteed that most outcomes would be difficult or impossible to interpret," and the study's fundamental flaw has "devastating consequences" with respect to its scientific merits.

The story of the Chicago ID report is, in other words, one that is sadly familiar these days. Facts are cooked to suit the theory. The cooked facts are then trumpeted to an uncritical media, and the public understandably gives its officials the benefit of the doubt. But sooner or later the truth comes out, as it did here when the Illinois "study" received the sort of critical scientific analysis that should have occurred all along. In fact, this was just the sort of scientific rigor that the Legislature ordered when it appropriated tax-payer funds to conduct the study in the first place.

We now know that the failure to heed the Legislature's guidance resulted in the waste of many taxpayer dollars and, at least as importantly, the waste of a critical opportunity to determine how to fix a system that is currently relying too heavily on demonstrably mistaken eyewitness testimony. As the scientists suggest, the failure of Illinois now means that we need to await the results of properly-conducted studies in other jurisdictions to obtain this information.