Eyewitness Blues, Part 2

The Controversy

Before reading this post, please read Part 1, which presents the evidence that sequential lineups are more accurate than simultaneous lineups.

Why do the police prefer the simultaneous method? First, it produces more identifications, which is their immediate goal. Since they are likely to believe that whomever they have arrested is the guilty party, they argue that the greater number of misses by the sequential method results in guilty suspects being allowed to go free, roam the streets, commit further crimes, etc. They are not as sensitive to the costs of false alarms since they underestimate how often they occur.

In his statement refusing to adopt sequential lineups, Mr. Bucar refers to a meeting of the National Academy of Sciences (NAS) in which there was “dissent among scientists who claim that the analysis methods used in the research between 2009 and 2011 was faulty and when corrected will actually show that . . . the simultaneous method is superior.” NAS's Committee on Science, Technology and Law held three meetings on eyewitness identification in 2013 and 2014. The program of these meetings, along with slide shows submitted by the participants, is available online. However, I couldn't determine the exact nature of the controversy to which Mr. Bucar refers. I am puzzled by his reference to analysis methods used between 2009 and 2011, since I don't detect any change in methodology. NAS is preparing a report on eyewitness identification to be released in the near future. I've signed up to receive a copy and I'll let you know their recommendations.

Meanwhile, here's my best guess as to what Mr. Bucar is concerned about. One of the great unknowns in the real world use of eyewitness identification is the base rate of culprit-present lineups. What percentage of lineups actually contain the prepetrator? Is it 50%? 75%? 90%? Neither the identification rate nor the conviction rate really answer this question.

In the studies referred to earlier, the base rate is 50%, since the researchers run an equal number of participants with culprit-present and culprit-absent lineups. In the real world, the lower the percentage of culprit-present lineups, the greater the danger of false alarms. For example, suppose the police conduct a near-random sweep of the neighborhood and show eyewitnesses many people who are not identified as suspects by any other type of evidence. Under these circumstances, it becomes more important to use the sequential method.

The police probably believe that close to 100% of their lineups are culprit-present. If they are right, the number of true identifications—or more likely, lucky guesses—lost in these culprit-present lineups using the sequential method might exceed the number of false identifications avoided in the (presumably) smaller number of culprit-absent lineups. Using Bayes' theorem, Steven Clark has identified the crossover point at which the misses exceed the false alarms using the sequential method. If the true base rate is greater than 85%, the simultaneous method will identify more guilty suspects.

Of course, no one will ever know what the true base rate of culprit-present lineups is. Your guess is likely to be influenced by whether you trust the fairness and efficiency of police investigative procedures or are more skeptical. One of the more important contributions of DNA testing is that it has greatly increased our estimate of the number of innocent people who are convicted of crimes in this country.

However, simply comparing the rate of false alarms in the simultaneous condition to the rate of misses in the sequential condition doesn't do justice to the true superiority of the sequential method because a false alarm is a more costly error for society than a miss. In both cases, the real culprit escapes detection, but when a false alarm occurs an innocent suspect is likely to be arrested and charged, with costs to that individual that range from considerable inconvenience to the complete ruin of his or her life. This is presumably the basis of Lord Blackstone's famous quote: “It is better that ten guilty men escape than that one innocent suffer.”

In a classic case of bad timing, Allegheny County police recently released a man arrested based on a mistaken identification from a culprit-absent lineup using the sequential procedure. Another man confessed to the crime. Of course, no one claims the sequential procedure is foolproof. Our best guess is that, when the culprit is absent, it will produce false alarms almost one-third (32%) of the time, which is disturbing, but not as bad as the 54% false alarm rate using the simultaneous procedure.

You may also be interested in reading:

Eyewitness Blues, Part 1

Book Review:  Failed Evidence, by David Harris

Another Miss By the Supremes

Eyewitness Blues, Part 1

The Issue

On July 31, Pittsburgh public safety director Stephen Bucar had an unpromising first day on the job. A week before, Allegheny County District Attorney Stephen Zappala sent him a letter requesting changes in the way the Pittsburgh city police conduct eyewitness identifications. Mr. Zappala said his office would no longer prosecute cases unless (1) the identification was made using a sequential rather than a simultaneous lineup and (2) the lineup was conducted by an officer not involved in the investigation. His demand may have been motivated by an abstract desire for justice, but it's more likely the result of two recent lawsuits filed by men who were mistakenly arrested based on faulty identifications using the simultaneous method.

But the Pittsburgh police, notorious for their resistance to change, said something like, “That's not how we do things around here.” Unfortunately, Mr. Bucar decided to back them up, citing what he believes to be conflict in the scientific literature over whether sequential or simultaneous lineups are more likely to lead to error. This is disappointing news for those of us who had hoped new Mayor Bill Peduto would bring more rationality to Pittsburgh government.

Stipulating that the identification be conducted by an officer not involved in the case assumes that this officer won't know which person in the lineup is the suspect. If so, this double blind procedure prevents bias in which the administrator unintentionally communicates the identity of the suspect to the eyewitness. Since Mr. Bucar did not mention blind testing in his statement, I presume he has no objection to it.

When an eyewitness views a lineup, there are four possibilities. The culprit is either present in or absent from the lineup, and the eyewitness either does or does not make an identification. If the culprit is present and the eyewitness correctly identifies the culprit, this is called a hit. When the eyewitness either identifies someone else or makes no identification, this is a miss. When the culprit is absent and the eyewitness erroneously makes an identification, this is a false alarm. Failure to identify anyone is a correct rejection. Obviously, the goal is to maximize hits and correct rejections, thereby minimizing misses and false alarms.

	Culprit Present	Culprit Absent
Identification	Hit	False Alarm
No Identification	Miss	Correct Rejection

In the traditional simultaneous procedure, the eyewitness is shown an array of (usually) six photographs and asked which one, if any, is the perpetrator. In the sequential procedure, the eyewitness is shown the photographs one at a time, and is asked to state whether each one is the perpetrator. In this procedure, eyewitnesses usually don't know how many photos they will be shown.

In studies that compare these methods, participants unexpectedly witness a simulated crime, either live or on video, and are later asked to make an identification. In those studies with complete designs, the perpetrator is present in half the lineups, while in the other half, he or she is absent. Half of each of these lineups are conducted using the simultaneous and the sequential procedure. Since the researchers know whether the culprit is present in each lineup, they can determine whether a correct identification was made.

Several decades of research show the sequential procedure to result in greater accuracy. In 2011, Nancy Steblay and her colleagues published a meta-analysis of all 72 known studies (from 23 labs involving over 13,000 participants) comparing simultaneous and sequential lineups. Simultaneous lineups produce higher choosing rates (76%) than sequential lineups (61%), suggesting than the simultaneous method encourages guessing. More importantly, identifications from a sequential lineup contain fewer errors.

To illustrate this, here are the data from the 23 studies with complete designs (as described above). Since all four conditions are present in these studies, it is possible to compare their average results.

Hits from culprit-present lineups

          Simultaneous                     52%

          Sequential                         44%

          Difference                           8% fewer hits in sequential

False alarms from culprit-absent lineups

          Simultaneous                     54%

          Sequential                          32%

          Difference                          22% fewer false alarms in sequential

There is a tradeoff. If all lineups were culprit-present, the simultaneous method might appear to be superior, since it produces 8% more identifications. However, since the simultaneous method encourages guessing, some researchers argue that this 8% difference should be treated as lucky guesses rather than true identifications. When the culprit is absent, the simultaneous method leads to 22% more false alarms. Therefore, the sequential method is more accurate.

Why is the simultaneous method more error prone? When an eyewitness is asked to view a lineup, he or she probably infers that the police think they have arrested the perpetrator. In a simultaneous array, eyewitnesses make a relative judgment. They compare the photos to one another and choose the one which most closely resembles their recollection of the perpetrator. In the sequential method, however, because they don't know how many suspects they will be shown, they must make an absolute judgment, comparing each suspect to their independent memory of the perpetrator.

Here is a 61-min lecture by researcher Gary Wells that I posted once before. Between minutes 14 and 30, he talks about the relative judgment process and the need for sequential lineups.

To be continued.

You may also be interested in reading:

Eyewitness Blues, Part 2

Book Review:  Failed Evidence, by David Harris

Another Miss By the Supremes

Book Review: Failed Evidence, by David Harris

Failed Evidence: Why Law Enforcement Resists Science, by David Harris, Professor of Law at the University of Pittsburgh, concerns one of the recurrent themes of this blog—the fact that sound social science research seldom has much effect on public policy. (Check out his seven short videos dealing with main themes of the book.)

Our legal system should be undergoing a crisis of confidence as we continue to discover the number of innocent people who have been sent to jail. DNA evidence has resulted in over 300 exonerations so far, but DNA is only available in 5% of criminal cases, suggesting that we are seeing only the tip of the iceberg. Dr. Harris reviews research which casts doubt on current practices in the three areas that account for the greatest percentage of exonerations: incorrect eyewitness identifications, inaccurate forensic testing, and false confessions.

In all three areas, current practices are deeply flawed, and research suggests straightforward remedies that would greatly decrease errors. Eyewitnesses are usually shown lineups and photospreads using procedures that encourage them to pick the suspect who most closely resembles the perpetrator. The administrator knows which person is suspected and gives positive feedback if the suspect is chosen, which increases the eyewitness's certainty. Forensic tests, i.e., identification of fingerprints, firearms, hair and blood samples, etc., are inherently subjective. The reliability that is sometimes claimed for them occurs only when the second examiner is told in advance of the first one's results. False confessions occur because police question suspects for long periods of time under extreme pressure, and are permitted to lie about evidence. This produces fatigue and confusion, and some innocent suspects confess only to put an end to this painful situation. Because the idea that an innocent person might confess is counterintuitive, I've included this 10 minute video from Saul Kassin which gives further explanation of how it occurs.

Dr. Harris reviews the reasons police officers give “in their own words” for their fierce resistance to such evidence and the remedies they imply: they are too expensive, that they will allow guilty people to go free, the research is less valid than their own personal experience, etc. The most perversely amusing example of police intransigence is their inconsistent reaction to DNA evidence depending on whether it implicates or exonerates the defendant. Since their objections are easily debunked, he speculates about the real reasons for resistance.

These are divided into two categories. Cognitive reasons focus on the thought processes of the individual officer. An example is cognitive dissonance, which occurs when police are told that their past behavior may have sent innocent people to jail. This possibility is vigorously denied, and they attempt to bolster the effectiveness of current practices. Institutional reasons cite the context in which police and prosecutors work. Police officers are evaluated on the basis of the number of arrests they make and prosecurors are judged by their conviction rate. There is nothing in the system that rewards either of them for doing justice by releasing an innocent person.

Dr. Harris then lists 16 recommendations for reform, i.e., videotape all police interrogations from beginning to end, and makes some suggestions for how to make these changes happen. For example, cognitive dissonance may be minimized by focusing on the future rather than past injustices. It's at this point that Dr. Harris begins pulling his punches by compromising in advance with what he believes to be implacable resistance from the law enforcement community.

For example, the following one hour presentation by Dr. Gary Wells explains the process of relative judgment, and why we should replace simultaneous lineups, which result in identification of the suspect who most resembles the perpetrator, with sequential presentations in which the witness makes an independent judgment about each suspect. A meta-analysis of relevant studies confirms that sequential lineups greatly reduce incorrect identifications. However, they also slightly reduce correct identifications, since the simultaneous lineup results in a small percentage of “lucky guesses.” The loss of those lucky guesses is interpreted by law enforcement as letting innocent people go free, which generates resistance. Unfortunately, Dr. Harris's response to this dilemma is to suggests that this recommendation—in my judgment, the most important one of all—be optional, since it is unlikely to be accepted anyway.

By the way, here is the soon-to-be-released study Dr. Wells refers to in his lecture. The upcoming Supreme Court case he mentions did not have a satisfactory outcome.

It's clear that Dr. Harris is not optimistic about the prospects for reform. (Dr. Wells is much more optimistic.) He gives many examples in which police and prosecutors, working through their professional organizationals, have easily defeated reform attempts. The primary victims, innocent people wasting away jail, have no political clout. He ends the book with a chapter in which he presents only three cases in which meaningful reform has taken place. (The Innocence Project lists other examples.) In two of these cases, a conservative politician converted to the cause of reform and was able to use his credibility as a strong advocate of law and order to make it happen. If advocates as sympathetic to police and prosecutors as Dr. Harris are unable to bring about bottom-up change through persuasion, the best hope for social scientists may be to go over their heads, either through federal legislation or court decisions, such as the Miranda ruling, which law enforcement personnel are required to obey.

Another Miss By the Supremes

There are few other topics in applied psychology that have been more heavily researched than eyewitness identification. More than 2000 studies of the subject have been published in professional journals. Yet the influence of this research on the legal system has been painfully slow. There have been no major changes in federal law governing eyewitnesses since 1977. Earlier this month, the Supreme Court took a major step backwards.  This decision doesn't belong on the same CD with "Stop in the Name of Love."

In a series of cases between 1967 and 1977, the Supremes ruled that inaccurate identifications violate the due process rights of the defendant. Eyewitness testimony can be suppressed before it is presented to the jury if the identification is made under “suggestive” conditions. In previous cases before the Court, those biased conditions were orchestrated by the police. At issue in the present case: What happens when the eyewitness is exposed to a suggestive procedure for which the police are not responsible?

In Perry v. New Hampshire, a witness called the police at 2:30 a.m. to say she observed a black man breaking into a car from her fourth story window. The police apprehended Barion Perry, a black man, in the parking lot with stolen objects in his possession. When a policeman went to the witness's apartment to interview her, she on her own initiative went to the window and identified Perry, who was standing in the lot next to another officer. This is not a reliable identification, since the distance was great and the lighting was poor. Furthermore, the fact that Perry was standing in police custody made it inherently obvious that he was the suspect. In fact, the witness failed to pick Perry out of a photo array a month later. Nevertheless, the prosecution put her on the stand, and Perry was convicted.

Although this is counterintuitive to most people, we are not very good at identifying persons whom we observed for only a short period of time. Incorrect identifications are a common occurrence. Extrapolating from large-scale field studies, the American Psychological Association (APA), in a brief filed in the case, estimated that about one-third of eyewitness identifications are incorrect. The Innocence Project, using DNA evidence, has exonerated nearly 300 prisoners who were falsely convicted. In 76% of these cases, an incorrect identification was involved.

Variables that affect eyewitness accuracy can be divided into two categories: estimator variables, which take place at the time the eyewitness observes the crime, and system variables, which occur when the identification is made, many of which are, or should be, under the control of the state. Estimator variables include the length of time the suspect was observed, the distance, the lighting conditions, and whether the eyewitness is under stress, such as that produced by the presence of a gun. Cross-racial identifications are less accurate than when the suspect and the eyewitness are of the same race. System variables under the control of the police involve the fairness of the identification procedure. If a six-person lineup is used, all five non-suspects must fit the perpetrator's description, and to avoid bias, the officer who conducts the lineup should not be aware of the suspect's identity. The most error-prone identification procedure of all is the one-person showup. However, not all system variables are under police control, as Perry illustrates. Suggestive identification procedures can also be instigated by the media, or by friends of the witness.

The Perry case concerns a system variable that was not under police control, although you might argue that it should have been. The Supremes ruled 8-1 against Perry. The lead singer in this case, Justice Ruth Bader Ginsberg, writing for the majority, acknowledged that Perry's identification occurred under suggestive conditions, but said that eyewitness testimony can be suppressed only if the suggestive procedure was caused by police action. This excludes the possibility of suppressing eyewitness testimony that is unreliable due to estimator variables, or due to system variables not under police control. (Ironically, if the police officer had suggested that the witness look out the window, her testimony would probably have been excluded.) The justification for suppression of eyewitness testimony in previous cases, Justice Ginsberg argued, was to deter the police from arranging unfair or biased identifications. This is a very narrow reading of the language of previous eyewitness cases.

Justice Ginsberg noted that the defense attorney can try to discredit the eyewitness through cross-examination, making it clear to the jury that the identification took place under biased circumstances. However, research indicates that mock jurors “overbelieve” eyewitnesses; that is, they believe witnesses whose identifications are known to researchers to be false. Here's how it usually happens. The studies show that eyewitnesses are overconfident of the accuracy of their identifications. As a result, there is no relationship between their confidence and their accuracy. Jurors, in turn, base their judgments of the credibility of an eyewitness primarily on his or her confidence. Since witnesses are just as confident of incorrect identifications as correct ones, jurors are unable to distinguish between accurate and inaccurate eyewitnesses. Neither a warning from the judge nor expert testimony regarding the fallibility of eyewitnesses are sufficient to eliminate the tendency to overbelieve eyewitnesses. (See the APA brief for references to specific studies which support these conclusions.)

The only dissonant note in the case came from Justice Sonia Sotomayor. She argued that the real purpose of suppressing tainted eyewitness testimony is to prevent miscarriages of justice caused by incorrect identifications. For example, in Manson v. Braithwaite (1977), the court said that “the primary evil to be avoided” is the “likelihood of misidentification.” Deterring the police from using biased procedures is just one way of avoiding misidentifications. This suggests that the court should exclude eyewitness testimony that is a product of suggestive circumstances, regardless of whether those circumstances results from police action, police inaction, or sources other than the police.

It is profoundly discouraging to social scientists when the Supremes base their decisions on technicalities and fail to appreciate the overall implications of sound scientific research.