Good Studies Go to the Back of the Bus

It's a rare day when my daily newspaper doesn't include at least one medical or health related article. My subjective impression is that they frequently report on potential “breakthroughs,” but many of them are never heard of again, suggesting that the early results were not reproducible.

A new study by Senthil Selvaraj and two colleagues suggests that newspapers do not publish the best available studies. In medical research, the main criterion of a good study is whether participants were randomly assigned to receive either the treatment or some control procedure such as a placebo. In medical jargon, this is called an RCT study, which stands for randomized controlled trial. The major alternative is an observational study, in which the participants are contrasted with a comparison group that may differ from them in uncontrolled ways (a cross-sectional study), or are compared to themselves at an earlier time (a longitudinal study). Some observational studies are merely descriptive and lack a comparison group.

© mercatornet.com

The authors selected the first 15 articles that dealt with medical research using human subjects published after a predetermined date in each of the five largest circulation newspapers in the US. Referring back to the original research reports, they classified each study on several dimensions, the most important being whether it was an RCT or an observational study. For comparison, they selected the first 15 studies appearing in each of the five medical journals with the highest impact ratings. These impact ratings reflect how often studies appearing in these journals are cited by other researchers.

The main finding was that 75% of the newspaper articles were about observational studies and only 17% were about RCT studies. However, 47% of the journal articles were observational studies and 35% were RCT studies. A more precise rating of study quality using criteria developed by the US Preventive Services Task Force confirmed that the journal studies were of higher quality than the studies covered by the newspapers.

They also found that the observational studies that appeared in the journals were superior to the observational studies covered by the newspapers. For example, they had larger sample sizes and were more likely to be longitudinal rather than cross-sectional.

In one sense, these results are not a surprise. We could hardly have expected newspaper reporters to be as good a judge of study quality as the editors of prestigious medical journals. The authors, like many before them, call for more scientific literacy training for newspaper reporters, but it's hard to be optimistic that this will happen.

What criteria do the reporters use in selecting studies to write about? I was struck by the fact that observational studies resemble anecdotes more than RCT studies do. In addition, the newspapers chose observational studies with smaller sample sizes. These results could be driven by the base rate fallacy—the fact that the average person finds anecdotes more convincing than statistical analyses of much larger samples. In fact, the lead paragraph of these stories is often a description of some John or Jane Doe who received the treatment and got better. The results could mean either that reporters fall victim to the base rate fallacy, or that they think their readers are more interested in anecdotal evidence.

You may also be interested in reading:

Letters, . . . We Get Letters

Death By Anecdote, Part 1

Letters, . . . We Get Letters

When it comes to persuading the Pittsburgh Post-Gazette to publish my letters, I'm batting about .050. My one success, almost ten years ago, was a note about music. Of course, newspapers get many letters, and it's their right to choose which ones to publish. Ordinarily, I wouldn't bore you by griping about their decisions. However, I've managed to persuade myself that my latest experience may be of interest to other letter writers, so I'm going to risk playing the fool.

My story begins on December 8, when the P-G published Molly Rush's letter advocating single payer health care. This was followed on December 16 by a reply from Jim Roth—a pharmaceutical salesperson(!)—implying that single-payer is too costly and denies health care to some citizens. If you're going to continue, you should stop and read Mr. Roth's letter.

The next day, I wrote the following:

Mr. Roth notes that all countries with single-payer finance it with a value added tax. However, the type of tax used to fund health care is irrelevant. The important point is that single payer costs those countries considerably less than our complex system of public and private insurance. According to a 2013 report of the Organization for Economic Cooperation and Development (OECD), the US currently spends on average $8,508 per person each year on health care, compared to an OECD average of $3,322. Yet the US is 26^th out of 40 OECD countries in life expectancy. The amount Americans spend on health care due to the combined burden of taxes, insurance and out-of-pocket costs would be greatly reduced under single-payer.

Mr. Roth claims that people in single-payer countries have longer wait times for elective surgery and are sometimes denied such care. This depends on the country and what you consider “elective surgery.” US insurance companies also refuse to cover some elective procedures. However, if these were serious problems, you would expect residents of single-payer countries to be dissatisfied with their country's health care system. A 2013 survey by the Commonwealth Fund compared consumer satisfaction in the US to nine European countries and Canada, all with single-payer. Americans were by far the most dissatisfied,with 75% saying the system needs fundamental changes or should be completely rebuilt.

Finally, Mr. Roth suggests that we could lower health insurance costs by allowing it to be sold across state lines. It is true that if some states were to deregulate health insurance and if residents of any state were allowed to buy that product, premiums might come down. But those people would be buying insurance with little value should they become seriously ill. The Affordable Care Act is intended to prevent exploitation of consumers by establishing a baseline definition of adequate health insurance.

Of course, the primary purpose of single-payer is not just to save money, but to save the lives of some of the millions of Americans who are currently uninsured.

On December 29, the P-G published two replies to Mr. Roth. Both offered primarily anecdotal evidence suggesting that at least one family—the author's—had lived in a single-payer country and was satisfied with their health care system. The main difference between them is that the first referred to the British system and the second the Dutch. While both were well-written and persuasive, I thought they were redundant, and might have been better supplemented by my data referring to larger numbers of people and countries.

It's possible my letter was rejected because it is poorly written or exceeds their 250-word limit. However, Mr. Roth's letter, at 318 words, also breaks this rule, as do many others they publish. They could easily have edited my letter. Clearly, exceeding the word limit was a mistake. In retrospect, I should have dropped the third paragraph.

My hypothesis, based on this and other previous experiences, is that my letter was rejected because it contained too much data. Imagine an experiment in which parallel letters to the editor are sent to a random sample of newspapers. Both letters would make exactly the same points, but one would support each point with research, while the other would support them with anecdotes or merely claim that these were the author's personal opinions. My guess is that fewer of the data-driven letters would be published.

I have two possible, though somewhat inconsistent, explanations for my hypothesis. The first assumes that the editors wanted to present the single payer argument sympathetically. It's based on a common cognitive error known as the base-rate fallacy. People find anecdotal evidence more persuasive that statistical base rates, even though the base rates summarize data from larger, more representative samples. The people who made the decision may have found the two letters they published to be more persuasive than mine.

My second explanation makes the reasonable assumption that the gatekeepers at the P-G are opposed to single-payer. If so, they may assume that my inclusion of data makes the letter too persuasive. That is, they may be willing to acknowledge that there are some Pittsburghers who favor single-payer, but it may be unrealistic to expect them to publish statistics suggesting that the arguments of single-payer advocates are factually correct.

I hope I'm wrong. I really want to encourage the use of research evidence to change the health care system, and society in general, for the better. If this strategy is counterproductive, that's genuinely disturbing.

You may also be interested in reading:

Death By Anecdote, Part 1

Are You Feeling Lonely?

Death By Anecdote, Part 2

Please read the first part of this article.

When a story is false, we run into a third problem: Even when misinformation is corrected, many people continue to believe it. For example, in one study, participants from Australia, Germany and the United States were asked about arguments made by the US government favoring the invasion of Iraq which were subsequently retracted, such as the claim that Iraq possessed weapons of mass destruction (WMDs). The Australian and German students were sensitive to retraction; that is, when they knew a claim had been retracted, they tended not to believe it. The Americans, however, were insensitive to retraction; on the whole, they tended to believe statements that they knew had been retracted about as much as statements they did not know had been retracted.

Paradoxically, attempts to correct misinformation can lead to a backfire effect, in which people are more likely to believe false information after it has been debunked. Sometimes this is due to increased familiarity with the claim as a result of its being repeated during the retraction. However, backfire effects are most likely to occur when the original claim is consistent with the ideology of the person who believes it. Not surprisingly, Republicans were more likely than Democrats to continue to believe that Iraq had WMDs even after the Bush administration admitted they didn't. When ideological backfire effects occur, people can be suspicious of the motives of the person or organization doing the correcting (often the news media) and discount the retraction.

Lewandowsky and others make three suggestions for successfully correcting misinformation: (1) warn people at the time of initial exposure that the information is suspect, (2) repeat the retraction several times, focusing only on the new, correct information, and (3) provide a plausible alternative explanation for the previous false belief. In the case of ideologically motivated false beliefs, they make a fourth recommendation: Affirm the target's ideology before attempting to correct the false belief. The first two remedies require news media cooperation that the Obama administration is unlikely to receive. Since no single alternative explanation accounts for all the anti-ACA anecdotes, providing plausible explanations for false beliefs requires extensive investigation of individual cases. The fourth suggestion would seem to require a statement like this: “We agree with you that Obamacare is a disaster, but in this case, you are wrong because . . .” 

Here is Lewandowsky discussing misinformation and its correction in the important area of climate change.

I see no magic bullet here. The best solution for the administration may be to appeal to the value Americans place on self-reliance and encourage them to explore their health care options for themselves. But they can't do that until their website is fixed. If they lose the cooperation of young people, the market will suffer from adverse selection—not enough people paying into the system, and too many older, sicker people drawing it down. Insurance rates could increase dramatically next year, just in time for the 2014 elections. Then Obama will see how much fun it would be if the Tea Party controlled the Senate as well as the House.

The HealthCare.gov fiasco is bad news for single payer advocates too. I'm afraid most people don't realize that it was Obama's reliance on an "overly-complicated, market-based Republican health care plan" that made the website so difficult to set up. They may simply conclude that government can't do anything right.

You may also be interested in reading:

Death By Anecdote, Part 1

Policy By Anecdote

Climate of Mistrust

Death By Anecdote, Part 1

You can lie with statistics, but a well-chosen anecdote is much more effective.

Here's something to look forward to, right along with your next colonoscopy.  The New York Times reports that the Republican Party plans to carry out a sustained, organized attack on the Affordable Care Act (ACA) for the next year, in the hope of gaining an advantage in the 2014 elections. The Republican campaign, described as a “multilayered sequenced assault,” is outlined in the House Republican Playbook, a 17-page strategy document prepared by their House leadership. It lists a series of talking points such as: “Because of Obamacare, I lost my insurance,” “Obamacare increases health care costs,” and “The exchanges may not be secure, putting personal information at risk.” House members are advised to collect anecdotes from constituents in support of these talking points through social media, letters and visits to their home distract. A new website, gop.gov/yourstory, centralizes the collection of these anecdotes. Republicans are instructed in the use of “messaging tools” for disseminating the stories, for example, a sample op-ed for submitting to local newspapers.

The idea is to flood the media with anecdotes in support of a particular talking point. If there is an effective counterresponse from Democrats, they will shift immediately to a different talking point. Topics waiting in the wings for possible use include insurance “rate shocks,” threats to being able to keep your doctor, and possible changes to Medicare Advantage policies.

The Republicans recognize that anecdotes can have a powerful influence on public opinion. When making inferences, people use judgmental heuristics, or mental shortcuts to make decisions quickly and easily. The use of heuristics is automatic and unconscious. They usually lead to correct inferences, but they sometimes lead us astray.

One inference we often make is to estimate the size of a category or the frequency of an event. For example, how many Americans are being harmed by the ACA? The availability heuristic suggests that the size of a category is judged by the ease with which examples can be brought to mind. Examples are more easily retrieved from memory if they are concrete rather than abstract, if they are dramatic and interesting, or if they happened recently or nearby. Personal experiences are particularly salient. When people are asked to estimate the frequency of various causes of death, they overestimate homicides and auto accidents, but underestimate strokes and diabetes. Clearly, their estimates are influenced by media coverage.

The problem gets worse when you consider the base rate fallacy, which states that people are inattentive to population statistics, and their judgments are not sufficiently affected by them. In one study, participants were given vivid stories about misbehavior by prison guards or welfare recipients. Attitudes toward these groups were equally negatively affected regardless of whether they were told that the anecdotes were typical of the population, not typical of the population, or they were given no base rate information. In another study, college students' intentions to take courses were affected by single brief face-to-face comments from a stranger, but hardly at all by statistical summaries of the course evaluations of much larger numbers of students who had previously taken the course.

The availability heuristic and the base rate fallacy suggest that even if people are given accurate information suggesting than the story is unrepresentative, their false impression is unlikely to be corrected. There was heavy media coverage of the first wave of anecdotes from people who claimed that their insurance costs went up due to Obamacare. When critics examined them more closely, many of these anecdotes were found to be misleading. Insurance companies cancelled policies and raised rates long before the ACA. The percentage of people whose policies were cancelled was small. Some of these people were able to get equal or better insurance through the exchanges without paying more. However, the corporate media can't be counted on to investigate anecdotes before airing them, and the debunking stories seldom receive anywhere near the attention given to the original report.

The Obama administration has apparently decided that the best defense is a good offense, so they are responding with anecdotes of their own—so-called Obamacare “success stories.” While this may be the best they can do under the circumstances, they are unlikely to get much cooperation from the corporate media in publicizing these stories. The media don't cover successful airplane landings--unless you land it in the Hudson River. Meanwhile, the administration and the media have largely overlooked another potential source of much more tragic stories: the 5.2 million people who are being denied health insurance entirely because they happen to live in states where Republican governors and legislatures have blocked Medicaid expansion. But to the corporate media, an upper middle class person losing a few dollars a month is much more newsworthy than a poor person losing his or her life.

But remember, the plural of anecdote is not data. To be continued.

You may also be interested in reading:

Death By Anecdote, Part 2

Policy By Anecdote

The Coverage Gap:  The Real Failure of Obamacare

Appeal to an Unlikely Scenario

I mentioned in a previous post that I am currently taking an online course in critical thinking through Coursera. This week, one of our topics is fallacies of relevance. A fallacy of relevance is an argument in which the premises are of questionable relevance to the conclusion. An example is an appeal to an unlikely scenario.

Yesterday, the Senate Judiciary Committee held a hearing on Senator Dianne Feinstein's (D-CA) proposal to reinstate the ban on assault weapons and to ban high capacity magazines (clips holding more than ten rounds). Most of the media attention went to two celebrities who testified, former Representative Gabrielle Giffords and NRA president Wayne LaPierre. All three witnesses who argued against the bill maintained that assault weapons are needed by law-abiding citizens for self-defense in the event of an attack by armed criminals. However, as Senator Sheldon Whitehead (D-RI) pointed out, none of the anecdotes of successful self-defense involved an assault weapon or high-capacity magazine. They all utilized traditional weapons such as pistols or shotguns, access to which is not threatened by the legislation.

However, the pro-gun lobby's imagination is not constrained by reality, so one of the witnesses, Gayle Trotter of the Independent Women's Forum, entertained the Senators with the following hypothetical scenario:

An assault weapon in the hands of a young woman defending her babies in her home becomes a defense weapon. And the peace of mind that a woman has as she's facing three, four, five violent attackers, intruders in her home, with her children screaming in the background, the peace of mind that she has knowing that she has a scary-looking gun gives her more courage when she's fighting hardened, violent criminals. If we ban these weapons, you are putting women at a great disadvantage . . .

Appeals to unlikely scenarios should be evaluated by assessing the positive or negative consequences put forward in the scenario, weighted by their probability of occurrence—which in an unlikely scenario is low. They are compared to the positive or negative consequences of the alternative course of action, in this case, not banning assault weapons, weighted by their probability of occurrence. The conclusion is obvious.

Unfortunately for Ms. Trotter, she agreed to be interviewed on The Last Word with Lawrence O'Donnell last night. He effectively deconstructed her argument and made her look quite silly.

Another example of the mischief done by appeals to unlikely scenarios is the defense of torture which asks us to imagine that we have captured a terrorist who refuses to tell us the location of a bomb which will kill a large number of people. Given limited time, would we not torture him to find the bomb? One of the reason people find these unlikely scenarios to be plausible is that they are often presented in fictional films and television programs, and we later remember these stories as having been true.

Bend Over

We have two definitely connected phenomena, often treated as separate and unconnected: a growing lawlessness in the financial sector, and an expanding, repressive, increasingly lunatic police apparatus trained at the poor, and especially the nonwhite poor.

Matt Taibbi

The Occupy Movement is expected to resume its protests of economic inequality this Summer, especially during the Elephant and Jackass political gatherings. But protesters, beware. The Supremes may have an unpleasant surprise for you.

The depressingly predictable 5-4 conservative majority has ruled that the police may strip search anyone they arrest for any offense, no matter how minor. They don't need a reason to suspect that the person is carrying drugs or weapons. As lead singer Justice Anthony Kennedy put it: “Every detainee who will be admitted to the general population may be required to undergo a close visual inspection while undressed.” About 13 million Americans are arrested each year.

Justice Stephen Breyer, writing for the dissenters, said that strip searches are “a serious affront to human dignity and to individual privacy,” and should only be used when police have a reasonable suspicion that the person is carrying contraband.

The case, Florence v. County of Burlington, involved a black man, Albert W. Florence, a passenger in BMW driven by his wife, who was pulled over for speeding. He was arrested when the police checked their computer and discovered he had an unpaid parking ticket. In fact, he had paid the ticket; he was the victim of a data input error. He was kept in jail for eight days until the police acknowledged their error. During that time he was strip searched twice.

Are blanket strip searches necessary to detect contraband? There was an interesting difference in the approach to evidence taken by the Court's majority and minority spokesmen. Justice Breyer cited a study of 23,000 people arrested in Orange County, New York, in which only five people were found to have forbidden materials concealed in their underwear or body cavities. In four of these cases, police had reasonable suspicion to conduct the search, resulting in a “hit rate” of one in 23,000. Justice Kennedy, on the other hand, cited an anecdote about a man arrested in Washington who “managed to hide a lighter, tobacco, tattoo needles and other prohibited items in his rectal cavity.” (Even if this story is true—and I have doubts—a strip search might not have been necessary. The police could simply have ordered the man to sit down on a hard chair.)

Social psychologists have discovered that vivid anecdotes are more memorable than statistical data, so you'll probably remember Justice Kennedy's story long after you have forgotten the results of Justice Breyer's study.

One detail that was overlooked in some news reports is that the Obama administration filed a amicus brief with the court in favor of the unlimited strip search policy. The Solicitor General wrote that corrections officials should have “appropriate latitude to implement those policies and practices they deem necessary to preserve institutional security.”

In reality, strip searches of prisoners have little to do with detecting contraband. Degrading and humiliating searches are a part of the process of exercising control over newly arrested prisoners. This technique was widely used against Occupy protesters last Fall.

We are facing a indefinite period of increasing social inequality, accompanied by elimination of the social safety net which keeps many people alive. This is one of several converging decisions that will make it more difficult for people who hope to exercise their right to protest these conditions. It follows recent decisions by the Supremes that police officers who serve invalid warrants cannot be sued and that police do not have to give Miranda warnings pefore interrogating suspects. Meanwhile, the police are being given high tech weapons to threaten and punish crowds. The Department of Homeland Security has greatly increased its warrantless surveillance of citizens, a power that has been used to infiltrate the Occupy movement, deliberately blurring the distinction between domestic terrorists and peaceful protesters. The Justice Department has announced that they can detain U. S. citizens accused of terrorism indefinitely without any due process protection. Finally, Attorney General Eric Holder has announced that the President claims the right to order the assassination of American citizens alleged to be terrorists without any judicial oversight.

In this instance, the message seems to be particularly directed at young women: “Peaceful protesters can be arrested at any time. If you are arrested, this is what will happen to you. Maybe you'd better stay home.”

Policy by Anecdote

On Monday, the Pennsylvania Department of Public Welfare (DPW) announced that, as of May 1, 2012, they will begin an asset test to determine if citizens are eligible for aid under SNAP, the Supplemental Nutritional Assistance Program, better known as food stamps. People under 60 will be ineligible for food stamps if they have more than $2000 in personal savings or other assets. Three items do not count toward the $2000 limit: their home, their retirement benefits, and one car.

In other words, if you lose your job and have $20,000 in the bank, you are not eligible for food stamps until you've spent $18,000 of your savings. This is not just mean-spirited, it is counterproductive. Prevailing theory and research on poverty suggests that poor people are trapped in a vicious cycle of poverty in which low income results in little or no savings, leading to intermediate outcomes such as poor health care and less education, which ultimately cause poverty to persist. Forcing people to be destitute before they qualify for public assistance makes it less likely they will ever break out of cycle. Since poverty has so many social costs, i.e., crime and urban decay, the new policy is not only harmful to poor people, but to society as well.

Pennsylvania's action is unusual during these economic hard times, since many states have eliminated their asset tests. The Rendell administration eliminated Pennsylvania's asset test in 2008. A total of 35 states have no asset test. Pennsylvania will join ten other states allowing maximum assets of $2000. The other four states have higher maximums.

What's going on? The change doesn't seem to be motivated by a desire to save money. SNAP is supported by the federal government. DPW only pays the cost of administering the program. But those costs will go up, because case workers will now have to survey each applicant's assets to determine whether they are eligible. If enforcement of the $2000 limit turns out to be sporadic or inconsistent, as certainly seems possible, this may permit prejudice and discrimination to influence whether people are eligible for food stamps.

What's really going on? Deep into the Post-Gazette article, we find a clue:

Some say the sudden call for an asset test coincides with the widely publicized story late last year of a Michigan man who won millions in a lottery but retained his food stamps.

DPW spokesperson Ann Bale acknowledged that they “constantly” hear from citizens complaining of food stamp abuse. However, Pennsylvania has one of the lowest food stamp fraud rates in the nation—one-tenth of 1%. Perhaps DPW gets phone calls not because of actual fraud, but due to stories such as this one on Fox News.

This case reminds me of Ronald Reagan's campaign fable about a Chicago “welfare queen” who allegedly cheated the government out of $150,000 and drove a new Cadillac. The story inspired a hit country and western song, “Welfare Cadillac,” by Guy Drake. It was a good story, but it was exaggerated and entirely unrepresentative of welfare recipients in this country.

Social psychologists point out that people are more easily persuaded by anecdotes than by statistical evidence. Belief in a statement, such as, “the food stamp program is widely abused,” depends largely on the ease with which we can recall or imagine examples that support the statement. Media exposure is a major source of the availability of anecdotes.

The anecdotes don't have to be true. For example, Melanie Green and Tim Brock did a study in which participants read a story, “Murder at the Mall,” about a young girl being killed by a recently released mental patient. Some were told the story was true; others were told it was a fictional short story. Both groups were more opposed to releasing former mental patients into the community than a control group that didn't read the story at all. There was no difference between those who were told that the story was true and those who were told it was made up.

I suspect Reagan's “welfare queen” played a role in Bill Clinton's eventual acquiescence to the pressure to "end welfare as we know it.” The problem, of course, is that you can make up an anecdote to “prove” virtually anything. In this blog, I'll argue that that public policy should be based on research evidence.

Update (2/1/12)

On February 1, the DPW announced a change in the asset limits.  People under 60 will be able to retain assets of up to $5,500, while those who are over 60 or disabled can keep up to $9,000.