Why Most Published Research Findings Are False

Abstract: Ioannidis JPA (2005), PLoS Med 2(8): e124: “There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.” (PLosMedicine… thanks, adam)
Technorati tags:

Counting the Civilian Dead in Iraq

“Thousands of Iraqi civilians have died as a result of violence since the war began in 2003. But it’s not known exactly how many died, where and at whose hands.

There have been several efforts to count the war’s toll on civilians, yielding reports ranging from 24,000 to 128,000 from last fall through last month. Compounding the complexity, all of these numbers were collected differently and count different things, so they aren’t directly comparable. For example, the widely cited number last month of about 25,000 counts only violent deaths that have been reported to the media. Meanwhile, a study conducted last fall that found 100,000 deaths arrived at that figure by calculating ‘excess’ deaths — all deaths, including those from illness and accidents, were included, but deaths from a comparable prewar period were subtracted out.

The uncertain and inconsistent numbers help explain why the civilian death toll — caused by criminals, terrorists, insurgents and soldiers from all sides — hasn’t been given much attention in major U.S. media, even as many newspapers report every death of U.S. soldiers in Iraq and, last December, headlined incomplete tsunami death-toll numbers for weeks.” (WSJ Online)


I was recently pointed to the Wikipedia article on this fascinating Japanese phenomenon. Hikikomori is a Japanese term referring to acute social withdrawal by adolescents or young adults.

“While there are mild and extreme degrees, the Japanese Ministry of Health defines a hikikomori as an individual who refuses to leave their parents’ house, and isolates themselves away from society and family in a single room for a period exceeding six months, though many such youths remain in isolation for a span of years, or in rare cases, decades. Many cases of hikikomori may start out as school refusals, or tohkohkyohi in Japanese. According to estimates by psychologist Saito Tamaki, who first coined the phrase, there may be 1 million hikikomori in Japan, 20 percent of all male adolescents in Japan, or 1 percent of the total Japanese population. Surveys done by the Japanese Ministry of Health as well research done by health care experts suggest a more conservative estimate of 50,000 hikikomori in Japan today. As reclusive youth by their very nature are difficult to poll, the true number of hikikomori most likely falls somewhere between the two extremes.

…Sometimes referred to as a kind of social problem in Japanese discourse, the hikikomori phenomenon has a number of possible contributing factors — young adults may feel overwhelmed by modern Japanese society, or be unable to fulfill their expected social roles as they have not yet formulated a sense of personal tatemae and honne needed to cope with the daily paradoxes of adulthood. The dominant nexus of the hikikomori issue centers around the transformation from young life to the responsibilities and expectations of adult life — indications are that advanced capitalist societies such as modern Japan are unable to provide sufficient meaningful transformation rituals for promoting certain susceptible types of youth into mature roles within society.”

Middle class affluence allows many families to support their isolative child indefinitely. There may be a contribution from the particularly Japanese codependent collusion (amae) between mother and son, making an effective response to the isolationism more difficult. And the decreasing job security in the Japanese corporate environment, combined with extreme performance pressure, may contribute to making social withdrawal rampant.

There appears to be considerable debate about whether hikikomori should be considered a sociological or psychological phenomenon. A variety of psychological diagnoses could contribute to its phenomenology, including anxiety disorders including agoraphobia and social phobia (social anxiety disorder); avoidant personality disorder; and depression. I think the social vs. psychological debate is a specious one, as there is likely a coalescence of internal and social factors at play here. There are a number of so-called “culture-bound syndromes” in which a behavioral symptom pattern appears to be particularly prevalent, and unique, in a given culture. These include amok, latah, wendigo, etc. I usually see them as variants of common psychiatric disturbances brought to the fore by the particular social stresses of a given culture.

Neuroscience gears up for duel on the issue of brain versus deity

To the editor: “The argument over evolution versus intelligent design, discussed in your News story ‘Day of judgement for intelligent design’ (Nature 438, 267; 2005), is a relatively small-stakes theological issue compared with the potential eruption in neuroscience over the material nature of the mind.

Siding with evolution does not really pose a serious problem for many deeply religious people, because one can easily accept evolution without doubting the existence of a non-material being. On the other hand, the truly radical and still maturing view in the neuroscience community that the mind is entirely the product of the brain presents the ultimate challenge to nearly all religions.” — Kenneth Kosik, Nweuroscience Research Institute, U.C. Santa Barbara (Nature)

One Third Of Patients Who Stop Treatment For Schizophrenia Early Do So Due To Poor Response

These are the findings of a study by a group of researchers at the pharmaceutical firm Eli Lilly. I haven’t read the study, only the press coverage to which this link points, but their findings are summarized thus: “Of the 866 patients who stopped treatment, 36% (315/866) did so because the treatment was felt not to be effective or because their symptoms worsened. Only 12% of patients who stopped treatment early did so because of adverse events such as dizziness, fatigue, vomiting or weight gain.” They make the point that discontinuation due to patient perception of poor response tends to occur early in the course of treatment and that patients who experience an early response were 80% more likely to complete treatment.

I would point out the likelihood of significant bias in this industry-sponsored study. Eli Lilly are the manufacturers of Zyprexa [olanzapine], one of the most-used ‘atypical’ [new-generation] antipsychotic medications which were touted as wonder drugs solving all the substantial toxicity issues of the older generation of antipsychotics, until we recognized that worrisome side effects such as drastic weight gain and glucose intolerance were common with this newer generation of agents. It is in Lilly’s interest in selling antipsychotic medications, of course, to establish, as the current study concludes, that side effects are not as worrisome as they are usually considered and that treatment should be more aggressive from the outset. But of course patient discontinuation early in the course of treatment is not likely to be from concern about side effects, because the worrisome metabolic changes caused by these medications are insidious and slow to develop!

I agree with the need for treating terrifying psychotic symptoms with alacrity to relieve distress and establish a treatment alliance. Certainly a schizophrenic patient beset by tormenting voices or paranoid ideation will not stick with treatment that does not offer rapid relief. However, I suspect the crucial variable which the study does not address is that patients who do not perceive that their treatment is effective are often patients with one particular symptom of severe psychotic illness — anosognosia, the denial of illness and lack of recognition of need for treatment at all. A patient cannot recognize benefit from a medication treatment if s/he does not recognize the symptoms and the need for treatment in the first place. Psychiatrists are very familiar with psychological denial, but I also think anosognosia has a neurologically-based, organic component as part of the brain dysfunction in schizophrenia and other psychotic illnesses, although this is controversial within my field.

In any case, it is emphatically not drug treatment, aggressive or not, which treats this symptom. Instead of funding research trying to sell more drug doses, Eli Lilly should be endowing a foundation for the dying art of talking to the sickest of our patients, creating and maintaining a relationship allowing skillful and sustained entry into the world of a terrified and tormented soul. This is itself the most important healing tool.