Dr. John Ioannidis’ 2005 article “Why Most Published Research Findings are False” is a provocative reflection on how vulnerable research can be to bias. With citation in over 1,400 papers over the last 9 years (http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0020124 ) Ioannidis has inspired and incited a lot of discussion. Although written for the medical community, the observations and concerns have as much application and importance to education as it does to medical research.

Finding what you want to see

Dr. Ioannidis defines bias as a “combination of various design, data, analysis, and presentation factors that tend to produce research findings when they should not be produced. Selective or distorted reporting is a typical form of such bias. ” Following are 6 aspects of bias that Ioannidis explores (with examples) in his article:

1. The smaller the studies conducted in a scientific field, the less likely the research findings are to be true.
2. The smaller the effect sizes in a scientific field, the less likely the research findings are to be true. This is particularly interesting given the controversy over the last year with John Hattie’s work.
3. The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true.
4. The greater the flexibility in designs, definitions, outcomes and analytical modes in a scientific field, the less likely the research findings are to be true.
5. The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true.
6. The hotter the scientific field (with more scientific teams involved), the less likely the research findings are to be true.

To help reflect on whether bias is an issue to be concerned about in your context, here are three questions to consider:

• How many articles or projects report insignificant findings? Knowing what doesn’t work is as important and valuable as knowing what does work. If there are few (or no) insignificant findings, this could suggest data-mining/dredging (conducting analysis until positive results could be found), negative results were withheld, observations were vulnerable to the Hawthorn effect or observations were vulnerable to confirmation bias.
• How consistent is the use and implementations of strategies and metrics across studies? Many of these considerations may be modified to align with the culture of a board. While this may strengthen the face validity of a study, it can weaken its generalizability or comparability to other studies. It is Dr. Ioannidis’ contention that the more meaningful findings emerge from research initiatives that are large scale, have multiple independent teams engaged in the inquiry and where consistency is ensured in the design, metrics and analysis.
• Are there any dissenting or critical voices for strategies or findings? Ioannidis reflects that research in popular fields of study may be more vulnerable to “rapidly alternating extreme research claims and extremely opposite refutations” as research teams build their reputations by promoting their most positive results and use negative results as a challenge to other research teams.