09/22/2025
MEDICAL EVIDENCE: A PRIMER
You may have noticed that I don’t make pronouncements on this page without citing the appropriate medical evidence. Anyone who tells you “because I said so” probably shouldn’t be listened to.
The scientific method is an empirical process for the acquisition of knowledge and involves: (1) observation, (2) forming a question, (3) forming a hypothesis, (4) designing and conducting an experiment to test the hypothesis while controlling the variables, (5) collecting and interpreting the data, and (6) forming a theory. The information gained is then continually refined by repeating the process (https://en.wikipedia.org/wiki/Scientific_method).
Traditionally, scientific journals have been the way in which scientists make other scientists aware of their research findings. Then, medical/scientific journalists disseminate information of perceived public interest. This publication process involves peer review, a process where the article is sent out to peers of the author(s) (other scientists) for review to ensure scientific integrity and to make sure that the conclusions are appropriate. However, unethical scientific journals have existed for a long time, leading to Beall’s List of Potentially Predatory Journals and Publishers (https://beallslist.net/). Unethical researchers (and sometimes ethical researchers, unwittingly) can pay exorbitant fees to have their research published on the internet without appropriate peer review.
The COVID-19 pandemic ushered in the proliferation of the preprint server. In an effort to get information “out there” in a timely fashion, researchers could upload their studies to the internet without peer review and before being accepted for publication in an established scientific journal. While a great deal of beneficial information was disseminated in this way, the public was increasingly exposed to subpar research.
The “publish or perish” culture of academia, the internet, and the proliferation of artificial intelligence (AI) have resulted in a surprising amount of fraud in medical research. It is estimated that 24% of medical studies include falsified data, altered images, significant plagiarism, or are created entirely by AI (https://www.science.org/content/article/fake-scientific-papers-are-alarmingly-common and https://www.nature.com/articles/d41586-023-02299-w). In a 2009 meta-analysis of studies about surveys of researcher behavior, 1.97% of scientists admitted to having fabricated, falsified, or modified data or results at least once –a serious form of misconduct by any standard– and up to 33.7% admitted other questionable research practices (https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0005738).
The bottom line is that a study is not a study is not a study. Good medical research is reproducible, and medical practice is usually not changed until the results have been reproduced (https://en.wikipedia.org/wiki/Replication_crisis). Significant change in medical practice is not determined by a simple popularity contest. While it is important to note how many studies support a change in medical practice versus those that don’t, it is important to look at the quality of the studies.
Case in point: Early in the pandemic, several studies and experts (see the accompanying infographic, which outlines the hierarchy of medical evidence – the only evidence worse than expert opinion is no evidence) suggested that hydroxychloroquine and ivermectin were effective for treating COVID-19. While these studies served the purpose of stimulating more conclusive research, many of them were later retracted by the journals in which they were published as having been falsified, were inappropriately biased (e.g., funded by the manufacturer of the medication in question), or included faulty methodology. Multiple well-done meta-analyses overwhelmingly showed that these medicines were not helpful for COVID-19 (https://www.nature.com/articles/s41591-021-01535-y). These papers lost both the popularity and scientific rigor contests. Now, Texas is inappropriately intertwining politics and medicine by passing House Bill 25, which makes ivermectin available without a prescription (https://www.texastribune.org/2025/08/27/texas-legislature-ivermectin-otc-prescription/).
While scientific evidence, appropriate medical oversight, and individual freedoms can coexist, the appropriate balance has been elusive.
