Skip to main content

Geisinger and Microsoft research finds widespread aversion to randomized experiments in healthcare and beyond

DANVILLE, Pa. – Randomized controlled trials (RCTs), commonly referred to as A/B tests, have long been the gold standard for scientific evidence in medicine and are increasingly used in other fields as diverse as education, government policy, and poverty reduction. A/B tests help to determine whether a new treatment or policy helps or hurts compared to an alternative. A new study by researchers at Geisinger and Microsoft Research published last month in the Proceedings of the National Academy of Sciences sheds light on previous controversies over A/B tests, finding that people often approve of an untested policy (A or B) being given to everybody but object to an A/B test of those same unobjectionable policies designed to determine which one works best.

The series of 16 studies with 5,873 participants from three diverse populations spanning nine domains in healthcare and beyond suggests an important obstacle to evidence-based policy and practice.

One study involved differing methods of communication aimed at educating providers on best practices for avoiding infections from catheterizations. One group of participants was told that the hospital director decided to print a checklist of safety steps for clinicians on the backs of their ID badges. Another group was told that the director instead decided to print the checklist on posters that will be hung on the wall. A final group of participants was told that the director thought of both these ideas and decided to conduct an experiment in which patients are randomly assigned to a doctor with a badge or a doctor who works in a room with a poster. They were also told that once the hospital director learns which policy is best at reducing infection rates, he or she will implement it for everyone.

Among the group who read about the director who implemented the badge policy, very few judged this decision to be inappropriate. So, too, with those who read about the director who implemented the poster policy. But many who read about the director who ran an A/B test to compare the badge and poster ideas thought this decision was inappropriate.

According to the study’s lead author Michelle Meyer, PhD, JD, Assistant Professor and Associate Director, Research Ethics, in Geisinger’s Center for Translational Bioethics and Health Care Policy, A/B tests necessarily treat people differently, and potentially unequally—at least temporarily. “But strikingly, we found that people still object to A/B tests even when the alternative was still that different people would randomly get different treatments,” Meyer said. “Because of who they happen to have trained with, or what journal articles they happen to have read, some doctors prescribe one drug for things like high blood pressure—we called it Drug A—and others prescribe Drug B. Patients who are prescribed a blood pressure medication in the ER or in a walk-in clinic are basically being ‘randomly assigned’ to one drug or the other due to chance. Yet many of our participants find that unobjectionable but object to having their treatment formally assigned by an A/B test that would figure out which drug is best so it can be offered to everyone.” 

Follow-up studies found the same widespread disapproval of A/B tests despite approval of just giving everyone either A or B. Results were consistent in a variety of other domains, including consumer genetic testing products, autonomous vehicle design, and social policy programs. Doctors and participants with advanced degrees were no different than other participants. 

In examining the reasons participants gave for objecting to the A/B tests, the researchers found that people often assume, incorrectly, that experts already know “what works.” The researchers concluded that more research is needed to better understand why people object to experiments that compare unobjectionable policies and how to address those concerns.

About Geisinger
Geisinger is committed to making better health easier for the more than 1 million people it serves. Founded more than 100 years ago by Abigail Geisinger, the system now includes 10 hospital campuses, a health plan with more than half a million members, a Research Institute and the Geisinger Commonwealth School of Medicine. With nearly 24,000 employees and more than 1,700 employed physicians, Geisinger boosts its hometown economies in Pennsylvania by billions of dollars annually. Learn more at Facebook, Instagram, LinkedIn and Twitter.

geisinger placeholder


For media inquires:

Matthew Van Stone
Media Relations Director

570-808-3344
mrvanstone1@geisinger.edu