New research shows disapproval of randomized controlled trials may impede ability to learn which policies and treatments work best
,
Geisinger and Microsoft research finds widespread aversion to randomized experiments in healthcare and beyond
DANVILLE, Pa. – Randomized controlled trials (RCTs), commonly referred to as A/B tests, have long been the gold standard for scientific evidence in medicine and are increasingly used in other fields as diverse as education, government policy, and poverty reduction. A/B tests help to determine whether a new treatment or policy helps or hurts compared to an alternative. A new study by researchers at Geisinger and Microsoft Research published last month in the Proceedings of the National Academy of Sciences sheds light on previous controversies over A/B tests, finding that people often approve of an untested policy (A or B) being given to everybody but object to an A/B test of those same unobjectionable policies designed to determine which one works best.
The series of 16 studies with 5,873 participants from three diverse populations spanning nine domains in healthcare and beyond suggests an important obstacle to evidence-based policy and practice.
One study involved differing methods of communication aimed at educating providers on best practices for avoiding infections from catheterizations. One group of participants was told that the hospital director decided to print a checklist of safety steps for clinicians on the backs of their ID badges. Another group was told that the director instead decided to print the checklist on posters that will be hung on the wall. A final group of participants was told that the director thought of both these ideas and decided to conduct an experiment in which patients are randomly assigned to a doctor with a badge or a doctor who works in a room with a poster. They were also told that once the hospital director learns which policy is best at reducing infection rates, he or she will implement it for everyone.
Among the group who read about the director who implemented the badge policy, very few judged this decision to be inappropriate. So, too, with those who read about the director who implemented the poster policy. But many who read about the director who ran an A/B test to compare the badge and poster ideas thought this decision was inappropriate.
According to the study’s lead author Michelle Meyer, PhD, JD, Assistant Professor and Associate Director, Research Ethics, in Geisinger’s Center for Translational Bioethics and Health Care Policy, A/B tests necessarily treat people differently, and potentially unequally—at least temporarily. “But strikingly, we found that people still object to A/B tests even when the alternative was still that different people would randomly get different treatments,” Meyer said. “Because of who they happen to have trained with, or what journal articles they happen to have read, some doctors prescribe one drug for things like high blood pressure—we called it Drug A—and others prescribe Drug B. Patients who are prescribed a blood pressure medication in the ER or in a walk-in clinic are basically being ‘randomly assigned’ to one drug or the other due to chance. Yet many of our participants find that unobjectionable but object to having their treatment formally assigned by an A/B test that would figure out which drug is best so it can be offered to everyone.”
Follow-up studies found the same widespread disapproval of A/B tests despite approval of just giving everyone either A or B. Results were consistent in a variety of other domains, including consumer genetic testing products, autonomous vehicle design, and social policy programs. Doctors and participants with advanced degrees were no different than other participants.
In examining the reasons participants gave for objecting to the A/B tests, the researchers found that people often assume, incorrectly, that experts already know “what works.” The researchers concluded that more research is needed to better understand why people object to experiments that compare unobjectionable policies and how to address those concerns.
The series of 16 studies with 5,873 participants from three diverse populations spanning nine domains in healthcare and beyond suggests an important obstacle to evidence-based policy and practice.
One study involved differing methods of communication aimed at educating providers on best practices for avoiding infections from catheterizations. One group of participants was told that the hospital director decided to print a checklist of safety steps for clinicians on the backs of their ID badges. Another group was told that the director instead decided to print the checklist on posters that will be hung on the wall. A final group of participants was told that the director thought of both these ideas and decided to conduct an experiment in which patients are randomly assigned to a doctor with a badge or a doctor who works in a room with a poster. They were also told that once the hospital director learns which policy is best at reducing infection rates, he or she will implement it for everyone.
Among the group who read about the director who implemented the badge policy, very few judged this decision to be inappropriate. So, too, with those who read about the director who implemented the poster policy. But many who read about the director who ran an A/B test to compare the badge and poster ideas thought this decision was inappropriate.
According to the study’s lead author Michelle Meyer, PhD, JD, Assistant Professor and Associate Director, Research Ethics, in Geisinger’s Center for Translational Bioethics and Health Care Policy, A/B tests necessarily treat people differently, and potentially unequally—at least temporarily. “But strikingly, we found that people still object to A/B tests even when the alternative was still that different people would randomly get different treatments,” Meyer said. “Because of who they happen to have trained with, or what journal articles they happen to have read, some doctors prescribe one drug for things like high blood pressure—we called it Drug A—and others prescribe Drug B. Patients who are prescribed a blood pressure medication in the ER or in a walk-in clinic are basically being ‘randomly assigned’ to one drug or the other due to chance. Yet many of our participants find that unobjectionable but object to having their treatment formally assigned by an A/B test that would figure out which drug is best so it can be offered to everyone.”
Follow-up studies found the same widespread disapproval of A/B tests despite approval of just giving everyone either A or B. Results were consistent in a variety of other domains, including consumer genetic testing products, autonomous vehicle design, and social policy programs. Doctors and participants with advanced degrees were no different than other participants.
In examining the reasons participants gave for objecting to the A/B tests, the researchers found that people often assume, incorrectly, that experts already know “what works.” The researchers concluded that more research is needed to better understand why people object to experiments that compare unobjectionable policies and how to address those concerns.
About Geisinger
Geisinger is among the nation’s leading providers of value-based care, serving 1.2 million people in urban and rural communities across Pennsylvania. Founded in 1915 by philanthropist Abigail Geisinger, the non-profit system generates $10 billion in annual revenues across 134 care sites - including 10 hospital campuses, and Geisinger Health Plan, with 600,000 members in commercial and government plans. The Geisinger College of Health Sciences educates more than 5,000 medical professionals annually and conducts more than 1,400 clinical research studies. With 26,000 employees, including 1,600 employed physicians, Geisinger is among Pennsylvania’s largest employers with an estimated economic impact of $14 billion to the state’s economy. On March 31, 2024, Geisinger became the first member of Risant Health, a new nonprofit charitable organization created to expand and accelerate value-based care across the country. Learn more at geisinger.org or connect with us on Facebook, Instagram, LinkedIn and X.
For media inquires:
Content from General Links with modal content