A patient arrived at the hospital late one night with a high fever and rapid heart rate. Hours later, staff realized that he was having severe symptoms, and they did not have enough time to react. This was because the sepsis prediction model embedded within this patient’s electronic health record, the same software installed at hundreds of U.S. hospitals, had failed to alert them.
This is not a purely hypothetical example. In 2021, researchers from the University of Michigan published an external study of one of the sepsis prediction models developed by Epic Systems Corporation (a leading vendor of electronic health records) in JAMA Internal Medicine. The authors reported this version identified only about one-third of patients who would ultimately develop sepsis.
Artificial intelligence is rapidly becoming a regular part of healthcare. As of early 2026, the Food and Drug Administration has cleared over 1,400 devices that use AI, and these tools are reviewing your X-rays, identifying your lab results and directing patient flow. However, as researchers reported in Nature Medicine, nearly half (226 of 521) of the approved devices they examined weren’t able to provide data showing how they perform in “real world” settings. Another review of about 700 authorized devices also found that less than 4% included racial/ethnic data for their test subjects.
I build clinical machine learning tools — algorithms that can predict disease onset, analyze diagnostic images and automate workflows. One of the first things we all learn when developing these systems is how unreliable the system’s level of confidence can be when it is determining if an answer is correct.
There are several reasons for this. The first step in testing a new AI-based decision-making tool is to train it using one set of data (e.g., patient records), and then to test it against another, separate set of data (the “quiz” group). If the model performs well on the quiz group, then you have achieved a high level of accuracy, and those numbers are easily generated and believed by everyone involved in evaluating the effectiveness of the model.
The problem with this process is that the model has only been tested on one specific population of patients; therefore, if you were to give it a different population of patients (i.e. in a different hospital or a different neighborhood, or patients who are younger or healthier than those on which it was trained) then the accuracy may drop off significantly. Therefore, when a physician is trying to determine if they should rely upon an AI-based system for making decisions about their patient, they need to know what impact the difference in population will have on the reliability of the results provided by that system.
Those who suffer the most from these drop-offs are generally the ones who receive the least help. The majority of the pulse oximeter’s (the “finger clip” for estimating blood oxygen) calibration was done using lighter-skinned individuals; a 2020 New England Journal of Medicine study reported that Black patients were almost three times more likely than white patients to have critically low oxygen levels that would be overlooked by this tool. These devices built on the wrong population don’t fail randomly; they fail where medicine always has, but now with a reassuring number on the screen.
FDA approval of medical devices does not confirm their clinical efficacy. Many of the FDA approvals are based on the “predicate” method, meaning they are compared to an already approved device with no studies or clinical trials demonstrating they are better than those previously used. Some would say delaying the release of new medical devices is detrimental to furthering product development, while others think we need to test their performance after they have become established in treatment. Unfortunately such post-market testing provides very little assurance for the safety of patients, and can be thought of as using the population (the patients) as a test group.
Instead of banning AI in the medical health space, hospitals could request that these companies show performance with their own patient populations and subpopulations. The FDA could require demographic data for performance reports. Payers could link payment to actual validation. This is not rejecting a potentially valuable tool, but holding software to the same standards that everything else in clinical practice is held. Clinical AI has earned no exemption, as the worthwhile tools will pass through a true validation study. The ones that have only shown results on paper are precisely the ones we need to capture first.
