Generative AI is moving into clinical software faster than the rules written for it. This week is a good moment to look at how regulators are thinking about generative AI medical devices, because the FDA's public comment window on its discussion paper closes on October 19.
In its August 18 announcement, the FDA outlined a possible two-axis framework for assessing risk and a premarket approach built on competency assessment, inspired at a high level by how physicians are trained and evaluated. In practice that means non-clinical benchmarking plus clinical confirmation before a device reaches patients, followed by postmarket monitoring scaled to risk. The paper also asks how foundation models and agentic AI systems should be handled. It is a discussion paper, not guidance, but the direction is clear: test these tools the way you would vet a new clinician, then keep checking.
Why generative AI medical devices need ongoing checkups
Traditional clearance treats a model as a fixed thing. Epic's Deterioration Index Version 2, cleared on September 18, shows the strengths and limits of that approach. As Physician in the Loop summarized from Becker's reporting, the model is locked and the clearance includes no predetermined change control plan, so a retrained version would likely need a new submission. Epic's retrospective testing showed an AUC of 0.83 for the combined outcome, but at the medium-risk threshold sensitivity was 26% for Black patients and 32% for Hispanic patients versus 51% for white patients, a gap that narrowed in a site-by-site analysis. Each hospital sets its own alert thresholds, which leaves much of the real-world monitoring to the people running it.
Oversight of clinical AI is also being worked out outside the FDA. Becker's Hospital Review reports that a Federation of State Medical Boards workgroup including the AMA, AHA, Anthropic and Elevance Health met in Washington in early October to draft recommendations for AI that performs clinical functions with limited or no direct physician supervision. A draft is expected in weeks to months, and because states license medicine, it could shape state rules.
For teams building health software, the practical takeaway is to treat evaluation as an ongoing job, not a launch gate. Benchmark against the tasks the tool actually performs, check results across the patient groups you serve, log what the model suggested and what a clinician decided, and make overriding it easy. Generative models will keep changing, and the review around them has to keep pace. We would like to hear how your organization checks AI tools after go-live.