Most health systems now have a process for saying yes to an AI tool. Fewer have a plan for what happens the month after. That gap is getting attention: in a healthsystemCIO webinar published today, leaders from Hartford HealthCare argued that monitoring AI after approval matters as much as the approval itself, because the tool you reviewed is not always the tool you are running six months later.
The examples are mundane, which is the point. A routine vendor upgrade can quietly put a large language model behind software that was approved years ago as a simple, predictable app. Vitea, which sponsored the session, said it finds 160 to more than 300 unknown applications at almost every health system it works with, and that the official application inventory and the actual network traffic have disagreed at every one. Hartford HealthCare estimates about 300 shadow AI capabilities of its own.
Why monitoring AI after approval is landing on the buyer
A JAMA Health Forum essay from October 16 describes the structural problem as an accountability inversion. Developers control how a model is designed and updated, but responsibility for safety is shifting to hospitals that often cannot see training data or model version histories. The numbers suggest local checking is uneven. Former FDA commissioner Scott Gottlieb, as reported by FDAweb, cited survey data showing that 65% of U.S. hospitals use AI models, yet only 61% of those using predictive models test most of them against local data, and 44% check them for bias.
What Hartford HealthCare describes is less a product than a set of habits. Keep an inventory that matches what is actually running. Write contract terms that require notice when a vendor adds AI or swaps the underlying model. Treat a model change like a new tool and send it back through review. Monitor against performance limits written into policy rather than the vendor's own safety claims, and give that job to the people who know the work well enough to notice when output drifts. One panelist put the current habit bluntly: AI gets tested like a new drug and then monitored like a fax machine.
None of this is specific to hospitals. Any team that builds or buys software with a model inside it inherits the same question: who notices when it changes? The practical answer is to plan the after-launch part before launch, with logging, a named owner, a way to roll back, and a person reviewing what matters. We would be interested to hear how your organization handles a vendor model change today.