Blogs

On September 17, Coalition for Health AI (CHAI) convened legal, clinical, policy, technology, and risk leaders at its Legal Summit in Boston to examine the operational questions emerging as AI becomes more deeply embedded in healthcare. The event explored health policy and the evolving regulatory landscape, payor-provider dynamics, privacy and data protections, the defensibility of AI use cases, and how liability should be allocated across the AI value chain. It also brought those issues into practice through live technology demonstrations and discussions of the AI tools organizations already have in use.

We spoke with Arvind P. Kumar, Digital Health Leader at EisnerAmper's Health Care Services Group, and Sue Cornacchio, RN, JD, an Executive Director and Advisor with a background in nursing, law, patient safety and risk management, about a central theme that emerged from the summit: responsible governance cannot stop at selecting and approving an AI tool. Instead, it must extend across the technology's lifecycle through ongoing monitoring, evidence collection, incident response, contracting and insurance.

What stood out to you most from this year's CHAI Legal Summit?

Sue Cornacchio: The discussion reflected a new level of maturity. Organizations are moving beyond forming committees and building inventories to asking how AI will be governed inside a real clinical environment. There was broad alignment that deployment is not the finish line. Because these models can behave differently across workflows and settings, health systems need sustained monitoring to understand whether they remain safe and effective.

Arvind P. Kumar: The legal lens made that lifecycle responsibility especially clear. Governance cannot end after intake, testing and/or approval. Organizations need to understand how a model performs in use, whether it drifts or produces unexpected outputs, and what evidence is available when performance varies from what was intended. Those questions connect patient and provider protection directly to contracting, liability and defensibility.

How has the governance conversation changed as healthcare AI adoption has accelerated?

Arvind P. Kumar: Healthcare has long used controls to manage technology and data, but AI introduces probabilistic behavior that is harder to predict through traditional rules-based approaches. AI is embedded in electronic health records and other workflows. Governance therefore has to assess the full system: the model, the integration, the people using it and the actions that follow.

Sue Cornacchio: The shift is from treating AI adoption as a technology implementation to treating it as a risk management discipline. Effective governance needs real authority, controls that operate while the system is in use and evidence of what AI and the human each did. It also requires leaders who understand both AI and established principles of patient safety, so oversight reflects how care is actually delivered.

Why are ongoing monitoring and evidence so important from both a safety and legal perspective?

Sue Cornacchio: When a patient safety event occurs, teams need to reconstruct what happened. With AI, that may require understanding which model and version were involved, what input it received, what output it produced, what the clinician saw and how the technology interacted with the workflow. Without that traceability, it becomes harder to learn from an event or determine responsibility.

Arvind P. Kumar: Monitoring should also identify meaningful changes before they become adverse events. Organizations need ways for clinicians to report concerning output without adding an unreasonable burden, along with thresholds that trigger review or intervention. Building that evidence is important for safety, but it may also affect an organization's ability to respond to litigation, demonstrate sound governance and secure appropriate insurance coverage. We refer to this as near real time “run” assurance.

What should healthcare organizations consider in vendor contracts and insurance coverage?

Arvind P. Kumar: Organizations should examine how responsibility is allocated when a model is controlled by a third party but used within the provider's workflow. Contract terms, indemnification, liability limits, monitoring obligations and access to evidence should align with the risks the technology creates. An AI capability embedded in a familiar platform still requires oversight.

Sue Cornacchio: Insurance coverage deserves the same scrutiny. The market is still determining how to assess and price AI-related exposure, and organizations need to understand what their existing policies cover, exclude or may require them to demonstrate. Governance, vendor contracting and insurance should be treated as connected parts of one risk strategy, not as separate conversations.

What do you expect legal and governance leaders to be discussing next on the horizon?

Sue Cornacchio: The next phase will be shaped by what organizations learn from AI operating at scale. Leaders will need to identify new safety signals, determine which monitoring approaches work and build efficient incident-response processes. They will also have to account for environments where clinicians interact with multiple AI tools that may produce inconsistent or competing recommendations.

Arvind P. Kumar: Large platform vendors will play an increasingly important role, but their scale does not remove the provider's responsibility to monitor performance within its own environment. The field still needs clearer, shared expectations for what should be monitored, what evidence should be retained and how incidents involving AI embedded in a workflow should be reported and investigated.

How can collaboration through CHAI help organizations navigate these issues?

Arvind P. Kumar: Healthcare providers, technology developers and other stakeholders need practical standards they can apply as AI evolves. CHAI brings together leaders who share the goal of accelerating adoption while protecting patients and providers. Participating gives organizations a way to help shape approaches that are grounded in how AI is actually used.

Sue Cornacchio: The multidisciplinary perspective is especially valuable. Legal, technical, clinical, safety and risk leaders see different parts of the same problem. Bringing those perspectives together helps the field move toward governance practices that are both rigorous and workable for organizations with different resources and levels of maturity.