Blogs

As generative and agentic AI capabilities continue to advance, deploying these new technologies is increasingly about more than just what the system can do. Health systems and health plans also need to understand how it will be evaluated, monitored, and governed across its lifecycle. Technology developers need to consider those requirements from the earliest stages of product development.

Those questions are becoming more consequential as expectations for responsible AI mature, including through frameworks such as CHAI’s AI Governance Playbooks and Joint Commission’s Responsible Use of AI in Healthcare certification.

For Autonomize, participating in the development of CHAI’s Governance Playbooks offered an opportunity to bring years of experience developing healthcare AI into conversation with the people responsible for deploying and governing those systems. Laksh Krishnamurthy, Chief Technology Officer, and Gina Collins, Chief Regulatory Officer, contributed to the work on behalf of Autonomize, bringing perspectives spanning AI development, implementation, safety and regulatory considerations.

We spoke with them about what technology developers can learn from the organizations deploying AI, why trust and safety need to be designed into products from the beginning, and how collaboration can help close the gap between rapid technical advancement and responsible implementation.

Agentic AI is changing what healthcare organizations need to govern. What makes this moment different?

Governance becomes more complex as AI moves from generating answers to taking action. With agentic AI, organizations need to evaluate more than the accuracy of an individual output. They also need to understand what an agent is authorized to do, which systems and information it can access, and when it must stop and involve a person. When multiple agents work together, that oversight needs to extend across the entire workflow.

At Autonomize, trust and governance have been fundamental to our approach since the beginning. As we work with health plans, we’ve been building security, trust, and safety into the platform itself, including approval processes for agents and prompts, as well as monitoring across different roles. Governance needs to continue after deployment as policies, models, and workflows change.

The important point is that governance can’t simply sit with an end user. Clinical, operational, compliance, security, and safety stakeholders need to help define appropriate boundaries and oversee how these systems perform. Organizations need clear accountability for agent actions, visibility into how decisions were reached, and the ability to intervene when necessary.

What did working directly with health systems and health plans add to Autonomize’s perspective on governance?

As product and technology leaders, we often approach governance through the systems we build. Working alongside health systems and health plans in the Governance Working Group deepened our understanding of what it takes to make governance work inside a healthcare organization. These are the people responsible for deciding who can approve an AI use case, what evidence is needed before deployment, and who responds when something changes or goes wrong.

Those conversations reinforced that governance needs to fit into existing responsibilities and workflows. A technical safeguard is only useful if the right people can understand it, act on it, and remain accountable for the outcome.

That made the collaboration bidirectional. We brought our implementation experience while learning directly from the organizations responsible for governing these technologies in practice. It reinforced the importance of giving those teams the visibility and controls they need to make governance part of everyday operations.

How has that perspective influenced the way you think about building AI products?

It reinforced that trust and safety need to shape product decisions from the beginning. An AI agent needs to be tested for more than whether the software works as intended. We also need to evaluate whether it operates within its authorized scope, handles uncertainty appropriately, and involves a person when a situation requires judgment or review. Depending on the use case, that evaluation also needs clinical, operational, compliance, and safety expertise.

That influences what we build into the platform: clear permissions, appropriate approval steps, visibility into agent actions, and the ability to investigate and correct problems. It also means designing for ongoing evaluation, because changes to models, policies, or workflows can affect how an agent behaves after deployment.

The Governance Working Group helped broaden our understanding of those requirements and reinforced that the people responsible for oversight need to be considered users of the product from the outset. Their ability to understand and govern the system is part of what makes it ready for use.

There are already many AI governance frameworks available. What was different about developing guidance with people implementing AI in healthcare?

There is no shortage of frameworks. The challenge is translating their principles into decisions people can make in everyday operations: what evidence is sufficient to approve a use case, how much oversight is appropriate for its level of risk, and who is responsible when an issue arises.

Working with people implementing AI in healthcare brings those questions into focus. They understand the constraints of existing systems, limited resources, and responsibilities that span multiple departments. Their perspectives help distinguish guidance that sounds reasonable from guidance a team can consistently put into practice.

The collaboration also helps identify common needs across organizations while recognizing that implementation will vary. That makes the guidance more useful and gives technology companies a clearer understanding of the capabilities healthcare organizations need to put it into practice.

Where do you see the biggest disconnect between how quickly AI can be built and how it needs to be implemented in healthcare?

Technically, it has become much easier to build an AI agent and demonstrate what it can do. Establishing that it can operate reliably within a healthcare workflow takes more work. That requires testing against real conditions, defining appropriate oversight, integrating with existing processes, and making sure someone is accountable when the system encounters something it cannot handle.

That matters even in administrative workflows. An error in prior authorization, claims, or benefits information can delay access to care or create additional burdens for patients and members. The level of evaluation and oversight needs to reflect those potential consequences.

Technology companies need to account for these requirements from the beginning and support customers through implementation and ongoing monitoring. As capabilities advance, our responsibility is to help organizations adopt them at a pace supported by evidence that they can be used safely and effectively.

How are the Governance Playbooks influencing Autonomize’s work today?

We’re still early in our work with CHAI, but the process has already helped us examine assumptions and better understand the governance priorities of health plans and health systems. It is prompting more specific questions about who needs visibility into an agent’s actions, where approval steps belong, and what evidence teams need to support oversight.

We’re beginning to incorporate those lessons into our platform and solutions. We also reference the Governance Playbooks and related work in conversations with customers and their AI centers of excellence as they think through adoption.

One immediate benefit is a shared language for those conversations. It helps us clarify what customers expect, how responsibilities should be assigned, and which requirements need to be addressed through technology, organizational processes, or both.

As AI governance evolves, what deserves more attention next?

Patient and member safety needs to remain central as AI systems take on more responsibility. That requires a comprehensive approach spanning clinical safeguards, protection of sensitive information, verification of agent outputs, and rigorous evaluation. We need to test how agents handle uncertainty, exceptions, and situations beyond their authorized scope, as well as whether they complete a task correctly.

One area that deserves more attention is how organizations detect and respond to problems after deployment. Teams need clear responsibilities for investigating unexpected behavior, pausing an agent or limiting its permissions, and determining when it is appropriate to resume operation. Those processes should also capture lessons that improve future testing and oversight.

Security is closely connected to that work. Traditional cybersecurity remains essential, while agents introduce additional questions about which systems they can access, what actions they can take, and how misleading or malicious inputs could influence their behavior. Defensive measures need to reflect both the information an agent handles and the consequences of the actions it can take.

Making this practical requires continued collaboration among technology developers, health plans, and health systems. That is what we value about working with CHAI: the opportunity to develop shared expectations, standards, and practices informed by the people responsible for delivering care and running health plan programs.