Our Work / Utah AI Sandbox
CHAI’s Qualification for its Third Party Evaluator Network
Utah’s Office of Artificial Intelligence Policy (OAIP) has selected CHAI as an evaluation partner for its Healthcare AI Sandbox. CHAI qualifies independent evaluators and connects them with sandbox participants, so that healthcare AI is assessed through a consistent, transparent process.
One application
Evaluators submit a single application and evidence package.
Qualified by use case
Evaluators are matched to the clinical areas they are equipped to assess.
Consistent review
Every application follows the same criteria, steps and recorded decisions.
Ongoing quality
Regular updates and annual requalification keep standards current.
About the sandbox
Utah’s AI Learning Laboratory, administered by OAIP, allows companies to pilot AI-enabled healthcare products under state oversight, including tools that support prescription renewal, diagnosis, triage and lab ordering. OAIP expects participants to show that their products are safe, non-inferior to current practice and beneficial to public health. Independent third-party evaluation helps build that evidence.
Working together
The sandbox depends on two complementary roles. CHAI provides a shared standard and a consistent path to qualification. Evaluators bring the clinical and technical expertise to carry out each assessment.
CHAI’s role
- Applies OAIP’s selection criteria and CHAI’s requirements to every evaluator
- Qualifies evaluators for specific clinical use cases
- Matches qualified evaluators with sandbox participants
- Shares the list of qualified evaluators with OAIP
- Collects feedback after each engagement to support quality over time
The evaluator’s role
- Meets the general and use-case qualification criteria
- Conducts independent evaluations using written, repeatable methods
- Reports complete results to OAIP
- Protects patient data and participants’ intellectual property
- Keeps CHAI informed of capacity, engagement status and material changes
Together
A consistent qualification process paired with independent expertise gives sandbox participants a clear path to evaluation and supports OAIP’s oversight with dependable evidence. Decisions about sandbox participation rest with OAIP.
How it works
Evaluators complete qualification once and add use cases over time. Sandbox participants are routed to CHAI by OAIP and matched with evaluators who have completed each step.
A single system for applications, review status and matching. Evaluator documents are visible only to CHAI reviewers.
Evaluator track
- Start
Apply
Submit one application and one evidence package.
- Step 1
General qualification
Security, HIPAA readiness, independence, methods and related criteria.
- Step 2
Use-case qualification
Demonstrate expertise for each clinical profile you intend to serve.
- Result
Qualified evaluator
Recognized on the CHAI partner page, listed by use case.
↓ Matching draws only from qualified evaluators ↓
Sandbox participant track
- Start
Sandbox participant
A company enters the sandbox through OAIP and is routed to CHAI.
- Intake
Profile assigned
Clinical functions, patient populations and OAIP stage determine the profile.
- Match
Evaluators matched
Qualified evaluators with capacity and no conflicts are identified.
- Result
Engagement
The participant selects an evaluator and engages them directly.
Step 1
General qualification
Every evaluator completes the same baseline review. The criteria draw on OAIP’s published selection criteria and CHAI’s requirements as convener. If a criterion is not yet met, the application is paused and can resume once it is addressed. The criteria are summarized below. Full qualification criteria, including the evidence requested for each item, are coming soon.
Active legal entity
Certificate of good standing or the equivalent in the evaluator’s jurisdiction.
Information security
SOC 2 Type II, HITRUST, ISO 27001 or equivalent for work involving PHI, models or live systems.
HIPAA readiness
Able to sign a BAA with each participant where PHI is involved, with documented PHI handling.
Independence
No relationship with, or role in building, the product being evaluated. Attestation for each engagement.
Methodological rigor
Written, repeatable methods, pre-specified metrics and complete reporting to OAIP.
IP protection
Controls that protect each participant’s model, prompts and code, with access logged.
Capacity
A named lead and team of staff and credentialed clinicians, with stated turnaround times.
Insurance
Professional liability (E&O) and cyber liability coverage.
CHAI participation
Signed CHAI evaluator participation agreement, including reporting on engagement status.
OAIP recognition
Expression of interest submitted to OAIP, with an MOU executed or in process.
Step 2
Use-case qualification
General qualification establishes a common baseline for all evaluators. Use-case qualification confirms the clinical areas each evaluator is equipped to assess, so that every match is made through a clear and transparent process. CHAI groups OAIP’s clinical scope into four profiles. Evaluators qualify for the profiles, add-ons and stages they can support by sharing their clinical team, methods and sample work for each.
- Initial focusProfile 1
Prescription renewal
AI that renews a prescription initiated by a clinician. Evaluation includes longitudinal efficacy evidence.
Expertise: primary care or internal medicine, pharmacist
- Profile 2
New prescribing or diagnosis
AI that supports diagnosis or initiates a new prescription. Evaluation includes full care-path mapping and clinician review of decisions.
Expertise: matched specialty, pharmacist, emergency medicine
- Profile 3
Triage and escalation
AI that routes patients to appropriate care without diagnosing or treating. Evaluation focuses on screening and escalation accuracy.
Expertise: primary, emergency or urgent care, nursing
- Profile 4
Lab ordering and results
AI that orders tests or delivers and explains results. Evaluation focuses on ordering logic, result delivery and intended scope.
Expertise: pathology or laboratory medicine, primary care
Add-ons for any profile
Pediatric, pregnancy, elderly and medically complex populations, and behavioral health safeguards. Each adds the relevant clinical expertise (pediatrics, OB, geriatrics, psychiatry, licensed counselors) along with subgroup and safeguard testing.
Within each profile, evaluators qualify by stage
- Proposal reviewReview of sandbox proposals
- Pre-deployment testingBefore the product reaches patients
- Post-deployment monitoringOnce the product is in use
Each decision is recorded with its reasoning: qualified, conditionally qualified (with stated conditions and a date) or not yet qualified. Additional profiles can be added later without repeating general qualification. Profiles are prioritized based on sandbox demand. Detailed requirements for each profile will be included in the full qualification criteria, coming soon.
Benefits of qualification
- Sandbox-Qualified Evaluator recognition on the CHAI partner page, showing profiles, add-ons, stages and turnaround
- Inclusion in the information CHAI shares with OAIP for its evaluator directory
- Matching with sandbox participants in your qualified profiles. Participants receive at least two matches where available
- A clear, consistent process for adding profiles and stages over time
Ongoing expectations
- Provide a capacity update each quarter
- Requalify annually
- Notify CHAI promptly of changes in security status, insurance, ownership or conflicts
- Participate in feedback from participants and OAIP after each engagement, which informs requalification
First cohort
CHAI is qualifying an initial group of evaluators ahead of the portal launch, using the same criteria. Prescription renewal is the first profile, with others to follow based on sandbox demand.
Next steps
- Express interest using the short form, noting the profiles and stages you would like to support.
- Review the full qualification criteria once published. Full details on each requirement and the supporting evidence requested are coming soon.
- CHAI will follow up with the full criteria when available, along with application and evidence package details.
Evaluators participate through CHAI’s Partner Program.
Learn more
Full qualification criteria, covering each requirement, the four use-case profiles and the matching process, are coming soon. In the meantime, organizations can express interest to receive them when available.
Qualification Criteria Coming SoonExpress interest as an evaluatorQuestions? admin@chai.orgFrequently asked questions
What is CHAI’s role in the sandbox?
CHAI qualifies independent evaluators against a shared set of criteria and matches them with sandbox participants. Qualified evaluators conduct the evaluations, and OAIP oversees the sandbox.
What happens if a criterion isn’t met yet?
The application is paused at that criterion and can resume once it is addressed. Materials already submitted are retained.
Can an evaluator start with one use case and add more later?
Yes. Evaluators can qualify for the profiles they are ready for now and add others in later rounds without repeating general qualification.
How are evaluators matched with a participant?
Each participant’s profile and stage are matched with qualified evaluators who have capacity and no conflicts. The participant selects from its matches. Sandbox decisions rest with OAIP.
Where can I find the full qualification criteria?
The full qualification criteria are coming soon and will be posted on this page. This page summarizes the process in the meantime. Organizations that express interest will receive the full criteria when they are available.
Are all sandbox evaluators CHAI partners?
No. OAIP may recognize other evaluators directly. Evaluators qualified through CHAI participate through CHAI’s Partner Program.