Every practice I talk to right now is either piloting a clinical AI agent or getting pitched one weekly. The demos are slick. The trials are cheap. And the failure mode is almost always the same: the agent goes live before anyone audited what it was actually going to touch.

An AI agent is not a feature you turn on. It is a new staff member with API access to your schedule, your chart, your inbox, and often your patients' phones. You would not hand a new MA the keys to all of that on day one without checking references and shadowing them for a week. Apply the same discipline here.

What follows is the audit I run before we let an agent touch a live workflow. It is written for practice owners and operators, not engineers. If your vendor cannot answer these questions clearly, that is your answer.

1. Map the workflow you are actually replacing

Before you evaluate any vendor, write down the current workflow in enough detail that a temp could execute it. Who receives the refill request? What do they check in the chart? What triggers a message to the prescriber? What are the exception paths, and how often do they fire?

In practice, this exercise kills about a third of AI projects on its own. Operators discover the workflow they thought they had is not the workflow the team is running. You cannot automate a process you have not documented, and you cannot measure lift from an agent if you never measured the baseline.

Concrete things to capture:

  • Median and 90th-percentile handling time per task, measured, not estimated.
  • Volume per day and how it spikes (Monday mornings, post-holiday, after a marketing push).
  • Every exception path and its frequency, including the "we just call the patient" catch-all.
  • Who owns the task today and what else they do with the same hour.

2. Audit your data before the agent sees it

Agents inherit the quality of your data. If your patient records have three different spellings of the same medication, five allergy fields that disagree, or a scheduling system where "Consult" means four different visit types depending on the provider, the agent will confidently produce garbage.

Run a data hygiene pass on the specific fields the agent will read or write. For a refill agent, that is medication list, last fill date, prescriber, pharmacy of record, and any protocol notes. For an intake agent, it is form templates, insurance fields (if applicable), and the mapping between what the patient submits and where it lands in the chart.

You are looking for two things. First, structural problems: fields that are free-text when they should be structured, duplicate patient records, orphaned tasks. Second, semantic drift: staff using fields for purposes they were not designed for, which the agent will not know about.

3. Define the failure modes you can tolerate

Every agent will fail. The question is how, how often, and who catches it.

Sort failures into three buckets. Silent failures are the dangerous ones: the agent does nothing when it should have acted, or acts wrongly and no one notices. Loud failures are annoying but safe: the agent escalates too much, asks for human review on easy cases, sends a confused message a patient calls about. Catastrophic failures are the ones that end up in a complaint or a board letter: wrong medication, wrong patient, wrong dose.

For each workflow, decide what your acceptable rate is for each bucket, and how you will measure it. "We'll just watch it for a while" is not a plan. You need a sampling protocol (I use 100% review for the first two weeks, then random 10% ongoing) and a person whose job includes doing the review.

4. Verify the compliance posture, in writing

This is where a lot of operators wave their hands and hope the vendor has it covered. Do not do that. Get answers, in the contract or a signed addendum:

  • Is there a BAA, and does it cover the specific AI processing (including any third-party model providers the vendor uses under the hood)?
  • Where is PHI stored, where is it processed, and is any of it used to train models, whether the vendor's or a subprocessor's?
  • What is the audit log? Can you, the practice, pull a record of every action the agent took on a given patient, with timestamps and inputs?
  • If a patient requests deletion or an accounting of disclosures, can the vendor support that?
  • What happens to your data if you cancel?

Confirm the specifics with your own counsel. Vendors will tell you they are HIPAA-compliant. HIPAA compliance is a posture, not a certificate, and the details matter more than the claim.

5. Pressure-test the human handoff

The handoff between agent and human is where most deployments quietly break. The agent does 80% of the task, then dumps a half-finished thread into a staff inbox with no context, and your MA spends more time reconstructing what happened than she would have spent doing the task herself.

Before go-live, walk through five real scenarios end to end with the vendor. Not demos. Actual cases from your last week, with names redacted. For each one, ask: when the agent escalates, what does my staff see? Can they read the full conversation? Is the patient's chart already open? Is the recommended next action clear, or do they have to guess what the agent was trying to do?

If the handoff is bad, the agent is a tax on your team, not a lift.

6. Plan the rollback

Decide, before you flip the switch, what conditions cause you to turn the agent off. Write them down. Share them with your team. Examples I have used:

  • Any catastrophic failure, full stop, immediate pause pending review.
  • Loud failure rate above X% for two consecutive weeks.
  • Staff time to review agent output exceeds staff time the agent saves.
  • More than N patient complaints referencing the agent in a month.

Rollback is not defeat. It is the thing that lets you deploy aggressively without betting the practice. If you cannot articulate what would make you turn it off, you are not ready to turn it on.

7. Train the team on what the agent is, and is not

Your staff will form a mental model of the agent within the first week. If you do not shape it, they will either treat the agent as infallible (and stop reviewing its work) or treat it as a toy (and duplicate everything it does). Both kill your ROI.

Spend an hour with the team before launch. Show them exactly what the agent does, what it does not do, what it is allowed to decide on its own, and what it must escalate. Show them where to see its work. Tell them how to report a failure and who reads those reports. Then do it again after two weeks, once they have real cases to talk about.

The short version

An AI agent inherits your workflows, your data, your compliance posture, and your team's trust. Audit each one before you deploy. The vendors that survive the next two years will be the ones who make this audit easy, because operators are going to demand it.

If you want to see how we think about this on the infrastructure side, particularly for compounding and concierge practices where the workflows are dense and the margin for error is small, talk to our team.

Frequently Asked Questions

How long should a pilot run before we decide to expand?

Long enough to see your real volume patterns, including a slow week and a spike week. For most practices that is six to eight weeks. Anything shorter and you are making a decision on a sample size that will embarrass you later.

Should the agent be allowed to send messages to patients without human review?

It depends on the message type and your risk tolerance. Appointment confirmations and generic logistics, usually yes after a supervised period. Anything clinical, including refill status that references a medication by name, I keep human-in-the-loop until we have months of clean data and a clear audit trail. Your counsel should weigh in on this specifically.

What if our EMR does not integrate with the agent we want?

Then the agent will run on screenshots, copy-paste, or a shadow database, and you will spend the savings on reconciliation. Integration is not a nice-to-have. If the vendor cannot read and write to your system of record cleanly, the deployment will erode instead of compound.

How do we measure ROI honestly?

Compare staff hours on the specific task before and after, not total staff hours (which move for a hundred reasons). Track the failure rates you defined in your audit. Track patient-side signals like time to first response and complaint volume. If all three move the right way, you have a real result. If only the vendor's dashboard moves, you have a marketing asset.