What Most AI Agencies Won't Tell You About Building Voice Agents

I've sat on the other side of enough sales calls to know what gets said and what doesn't. AI voice agents are being sold as magic right now — drop them in, watch your problems disappear. The demos look incredible. The ROI projections are compelling.

Most of it is real. Some of it isn't. Here's the version without the gloss.

The Demos Are Real But Controlled

When an AI voice agency shows you a demo, they're showing you a best-case scenario: clean audio, cooperative IVR system, no unexpected prompts, no hold time complications. Real production environments are messier. Background noise affects transcription. Payer systems have unexpected prompts. Calls get dropped. Authentication fails.

A well-built system handles all of this gracefully. But "handles it gracefully" means different things to different vendors. Ask specifically: what happens when a call is dropped mid-conversation? What happens when the IVR changes? What's the fallback when the AI doesn't understand a payer response?

Latency Is Everything and Nobody Talks About It

In a real phone call, a pause of more than 2 seconds feels awkward. More than 3 seconds feels broken. Most AI voice stacks have a pipeline that looks like this: audio capture → speech-to-text → LLM processing → text-to-speech → audio playback. Each step adds latency.

Getting end-to-end latency below 800ms requires careful architecture choices at every layer — model selection, streaming TTS, optimized prompts, caching common responses. Vendors who haven't solved the latency problem will show you demos where they talk slowly and pause frequently to mask it.

The Payer Problem Is Real

Some payers have explicit policies against AI agents. Ambetter is the most well-known but not the only one. This isn't a problem that technology can fully solve — it's a policy constraint that requires a workflow solution.

Any agency that tells you their agent can handle every payer without restriction is either misinformed or not being honest. The right answer is a hybrid model: AI handles what it can, and a streamlined human process handles the exceptions.

Integration Is Usually the Hard Part

The voice AI piece — the model, the TTS, the STT — is actually the easiest part to get right in 2026. The ecosystem is mature and the tools are excellent. What takes time and expertise is integrating with your specific EHR, your specific payer list, your specific workflow.

Practices on Epic have a different integration path than practices on Kareo. FHIR R4 integration is different from HL7 v2. A vendor who quotes you a fixed price for "EHR integration" without knowing what system you're on is guessing.

What to Ask Before You Sign Anything

• Can I see a live demo on my payer list, not a curated selection?
• What's your average end-to-end latency in production?
• How do you handle payers that block virtual agents?
• What does the integration with my specific EHR look like?
• Who owns the code and the prompts after delivery?
• What's your SLA for when the portal changes break the automation?

If the answers are vague, that's your answer. Good AI agencies are specific because they've built enough systems to know where the complexity lives.