The Self-hosted Voice Agent Stack

A self-hosted voice agent stack replaces a managed platform with four components you run yourself: speech-to-text, a language model, text-to-speech, and an orchestration framework that connects them all in a live phone call.

Instead of paying per minute, you pay for GPUs and engineering time. That trade-off only makes sense once you reach a certain call volume, but most teams never calculate where that break-even point actually is.

This page walks through the full architecture and the cost crossover. For a broader look at managed alternatives, including the open-source route, see our Vapi alternatives guide. If you're deciding which model should sit at the center of the stack, see our best LLM for voice agents guide. Neither guide covers hardware costs.

All prices and specifications below come from vendor and project pages as of August 2026. Verify them before using them to build a budget.

What This Page Adds

This page adds two things the nearby guides do not carry: the full four-layer architecture in one place, and a transparent cost crossover you can check against your own call volume.

Our Vapi alternatives guide names LiveKit and Pipecat as two entries in a platform roundup. Our LLM guide covers the speech-to-speech versus chained-stack choice for the model layer. Our alternatives comparison covers open-source voice models on their own.

None of those answers the question a technical buyer actually asks: what does the whole thing cost to run, and at what volume does it beat a per-minute bill.

Layer 1: Speech to Text

The speech-to-text layer turns caller audio into text, and it is the layer most likely to decide your latency. Whisper is the default open-weight choice, but you almost never run the original implementation in production.

faster-whisper is the reimplementation most teams use. Its README claims up to four times the speed of the original at the same accuracy while using less memory, and it ships under the MIT license.

The published benchmark is worth reading closely. On an RTX 3070 Ti, 13 minutes of audio through Large-v2 took 2m23s on the original and 1m03s on faster-whisper, using about 4.5GB of VRAM.

The managed contrast is Deepgram. Its Nova-3 streaming model is published at $0.0048 per minute monolingual on pay-as-you-go. That is cheap enough that many self-hosted stacks keep this one layer managed and self-host everything else.

• faster-whisper (MIT): the usual production choice, GPU or CPU.
• whisper.cpp: for CPU-only or edge deployments where no GPU exists.
• Deepgram Nova-3: the managed option, priced per streaming minute.
• Phone audio is 8kHz and noisy, so test on real calls, not clean files.

Layer 2: the LLM

The language model layer decides what the agent says, and it is the one layer you should think hardest about self-hosting. Open-weight families like Llama, Qwen and Mistral all run locally with vLLM or similar servers.

The practical question is size. A small model that answers in under a second beats a large model that thinks for three. On a phone call, speed is most of the perceived intelligence.

Self-hosting the model is also what forces you onto expensive hardware. Speech and voice models fit comfortably on a 24GB card. A large language model does not.

Many teams get the best result from a split: self-host speech and voice, call a hosted model API for the brain. Our LLM guide covers the speech-to-speech versus chained choice, so we will not repeat it here.

• Open-weight options: Llama, Qwen and Mistral families, served with vLLM.
• Small and fast beats large and clever for live conversation.
• A hosted model API keeps your GPU requirement small and cheap.
• Self-hosting the model is the decision that drives your hardware bill.

Layer 3: Text to Speech

The text-to-speech layer turns the model's answer back into audio, and open-weight quality here has improved enough to be usable on customer calls. The catch is licensing, which trips up more projects than performance does.

Kokoro is the current default for commercial work. It is an 82-million-parameter model under the Apache-2.0 license, small enough to run in a couple of gigabytes of VRAM, which keeps your hardware requirement low.

Orpheus from Canopy Labs is the option when you need expressive speech and voice cloning. It is built on a Llama-3b backbone under Apache-2.0, and its README claims about 200ms streaming latency, reducible to roughly 100ms with input streaming.

Piper is the lightweight choice for constrained hardware. Note that the original rhasspy project was archived and active development moved to a GPL-3.0 fork, which matters if you plan to embed it in closed-source software.

XTTS-v2 from Coqui is the licensing trap. Its 17-language model card is published under the Coqui Public Model License, which permits non-commercial use only. Coqui shut down, so there is no longer a party to sell you a commercial license. Check this before you build on it.

• Kokoro (Apache-2.0): 82M parameters, small VRAM footprint, commercial-safe.
• Orpheus (Apache-2.0): expressive, voice cloning, about 200ms streaming.
• Piper: light and fast; the maintained fork is GPL-3.0, so check your use.
• XTTS-v2: strong cloning, but the weights are non-commercial under CPML.

Layer 4: Orchestration

The orchestration framework is what turns three models into a conversation. It handles audio transport, turn detection, interruptions, tool calls and the phone connection, and it is the layer you cannot skip.

LiveKit Agents is the heavier option. The media server and the agents framework are both open source and free to run on your own infrastructure, with built-in turn detection and real-time transport.

Pipecat is the lighter option. It is a BSD-licensed Python framework from Daily built as a pipeline of swappable blocks, and it includes its own small WebRTC transport with no external service dependency.

The distinction that matters: LiveKit is infrastructure with a framework attached, Pipecat is a framework that borrows your infrastructure. Pick LiveKit if you also need rooms, video or multiple human participants. Pick Pipecat if you only need one caller and one agent.

Both still need a carrier for phone numbers. Telephony from a provider like Twilio is billed the same way whether you self-host or not, so it drops out of any comparison.

• LiveKit Agents: open source, self-hostable, strongest on real-time transport.
• LiveKit Cloud: a managed path at about $0.01 per agent minute if you skip self-hosting.
• Pipecat: BSD-licensed Python pipeline, provider-agnostic, lighter to run.
• Telephony is a separate bill on both routes and cancels out in comparisons.

What the Hardware Really Costs

Your hardware bill is set by one decision: whether you self-host the language model. Speech-to-text and text-to-speech fit on a single 24GB GPU with room to spare.

The published memory figures make this concrete. faster-whisper Large-v2 used about 4.5GB in its benchmark, and Kokoro is an 82-million-parameter model. Both fit on one consumer-class card together.

Rented GPU rates are public. On RunPod's Secure Cloud, an RTX 4090 with 24GB is listed at $0.74 per hour and an L40S with 48GB at $0.99 per hour. An A100 80GB is listed at $1.39 per hour.

Voice needs the models warm. A cold start adds seconds, which is fatal on a live call, so plan for a GPU that runs continuously rather than a serverless one that scales to zero.

The number you cannot look up is concurrency: how many simultaneous calls one GPU holds while still meeting your latency target. That figure depends on your models, your audio and your tolerance, and you have to measure it.

• Speech plus voice only: a single 24GB card is usually enough.
• Self-hosted LLM: expect 48GB or 80GB class hardware, or several cards.
• Keep the GPU warm; scale-to-zero breaks conversational latency.
• Measure concurrent calls per GPU before you extrapolate any cost.

The Engineering Cost Nobody Prices In

Engineering time, not GPU rental, is the largest cost of a self-hosted voice stack. A managed platform sells you an operations team you never see, and self-hosting means you become that team.

The work is continuous, not a one-off build. Someone owns latency tuning, model updates, failover when a GPU node dies mid-call, and the on-call rota when calls fail at 9am on a Monday.

Name the owner before you start. A stack with no named maintainer degrades quietly: a model update changes voice quality, nobody notices, and complaints arrive weeks later.

There is a second hidden cost in observability. Managed platforms give you call transcripts, recordings and dashboards for free. Self-hosting means you build that too, or you run blind.

• Latency tuning across four layers, repeated after every model change.
• Failover and capacity headroom for concurrency spikes.
• On-call ownership when a live call path breaks.
• Transcripts, recordings, logging and dashboards you have to build.
• Licence review each time you swap a model.

The Call-volume Crossover: an Estimate

Self-hosting starts to beat per-minute pricing at roughly 6,750 to 9,000 call minutes a month on hardware alone, per GPU. That is an estimate reasoned from public rates, not a measured benchmark, and the inputs are stated below so you can redo it with your own numbers.

The inputs are three public figures. Managed voice is $0.080 per additional call minute on the ElevenLabs Agents pricing page. RunPod lists an RTX 4090 at $0.74 per hour and an L40S at $0.99 per hour on Secure Cloud. A month running continuously is 730 hours.

The arithmetic for the cheaper card: $0.74 multiplied by 730 hours is $540.20 a month. Divide $540.20 by $0.080 per minute and you get 6,753 managed minutes. So one 24GB GPU costs about what 6,750 managed voice minutes cost, which is roughly 112 hours of talk time.

For the 48GB card: $0.99 multiplied by 730 is $722.70, which buys about 9,030 managed minutes, or roughly 150 hours of talk time.

Now add engineering, and the line moves a long way right. Suppose you allocate $4,000 a month of engineer time to the stack. That is an assumption, not a market rate. Total cost becomes $4,540, and $4,540 divided by $0.080 is about 56,750 minutes a month before you break even.

Two caveats decide whether that figure is real for you. Telephony is billed on both routes, so it is excluded here. And the managed rate excludes the language model, so if you self-host the model too, add its GPU cost to your side and its API cost to theirs.

• Inputs: $0.080 per managed voice minute, $0.74/hr and $0.99/hr GPU rates, 730 hours a month.
• Hardware-only break-even: about 6,750 minutes (24GB) or 9,030 minutes (48GB) per month.
• Add $4,000 a month of assumed engineering and it moves to about 56,750 minutes.
• Excluded from both sides: telephony, which is billed the same either way.
• Unverified variable: how many concurrent calls one GPU actually holds.

Who Should Self-host, and Who Should Not

Self-host when a rule forces you to, or when your volume is high enough that the arithmetic above clears comfortably. Those are the two good reasons, and cost alone is rarely the stronger one.

Data residency is the reason that does not depend on volume. If audio cannot leave your network under any circumstances, no managed vendor qualifies at any price.

Most small businesses should not self-host. At a few hundred call minutes a month, a managed platform is cheaper, faster to launch and better observed. Our small business voice agent guide covers that path.

A useful middle route exists. Run a managed platform now, keep your prompts and tools portable, and revisit self-hosting when your monthly minutes approach the crossover you calculated.

• Self-host if audio or transcripts legally cannot leave your infrastructure.
• Self-host if your monthly minutes clear the crossover with headroom.
• Do not self-host to save money at low volume, because you will not.
• Do not self-host without a named engineer who owns the stack.