How to Use ElevenLabs Agents: A Build Walkthrough

You can build a simple ElevenLabs Agent in about an hour. A production-ready one can take a week. Setting it up in the console is the easy part; most of the work goes into the prompt, knowledge, and testing.

This walkthrough focuses on the ElevenLabs console and assumes you've already decided a voice agent is right for the job. If you haven't, start with our vendor-neutral guide to AI voice agents for small business. It covers how to choose the job, write the script, and set escalation rules on any platform.

Here, we'll focus on the ElevenLabs-specific setup: creating the agent record, writing the vendor's recommended six-block system prompt, configuring the knowledge base and RAG, adding tools and phone numbers, adjusting turn-taking settings, building workflows, and running the three types of tests you should complete before the agent speaks to a real caller.

For a broader look at the product, read the platform explainer.

What to Have Ready Before You Open the Console

Have three things ready: the single job the agent will do, the documents that hold the answers, and a phone number nobody has published yet. Missing any one of them turns the build into guesswork.

Pick one job, not five. An agent that books appointments is testable. An agent that handles anything a caller might ask is not.

Collect the source documents now. The knowledge base accepts PDF, Word, plain text, Markdown, HTML, and EPUB files up to 20MB each, plus URLs and pasted text.

Write down the escalation rule in one sentence before you start. Something like: any billing dispute goes to a person. You will paste that into the prompt and wire it as a transfer later.

• One job, described in a single sentence.
• Source documents in a supported format, under 20MB each.
• A test phone number that is not your main line.
• The escalation rule, written before you build.

Step 1: Create the Agent

Create the agent in the Agents section of the ElevenLabs dashboard, either from a blank agent or from a template. The template is worth using for a first build because it fills in sane defaults you can then read.

Set the voice, the language, and the language model on this first screen. The platform offers models from several providers, and it also accepts your own endpoint if you host something in-house.

Write the first message the agent speaks. This one line does more work than people expect. It sets the caller's expectations for what the agent can do, and it is where you disclose that they are speaking with an AI.

Save it. The agent now exists and can be talked to in the browser, before any phone line is attached.

• Start from a template, then strip what you do not need.
• Pick a fast model first; you can trade up later.
• Use the first message to disclose the AI and set scope.
• Test in the browser widget before touching telephony.

Step 2: Write the System Prompt in Six Blocks

The vendor's prompting guide recommends six named blocks: personality, environment, tone, goal, guardrails, and tools. Writing to that structure is faster than writing prose, and it makes the prompt easier to fix later.

Personality is the role. Environment is the context, such as a phone call with background noise. Tone is how it speaks, including whether it uses short sentences. Goal is the actual workflow, step by step.

Guardrails are the non-negotiables: what the agent never says, never promises, and never guesses. Tools describes each function and, critically, the exact format each parameter needs.

Two pieces of vendor guidance are easy to miss and change output noticeably. First, spell numbers and symbols out as words, because the prompt is being spoken aloud. An email becomes name at gmail dot com.

Second, repeat your most important one or two instructions twice in the prompt, and mark them as important. This feels wrong to write, and it works.

Keep instructions short and action-based, use Markdown headings, and separate blocks with whitespace. A prompt that reads like a checklist is a prompt you can debug.

• Personality, environment, tone, goal, guardrails, tools.
• Spell out numbers, symbols, and email addresses as words.
• Give exact parameter formats in the tools block.
• Repeat the one or two critical rules twice.

Step 3: Attach a Knowledge Base and Decide on RAG

Attach knowledge from files, URLs, or pasted text, then choose whether each document sits in the prompt or gets retrieved. That choice is the one most first builds get wrong.

Full context puts the whole document in the system prompt on every turn. It only works for documents that fit inside roughly 300,000 characters, and it costs tokens on every single turn.

RAG retrieves only the passages relevant to what the caller just asked. It is the default under auto mode and it is what lets you attach a large document set. Folders are always accessed through RAG and are never placed in the prompt.

The failure mode here is subtle. A small policy document in full context behaves consistently, while the same document under RAG can miss a clause the caller asked about sideways. Test the awkward phrasings, not the obvious ones.

Each document can be set to auto, forced into the prompt if it fits, or left to retrieval. Total indexable size depends on your subscription tier, so check the vendor page for your plan.

• Sources: files, URLs, or pasted text.
• Full context fits roughly 300,000 characters, and costs tokens every turn.
• RAG is the auto default and scales to larger sets.
• Folders always use RAG, never the prompt.
• Indexable size varies by plan tier.

Step 4: Add the Tools the Agent Will Call

Tools are what turn a talking agent into a working one, and there are two kinds: system tools the platform ships, and your own tools that call your APIs. Without them the agent can only describe an action it cannot take.

The system tools cover end call, language detection, agent transfer, transfer to number, skip turn, play keypad touch tone, and voicemail detection. Each is switched on per agent.

Two of those matter more than the rest on real phone lines. Voicemail detection stops an outbound agent from talking to an answering machine. Keypad tones let it work through another company's phone menu.

Your own tools are described in plain language, and the model decides when to call them. That is why the parameter formats in your prompt matter so much: a vague format description produces a malformed API call, not an error message.

• System tools: end call, transfers, skip turn, keypad tones, voicemail detection.
• Custom tools call your APIs mid-conversation.
• The model picks the tool from your plain-language description.
• Give exact parameter examples or expect malformed calls.

Step 5: Wire a Phone Number

Connect a phone number either through the native Twilio integration or through SIP trunking, then assign it to the agent. Both routes are documented by the vendor, and both leave the telephony bill with the phone provider.

Twilio is the shorter path. You import the number with its credentials, assign the agent, and place a test call.

SIP trunking is the route when you already have carrier contracts or a phone system you cannot move. It also enables SIP REFER transfers, which hand a call off at the protocol level.

Assign a throwaway number first. An agent attached to your published main line while you are still tuning the prompt is a bad afternoon.

For outbound work, batch calling takes a CSV or XLS list with a mandatory phone number column and per-row variables like a customer name. Wiring details for CRM and telephony are covered in the integrations guide.

• Native Twilio import, or SIP trunking for existing carriers.
• SIP trunking enables SIP REFER transfers.
• Telephony is billed by the phone provider, not bundled.
• Batch calling takes CSV or XLS with per-row variables.

Step 6: Set Interruption and Turn-Taking Behavior

Turn-taking settings decide whether the agent feels human or feels rude, and they live in the conversation flow settings. Three of them do most of the work.

Turn eagerness has three modes: eager, normal, and patient, with normal as the default. Patient suits callers who think out loud, such as someone reading a policy number off a letter.

Take turn after silence sets how long the agent waits before speaking, from 1 to 30 seconds. The vendor suggests 5 to 10 seconds for casual conversation and 10 to 30 when people need thinking time.

A soft timeout plays a short filler while the agent waits, ranging from 0.5 to 8 seconds, disabled by default, with 3 seconds recommended. It can play a fixed filler or a model-generated one.

Interruptions are toggled in the Advanced tab. Turn them off where the whole message must be heard, such as a legal disclaimer, and on everywhere else.

Maximum conversation duration runs from 60 to 7,200 seconds and defaults to 600. That default is a cost control as much as a design choice, and it is worth setting deliberately. Deeper latency and barge-in behavior is covered in the Agents limits guide.

• Turn eagerness: eager, normal, or patient. Default normal.
• Take turn after silence: 1 to 30 seconds.
• Soft timeout filler: 0.5 to 8 seconds, off by default.
• Interruptions off for disclaimers, on for normal talk.
• Max duration: 60 to 7,200 seconds, default 600.

Step 7: Build a Workflow If the Call Branches

Use the visual workflow builder when the call has real branches, and skip it when the call does not. A single prompt handles a simple job better than a graph does.

Workflows are built from nodes. Subagent nodes change the prompt, model, voice, knowledge, or tools at a point in the call. Tool nodes force a tool call to happen rather than hoping the model picks it. Transfer nodes hand off to another agent or to a person. End nodes close the call.

Edges connect them. They can move forward, loop backward for a retry, route on a plain-language condition, or route on a deterministic expression.

The tool node is the underrated one. If a step must always run, such as logging a case number, a tool node guarantees it in a way a prompt instruction never can.

Workflow analytics overlay per-node entries, average time spent, and where calls terminate. That view is how you find the step where callers give up.

• Subagent nodes change behavior mid-call.
• Tool nodes guarantee a step runs.
• Transfer nodes hand off to an agent or a human.
• Edges route on plain-language conditions or expressions.
• Node analytics show where callers drop.

Step 8: Test Before a Real Caller Hears It

Run automated tests in the console before the agent takes live calls, using the three test types the platform provides. Manual dialling is not testing, because you cannot repeat it fifty times.

Simulation testing runs a full multi-turn conversation against a simulated user and checks whether the call reached the outcome you defined. Next-reply testing checks only the agent's immediate response, which is the fast way to police tone and policy.

Tool-call testing confirms the agent calls the right tool with the right parameters, validated by exact match, regex, or a model judge.

The setting that matters most is repetition. Tests can be run 2 to 20 times so you measure a pass rate rather than one lucky answer. Language models are probabilistic, so a single pass proves very little.

Tests can run from the dashboard, the CLI, or the API, which means they can sit in a deployment pipeline rather than in someone's memory.

• Simulation testing: full multi-turn outcomes.
• Next-reply testing: tone and policy on one message.
• Tool-call testing: right tool, right parameters.
• Repeat 2 to 20 times to get a pass rate.
• Run from dashboard, CLI, or API.

Step 9: Version It, Then Watch It Live

Keep the agent's configuration in version control and watch every call after launch, because a prompt change that helps one call type can quietly break another. The platform supports this rather than forcing it, so it is on you to set up.

A workflow can be designed in the visual editor and then committed as JSON to your repository. The vendor also ships a CLI for versioning and deploying agents from a terminal.

Post-call webhooks push data to your endpoint after each call finishes. There are transcription webhooks with the full conversation, audio webhooks carrying the recording, and call initiation failure webhooks for calls that never started.

That third webhook is the one teams forget to handle. A call that fails to start produces no transcript and no recording, so without it the failure is invisible in your own reporting.

Success evaluation scores each conversation against criteria you define, returning success, failure, or unknown with a rationale. Track that rate weekly. It is a better health signal than call volume.

• Commit workflow JSON to your repository.
• Use the CLI to version and deploy agents.
• Post-call webhooks: transcript, audio, and initiation failures.
• Success evaluation gives a per-call pass, fail, or unknown.
• Review the failure rationales weekly.

Common Mistakes on a First ElevenLabs Agent Build

Most first builds fail on four things, and none of them are the voice. The voice is the part that works out of the box.

The first is a prompt written as prose instead of the six blocks, which makes every later fix a rewrite. The second is a knowledge base uploaded without testing the awkward questions.

The third is skipping tools, so the agent talks about checking your account without ever checking anything. The fourth is going live on the main line before running repeated tests.

There is a fifth that only shows up on the bill. A generous maximum conversation duration plus a chatty prompt produces long calls, and calls are the billing unit. Which jobs are safe to run unattended is covered in the use-case readiness guide.

• Prose prompts instead of structured blocks.
• Untested knowledge base retrieval.
• No tools, so no real actions.
• Live on the main line too early.
• Long default call duration, quietly expensive.