What a Self Hosted AI Agent Costs to Run

A self-hosted AI agent swaps your per-token bill for a hardware bill, and a maintenance job someone on your team has to own.

At Layer3Labs, we build and run automation inside other companies, while using the same infrastructure across our own portfolio of sites. So we evaluate this tradeoff often.

Self-hosting tends to pay off in a narrow set of cases: high-volume, repetitive work; data that can't leave your network; or processes that need to keep running without internet access. Outside of those cases, a cloud API is usually cheaper and better.

Before you buy anything, run the numbers below.

What a Self Hosted AI Agent Costs Upfront

Self-hosting an AI agent costs you four things: a graphics processing unit (GPU) with enough video memory, the machine around it, the electricity it draws while it works, and disk space for model weights.

Video memory, or VRAM, is the binding constraint. The model weights have to sit in VRAM for the agent to answer at a usable speed, so the card you buy sets the size of model you can run at all. Reported requirements for the local agent Perplexity released with NVIDIA put the floor at 24GB of VRAM, with 32GB cited as the recommendation, which rules out most consumer desktops before the conversation starts.

A rough sizing rule holds across runtimes, and our Ollama guide sets it out in full: a quantized 7B model wants around 8GB of memory, a 13B model around 16GB, and a 27B to 33B model around 32GB. Treat those as starting points and check the model card for the exact build you plan to run, because the quantization level moves them.

Purpose-built boxes exist if you would rather not assemble one. The NVIDIA DGX Spark was listed at $4,699 on NVIDIA's marketplace as of 27 August 2026, and multiple units can be linked over shared memory when a model does not fit on one.

Electricity is the line most budgets miss. Read your card's rated board power on its own product page, multiply by the hours a day the agent is genuinely under load, and price it at your utility rate. A GPU pinned for eight hours every day is a different bill from one that wakes up twice.

Storage is cheap but not free. Each set of model weights runs to tens of gigabytes, and you will keep more than one, because you need the previous version on disk the day a new one changes your outputs.

• A GPU with 24GB VRAM as a practical floor, 32GB for 27B-class models
• The host machine around it: CPU, system RAM, and a fast SSD
• Electricity for every hour the GPU is under load
• Disk for several model versions at tens of gigabytes each
• One named person who owns the box when it stops answering

What You Stop Paying For

You stop paying the per-token API bill. Once the hardware is bought, running one more task costs electricity and nothing else, which is why volume is the argument that carries this decision.

Work the break-even from three numbers you already have.

• Your average monthly API spend across the last three months of invoices
• The full hardware cost, including the machine wrapped around the GPU
• Any subscription the local product you pick requires on top of the hardware

How to Run the Break-Even

Divide the hardware total by your average monthly API spend, and the answer is how many months of current usage the box has to survive before it is the cheaper option.

Two forces bend that number in opposite directions. Usage tends to climb once people trust the agent, which shortens the payback. The hardware also ages and will want replacing inside a few years, which stretches it. If the two roughly cancel, your first estimate is close enough to decide on.

Itemise the spend by workflow before you buy anything, because the exercise usually turns up a cheaper move than a purchase. Our guide to cutting AI token costs walks that accounting.

In the routines we run across our own portfolio of sites, bulk classification steps go to small cheap models and only the hard judgment calls reach a large one. That split cut the bill without any hardware at all, and it is worth trying first.

The Maintenance Hours You Are Buying

Self-hosting moves the operations work from a provider's on-call rota onto your team, and that work does not stop when setup finishes.

The recurring jobs are driver and runtime updates, model swaps with a round of prompt re-testing after each one, disk cleanup as weights pile up, and watching for runs that failed quietly. None of them is hard. All of them need somebody whose week has room for them.

Across the workflows we have automated for SMB teams, the recurring cost is rarely inference. It is the person who has to notice that last night's batch never finished and decide whether to rerun it or fix it.

Failures also ignore office hours. An agent chewing through an overnight queue fails overnight, and unless someone is reachable, the work is simply not done by morning. Price that as a rota with named people on it. If nobody on your team currently carries a pager for anything, a self-hosted stack adds the first one.

• Driver, runtime, and OS updates that have to be tested before they land
• Model swaps, each followed by re-testing the prompts that depend on the old one
• Disk cleanup as old weights accumulate
• An on-call arrangement for overnight and weekend failures

What Breaks in Practice

The failures that recur in self-hosted agent setups are memory limits, driver mismatches, model swaps that change the output format, and queues that stall without telling anyone.

Each has a cheap preventive fix, and putting them in before launch costs an afternoon.

• Out of memory on an unusually long input. Cap input length inside the agent and reject oversize jobs with a clear error instead of letting the run collapse.
• A driver or CUDA update that leaves the GPU invisible. Pin your driver and runtime versions, and test upgrades on a second machine before the production box.
• A model swap that changes the shape of the output and breaks whatever parses it. Keep the previous weights on disk and run a fixed set of test inputs through the new model before you switch.
• A stalled queue nobody noticed. Add a health check that alerts a human when nothing has completed in an hour.
• Long documents degrading quietly. Perplexity advertises a 260,000-token context for its local agent and reports quality falling off past roughly 100,000 tokens, so chunk long inputs rather than trusting an advertised ceiling.

Which Workloads Justify a Self Hosted AI Agent

Three workloads justify a self hosted AI agent: high-volume repetitive tasks, data barred from leaving your network, and processes that must run without internet access.

Volume is the clearest case. Classification, field extraction, tagging, and first-pass summarising each cost very little per run and a great deal in aggregate when they fire thousands of times a month. Those tasks also sit comfortably inside what a 27B-class model does well, so you give up almost nothing on quality.

Data residency is the second, and it changes the maths entirely. In the client-intake and engagement-letter automation we have built for law firms, the question that settled the architecture was never accuracy. It was which systems the document is permitted to touch. When the permitted list is your own network, local stops being the expensive route and becomes the only one.

Offline is the third and the narrowest. A plant with a flaky connection, a secure facility, or a field crew gets an agent that keeps working when the link does not.

• Invoice and document field extraction running thousands of times a month
• Ticket and email classification before anything reaches a human queue
• Client files under a contract that bars third-party processing
• Sites where the network drops and the work still has to happen

Who Should Not Self Host

Self-hosting is the wrong call for low-volume work, for hard multi-step reasoning, and for teams with nobody to own the machine.

If your monthly API spend is small, the hardware will be old before it pays for itself. Keep paying per token, and revisit the question when the bill is large enough that someone complains about it.

If the work is hard reasoning across long documents, the models you can run on one machine will frustrate you. Perplexity states plainly that compact models trail frontier models on hard reasoning, and that limit applies to every local setup, not only to theirs. Route those tasks to a cloud model and self-host the rest.

If your team runs on Apple silicon, the packaged NVIDIA-based option is unavailable and none is on the roadmap. Model runners like Ollama do run on Apple hardware, so the route left open to you is assembling your own stack rather than buying a finished one.

Two things would change this answer. A compact model that closes the reasoning gap would widen the band of work worth keeping local. So would a second-hand market that puts 32GB cards well under current prices, because it shortens the payback period for anyone running this arithmetic.

The Quality Ceiling of Models Small Enough to Self Host

The models that fit on a single self-hosted machine today sit around 27 billion parameters, and they trail frontier cloud models on hard reasoning.

Perplexity's local agent ships with Qwen 3.8 27B and PPLX 27B, its own post-trained version of that Qwen model, with a third model listed as coming rather than shipped. That is a fair picture of the class you get to work with locally.

Perplexity's published figures are vendor-reported rather than independent testing, and they show the gap clearly. Its local setup scored 82.6% on the Local Knowledge Work Bench, a 53-task knowledge-work suite, against 77.6% for the comparison harness it tested alongside. On a coding benchmark it scored 59.6% running purely locally, rising to 73.0% when the run was allowed to escalate to a cloud advisor model, at roughly $0.415 a task by its own count.

That spread is the useful part. Routine work lands close to what a cloud model gives you, hard work still wants the larger model, and the escalation carries a price per task. Decide the split before you buy the hardware: which jobs stay local, which go out, and what rule sends them.

• 27B-class models handle summarising, extraction, classification, and drafting well
• Multi-step reasoning and long-document work remain weaker than frontier cloud models
• Escalating the hard cases to a cloud model adds a metered cost line of its own

Where Packaged Local Products Fit

A packaged local product trades model and deployment flexibility for an assembled stack somebody else already built and tested.

Perplexity announced its Portable Computer on 25 August 2026, built with NVIDIA, as the local-first version of its existing cloud agent. Model inference, the orchestrator, the planner, the tool router, the scheduler, the durable task queue, the local search index, private-document processing, and PII classification all run on the device. Web searches and connector calls still leave it, along with an optional escalation to a frontier advisor model that returns text guidance only and cannot touch local files or local tools.

The orchestrator, scheduler, and durable queue are the parts teams underestimate when they build their own, so buying them assembled removes weeks of work you would otherwise carry. What you give up is the freedom to swap in any model you like and to deploy the pieces where you want them.

The constraints are real. Linux is the only platform at launch, with Windows support stated for September 2026, and it unlocks with Pro, Max, Enterprise Pro, and Enterprise Max subscriptions rather than as a separate purchase. Confirm which tier you hold on Perplexity's pricing page before assuming access, and read what Portable Computer runs on-device if you are weighing it against building your own.

Self Hosted AI Agent vs Cloud API vs Packaged Box

Three routes are open, and they differ mostly on who absorbs the setup work and who answers the phone at 2am.

Pick the self-assembled stack when volume is high, the workload is repetitive, and you have someone on a rota. Pick a packaged box when you want local processing without building orchestration yourself and you can live inside its hardware and platform rules. Stay on a cloud API when your spend sits below the break-even, or when the work needs reasoning the local models do not yet reach.

Start with the invoice rather than the hardware. Pull your last three months of API spend and mark which workflows are repetitive and which carry data that cannot leave your network. That list tells you whether a self hosted AI agent earns its place, and which tasks belong on it first.