Self-Hosted AI Chatbot: How to Build a Private Assistant
A practical build guide for running your own AI assistant on hardware you control, model choice, runtime, hardware sizing, and connecting it to your own documents.
A self-hosted AI chatbot runs an open-weight model on hardware you own or rent, instead of calling a vendor's hosted API. The appeal is data control (nothing leaves your infrastructure), predictable cost at high volume, and the ability to run fully offline if that matters for your environment.
This is a different question from "which hosting option should I pick", our guide on running open-weights models covers that broader map of desktop apps, inference servers, and hosted APIs. This page is the applied build project: picking a model and runtime, sizing real hardware, and wiring the result to your own documents so it actually functions as an assistant rather than a bare chat window.
It is also not automatically the private or secure option people assume. Self-hosting moves the responsibility for patching, network exposure, and access control onto you. That tradeoff is worth understanding before you commit hardware budget to it.
Who Actually Needs a Self-Hosted Chatbot
Self-hosting makes sense for a specific set of situations, not as a default upgrade over a hosted API.
- Regulated data that cannot leave your environment under your compliance obligations.
- High query volume where a hosted API's per-token cost would exceed the cost of owning or renting the hardware.
- A genuine offline requirement, field operations, air-gapped environments, unreliable connectivity.
- A team that wants full control over the model version and behavior, without a vendor pushing an update mid-project.
Scoping a self-hosted AI chatbot for your internal documents or customer support? We help size the hardware and the RAG layer before you spend on equipment.
Book a ConsultationStep 1: Pick the Model
Model choice is the first decision and it drives everything else, since a larger model needs more hardware to run at usable speed. See our guide on the best open-source LLMs for a current shortlist, and our per-model guides (Llama, Qwen, Mistral, DeepSeek) for hands-on setup steps for each family.
As a starting rule: a model in the 7–14B parameter range runs comfortably on a single consumer GPU and is the right default for most business chatbot use cases. Reach for a larger model only once you have measured that the smaller one's answers are not good enough for your specific documents.
Step 2: Pick a Runtime
The runtime is the software that actually loads and serves the model. Ollama is the most common starting point for a business deployment: it handles model downloading, quantization, and serving through a simple local API with minimal setup. Larger production deployments often move to vLLM for higher-throughput serving once a single Ollama instance becomes the bottleneck.
Whichever runtime you pick, budget time for quantization tradeoffs: a heavily quantized model runs faster and needs less memory, at some cost to answer quality. Test both ends before locking in a production configuration.
Step 3: Size the Hardware Realistically
Hardware sizing is the step teams most often get wrong, either overbuying an enterprise GPU for a light workload or underbuying and getting unusable response times. Use our free local AI hardware calculator to size your setup against the model and query volume you actually need, rather than guessing from a spec sheet.
If you don't want to buy hardware outright, renting GPU capacity from a provider like RunPod lets you test a configuration under real load before committing to a hardware purchase, a sensible first step if you are not yet certain of your long-term volume.
Step 4: Wire It to Your Own Documents
A bare self-hosted model only knows what it was trained on, which usually stops well before your company's actual documents. Retrieval-augmented generation (RAG) is what connects it to your own content: chunking your documents, embedding them, and retrieving the relevant pieces at query time.
For a straightforward document lookup, standard RAG is enough. For a system that needs to plan across multiple documents or check its own answer, see our guide on agentic RAG architecture for the added components that requires.
Step 5: Connect It to Your Applications
Once the model, runtime, and retrieval layer are working, wrap the whole thing behind a simple internal API so other tools, a support dashboard, an internal chat client, a Slack bot, can call it without knowing anything about the underlying model.
What Self-Hosting Does Not Solve
Running the model yourself does not automatically make the deployment secure. You are now responsible for patching the runtime, restricting network access to the inference endpoint, and controlling who inside your organization can query it, all things a hosted vendor otherwise handles for you.
It also does not remove the need for evaluation. A self-hosted model can still produce a wrong or fabricated answer with the same confidence as a hosted one; the fix is the same review process either way.
A Note from Our Own Hardware Sizing Work
We built and maintain our own local AI hardware calculator for exactly this sizing step, and the recurring pattern we see is businesses defaulting to a much larger GPU than their actual query volume needs, a 7–14B model on a single consumer GPU handles most internal-document chatbot workloads, and the jump to enterprise-grade hardware is rarely justified until query volume is measured, not assumed.
What you need to run open-weights models yourself
How much hardware you need depends entirely on the model’s size. A small model runs on a laptop; a frontier model needs a server or the cloud. Here is the map by size class — pick the row that matches the model you have in mind.
| Path | What it is | Best for | Get started |
|---|---|---|---|
| Small models (≤14B) | Run on a single 16–24GB GPU, an Apple Silicon Mac, or a mini PC | Phi / Gemma / small Qwen-class | NVIDIA GeForce RTX 4090 |
| Mid-size (~15–150B) | One 48GB pro GPU or a large unified-memory Mac | Llama-70B / Mixtral-class | Apple Mac Studio (M4 Max, 128GB) |
| Frontier (>150B) | Rent H100 / A100 nodes, or run a multi-GPU rig | GLM / DeepSeek-class | RunPod |
| Any size, no hardware | Call a hosted API and pay per token | Trying models before committing | OpenRouter |
Whichever size you land on, a one-click runner like Ollama or LM Studio gets small and mid models going in minutes; for a hosted endpoint, point Cursor at the model through OpenRouter. For specific hardware picks, see Best mini PCs for local AI and Local AI hardware calculator.
Frequently Asked Questions
- A self-hosted AI chatbot runs an open-weight language model on hardware you own or rent, rather than calling a vendor's hosted API. It gives you full data control and predictable cost at high volume, at the cost of taking on the hardware, security, and maintenance work yourself.
- It depends on model size and query volume, not a single fixed answer. A 7–14B parameter model runs comfortably on a single consumer GPU for most business chatbot workloads. Use a hardware calculator against your specific model and volume rather than guessing from a spec sheet.
- It can keep your data from ever leaving your infrastructure, which is genuinely more private in that specific sense. But self-hosting is not automatically secure: you take on responsibility for patching the runtime, restricting network access, and controlling who can query the endpoint, which a hosted vendor otherwise handles.
- Ollama is the simpler starting point: minimal setup, good for a single-server or small-team deployment. vLLM is built for higher-throughput production serving and is worth the added setup complexity once a single Ollama instance becomes your bottleneck.
- Yes, renting GPU capacity from a cloud provider like RunPod lets you test a real configuration under load before committing to a hardware purchase, which is a sensible way to validate your sizing before spending on equipment.
- Through retrieval-augmented generation (RAG): chunk your documents, embed them into a vector store, and retrieve the relevant pieces at query time so the model can answer from your actual content instead of only what it was trained on.
Sizing hardware for a self-hosted assistant?
Layer3 Labs helps businesses scope a self-hosted AI chatbot build, model choice, hardware sizing, and the RAG layer that connects it to your own documents.
Book a Consultation