Ollama Explained: The CLI Tool for Running AI Models Locally
What Ollama does, what hardware it needs, and when local beats cloud
Ollama is a free, open-source command-line tool that runs large language models directly on your own computer. You download a model once, then chat with it, script it, or wire it into an app without sending a single token to a cloud API.
It matters because it removes two costs at once: per-token API bills and the requirement to send your data anywhere. A developer testing prompts all day, or a company that cannot let contract text leave the building, gets a real option instead of a workaround.
This guide covers what Ollama actually is, what hardware it needs, how the model catalog works, how to install it, and where it breaks down compared to a hosted model. We tested the honest limits, not just the sales pitch.
What Is Ollama?
Ollama is a local runtime for open-weight language models, distributed as a single binary for macOS, Linux, and Windows.
It wraps llama.cpp-style inference under a simple CLI: run "ollama pull llama3.1" to download a model, then "ollama run llama3.1" to chat with it in your terminal.
Under the hood, it also starts a background server on port 11434 that exposes a REST API, so any app on your machine can call the model the same way it would call OpenAI's API.
The project is MIT-licensed and free. Ollama also sells an optional paid cloud tier for running larger models on hosted GPUs, but the local tool itself has no subscription.
- CLI + local REST API in one install
- Runs fully offline once a model is downloaded
- MIT license, free and open source
- Optional paid cloud tier for bigger models (verify current pricing on ollama.com)
Not sure whether local models or cloud APIs fit your workflow? We can map it out.
Book a ConsultationWhat Hardware Does Ollama Need?
Ollama's hardware need scales directly with model size, because the model's weights have to fit in RAM or VRAM to run at a usable speed.
A quantized 7B model needs roughly 8GB of RAM. A 13B model needs about 16GB. A 33B model needs around 32GB, and a 70B model needs roughly 64GB.
These numbers are for CPU-only inference and cover the model file plus runtime overhead like the KV cache. A GPU with enough VRAM speeds things up a lot, but Ollama runs on CPU alone if no compatible GPU is present.
Apple Silicon Macs (M1 through M5) use unified memory, so a 16GB or 32GB MacBook can run mid-size models reasonably well without a discrete GPU.
- 7B models: ~8GB RAM
- 13B models: ~16GB RAM
- 33B models: ~32GB RAM
- 70B models: ~64GB RAM
- GPU (NVIDIA CUDA, AMD ROCm, Apple Metal) is optional but speeds up generation significantly
What Models Can You Run on Ollama?
Ollama's model library covers most major open-weight families, pulled straight from a searchable catalog on ollama.com.
That includes Llama, Qwen, DeepSeek, Mistral, and Kimi, plus smaller purpose-built models like Gemma and Phi.
Each model ships in multiple quantization levels, so you can trade accuracy for speed and memory footprint depending on your machine.
New releases usually land in the Ollama library within days of the model's open-weight launch, since the format conversion is largely automated.
- Llama family (Meta)
- Qwen and DeepSeek (frequently updated, strong coding performance)
- Mistral models (efficient at smaller sizes)
- Kimi and other newer open-weight releases
- Multiple quantization tiers per model (e.g. Q4, Q8) for different hardware
How Do You Install and Run Ollama?
Installing Ollama takes one command or one download, depending on your OS.
On macOS and Windows, download the installer from ollama.com and run it. On Linux, run the official install script from the terminal.
Once installed, pull a model and run it with two commands: "ollama pull <model-name>" then "ollama run <model-name>".
Ollama also ships a Docker image, which is the more repeatable option if you're running it on a server rather than a laptop.
- macOS / Windows: download the installer from ollama.com
- Linux: official shell install script
- Docker: official
ollama/ollamaimage - Package managers: Homebrew, Pacman, Nix also carry it
How Does Ollama Integrate With Other Tools?
Ollama integrates with other tools through its local REST API, which most chat UIs and dev frameworks already know how to speak.
Open WebUI is the most common front end — it gives Ollama a browser-based chat interface that looks like ChatGPT, running entirely against your local models.
Continue and Cline, both VS Code extensions, can point at Ollama instead of a cloud API for AI-assisted coding, which keeps proprietary code off external servers.
LangChain, CrewAI, and AutoGen all ship native Ollama connectors, so you can build local-first agents without swapping frameworks.
- Open WebUI — browser chat interface
- Continue, Cline — local coding assistants in VS Code
- LangChain, CrewAI, AutoGen — agent frameworks with native Ollama support
- Any OpenAI-API-compatible tool, via Ollama's compatibility layer
Ollama vs. Cloud APIs: When Does Local Win?
Ollama wins when data privacy, offline access, or per-token cost control matters more than raw model quality.
A cloud model like a frontier Claude or GPT release will outperform anything you can run on a laptop, because those models are far larger than what fits on consumer hardware.
The tradeoff is real: a local 7B or 13B model handles summarization, drafting, and simple classification fine, but struggles with multi-step reasoning or long-context tasks that a hosted model handles easily.
Teams that need both often run Ollama for high-volume, low-stakes tasks and route anything complex to a cloud API — the same split-routing pattern we use across our own routine automation portfolio, where cheap local or small-model steps handle bulk classification and only the harder judgment calls go to a larger cloud model.
- Choose Ollama for: privacy-sensitive data, offline use, unlimited local iteration, zero per-token cost
- Choose a cloud API for: frontier reasoning, long context, highest accuracy on hard tasks
- Common pattern: local model for bulk/cheap steps, cloud model for the hard steps
What Are Ollama's Real Limits?
Ollama's biggest limit is model quality — you're capped by what fits on your hardware, not by what the best available model can do.
Multi-user serving is another weak spot. Ollama is built for one machine running one or a few concurrent requests, not for serving hundreds of users the way a production API does.
Context window varies by model and by how much RAM you can spare for the KV cache, so a long document that a cloud model handles easily can choke a local model on modest hardware.
Setup also has real friction: picking the right quantization, managing GPU drivers, and troubleshooting slow generation are all on you, with no support line to call.
- Model quality ceiling is your hardware, not the state of the art
- Not built for high-concurrency multi-user serving
- Effective context window shrinks on lower-RAM machines
- No vendor support — you're your own IT department
Ollama vs. LM Studio: Which Should You Pick?
Ollama fits developers who want a CLI and API-first workflow; LM Studio fits people who want a graphical app with no terminal involved.
LM Studio ships a full desktop UI for browsing, downloading, and chatting with models, which makes it friendlier for non-technical users.
Ollama's strength is scriptability — it's easier to wire into an app, a cron job, or an agent framework because everything is a command or an API call.
For a full side-by-side, see our Ollama vs LM Studio comparison.
- Ollama: CLI + API first, better for scripting and integration
- LM Studio: GUI first, better for browsing and manual chat
- Both are free and run the same open-weight model families
Frequently Asked Questions
- Yes. Ollama the local tool is free and open source under the MIT license. Ollama also offers an optional paid cloud tier for hosted GPU access — verify current pricing on ollama.com before assuming it applies to local use.
- No. Ollama runs on CPU alone, though a GPU with enough VRAM makes generation significantly faster. Apple Silicon Macs use unified memory and can run mid-size models well without a discrete GPU.
- It depends on the model size: roughly 8GB for a 7B model, 16GB for a 13B model, 32GB for a 33B model, and 64GB for a 70B model. These are approximate figures for quantized models — check the specific model page on ollama.com for exact sizing.
- Yes, once a model is downloaded. Ollama pulls the model file once, then runs entirely offline with no network calls needed for inference.
- No, not for hard reasoning tasks. Local models are capped by what fits on consumer hardware, so they lag behind frontier cloud models on complex, long-context, or multi-step tasks. They work well for simpler tasks like summarization, drafting, and classification.
- Ollama is used to run open-weight language models locally for privacy-sensitive work, offline development, cost-free iteration, and building local-first AI apps and coding assistants.
- Yes. Extensions like Continue and Cline connect directly to Ollama's local API, letting you use AI coding assistance without sending code to an external server.
Deciding Between Local and Cloud AI Models?
The right split between local models and cloud APIs depends on your data sensitivity, volume, and the tasks you actually need solved. We help teams design that routing instead of guessing.
Book a Consultation