How to Run Inkling Locally

Running Inkling locally is possible but demanding. Thinking Machines' flagship is a frontier-class Mixture-of-Experts model with a parameter count in the range of DeepSeek-V3 or GLM-5.2, which means it is a multi-GPU server project, not a laptop download.

Inkling is notable as a frontier open-weight model from a Southeast Asian AI lab, demonstrating that frontier capability is no longer limited to a handful of US and Chinese labs. The open-weight license permits commercial use.

This guide gives you the honest hardware reality, the quantization options that lower the entry point, which serving tools handle a model this large, and the case for using a hosted endpoint instead — which is the right default for most teams.

Can You Run Inkling Locally?

You cannot run the full Inkling model on a laptop or a single consumer GPU. At frontier MoE scale, the full weights need a multi-GPU server with high-bandwidth interconnect, similar to what DeepSeek-V3 or GLM-5.2 require.

Inkling uses a Mixture-of-Experts architecture, so it activates only a fraction of its parameters per request. This lowers compute per token, but the full set of expert weights still must be held in memory across the cluster. Memory, not compute, is the binding constraint.

The honest answer: an organization with real GPU infrastructure can self-host Inkling. An individual on a workstation cannot run the full model. Plan for a server project with specialized hardware and engineering time.

• Not a laptop model — the full weights need a multi-GPU server.
• MoE lowers compute per token but not the memory to hold all expert weights.
• Realistic for organizations with GPU clusters; not for a single workstation.

The Hardware Reality of a Frontier MoE Model

The core requirement is enough GPU memory to hold all of Inkling's expert weights across many accelerators at once. A frontier MoE model does not fit on one or two GPUs, so you need a multi-GPU, often multi-node, cluster with high-bandwidth interconnect between the cards.

Expect to need a cluster of 8 or more high-memory GPUs (A100 80 GB, H100, or equivalent) at full precision. The exact count depends on the model's total parameter count, the precision you run, the quantization you apply, and your throughput target. Those choices move the requirement substantially.

Beyond the GPUs, budget for the surrounding system: NVLink or InfiniBand networking so the experts communicate without stalling, substantial host RAM and fast NVMe storage to load the weights, and power and cooling for a sustained heavy workload.

• Memory-bound — all expert weights must fit across the GPU cluster simultaneously.
• Expect 8+ high-memory GPUs (A100 80 GB, H100) at full precision.
• Multi-node clusters need NVLink or InfiniBand for acceptable latency.
• Plus host RAM, fast NVMe storage, power, and cooling.

Quantization: Fitting Inkling on Less Hardware

Quantization is the main lever to reduce Inkling's hardware footprint. Storing the weights at lower numeric precision (INT8, INT4, or FP8) shrinks the memory requirement so the model fits on fewer GPUs.

For frontier MoE models, the quality-cost trade-off is real. Aggressive quantization (4-bit) can noticeably degrade output on hard reasoning tasks, because the expert routing is sensitive to weight precision. INT8 or FP8 is a safer starting point that still cuts the GPU count meaningfully.

Watch for community quantized builds after the open-weight release. Popular frontier models usually get GPTQ, AWQ, and GGUF variants on Hugging Face within weeks, which can meaningfully lower the entry hardware for Inkling.

• INT8 or FP8 — safest quantization for MoE models, meaningful memory savings.
• 4-bit (INT4) — more aggressive, may degrade quality on hard reasoning tasks.
• Community builds (GPTQ, AWQ, GGUF) typically appear within weeks of release.
• Test your own prompts at each precision level before committing to a quantization.

Ollama, vLLM, and SGLang: Which Tool?

For serving Inkling at production scale, high-throughput engines like vLLM and SGLang are the right tools. They handle multi-GPU tensor parallelism, expert routing, and concurrent request batching — all essential for a frontier MoE model. Ollama is not designed for models at this scale.

SGLang in particular has strong MoE support and may be the best fit for Inkling if the community validates compatibility after the open-weight release. vLLM is the most broadly adopted alternative with solid multi-GPU serving.

For development and testing, run a smaller open model in Ollama to build and test your workflow, then swap the endpoint to a full Inkling deployment for production. The OpenAI-compatible API interface means your client code stays the same.

• Production serving — vLLM or SGLang across a multi-GPU cluster.
• SGLang — strongest MoE support; likely the best fit pending community validation.
• Ollama — not practical for the full model; use it for prototyping with smaller models.
• OpenAI-compatible endpoints let you swap between local and hosted seamlessly.

A Realistic Setup Outline

Self-hosting Inkling follows a repeatable path, but each step involves real infrastructure work. The point of writing it out is to make the effort visible before you commit hardware.

Start with a hosted endpoint first. Validate that Inkling's output quality justifies the model over a smaller open-weight alternative. Only move to self-hosting for the workloads that genuinely need it.

Keep an evaluation harness from day one. The same set of real prompts you use to test quality also tells you when a quantization level, a serving config, or a new model build has regressed.

• Step 1 — Validate quality on a hosted Inkling endpoint first.
• Step 2 — Download the weights from Thinking Machines' official Hugging Face or GitHub release.
• Step 3 — Choose precision and quantization (INT8 or FP8 recommended) for your hardware and quality bar.
• Step 4 — Serve with vLLM or SGLang across a multi-GPU cluster with tensor parallelism.
• Step 5 — Put an OpenAI-compatible gateway in front so app code stays unchanged.
• Step 6 — Run your eval harness and monitor cost, latency, and quality continuously.

When to Self-Host vs Use the API

Self-host Inkling when data residency or a compliance rule requires it, and you have the GPU infrastructure and ML engineering team to run a frontier MoE model. That is the only situation where the hardware burden is worth carrying.

Use a hosted API for everything else. For most teams and most workloads, the hosted endpoint is faster to start, cheaper at low to medium volume, and requires no infrastructure management. The API gives you the same model quality without the cluster.

A hybrid is often the most practical approach: route regulated or sensitive traffic to a self-hosted deployment and everything else to the hosted API. This way you pay the self-hosting premium only where it actually buys you compliance or data control.

• Self-host — data residency, compliance, and in-house ML engineering team.
• Hosted API — speed, lower cost, no hardware, fits most teams and workloads.
• Hybrid — self-host only the regulated traffic; API for everything else.

What you need to run Inkling yourself

Inkling is a frontier-scale Mixture-of-Experts model, so "running it yourself" is a real infrastructure decision — not something a single laptop or gaming GPU can do. Match the path below to how seriously you need to self-host. For most teams the API or rented GPUs are the right answer; buying hardware only pays off at steady, high volume or when your data can never leave your walls.

Once Inkling is running, the fastest way to put it to work day to day is inside Cursor — point it at the model through OpenRouter as a custom model. And if you would rather run a model on one affordable box, see Best mini PCs for local AI and Local AI hardware calculator.

NVIDIA RTX 6000 Ada (48GB)

Power users and small clusters that want cards they own

View on Amazon →