How to Run Nemotron Locally

Yes, you can run Nemotron locally, and if you already own NVIDIA GPUs, it is one of the most hardware-optimized choices available. Nemotron models are trained and tuned specifically for CUDA and TensorRT-LLM, which means they extract more performance per GPU dollar on NVIDIA hardware than a generic open-weight model would.

Nemotron ships in two main tiers: Nemotron-2 (smaller variants in the 15B range) and Nemotron-3, a hybrid Mamba-Transformer mixture-of-experts family that comes in Nano, Super (120B total, 12B active) and Ultra (550B total, 55B active) sizes. Nemotron 3 Super's NVIDIA Nemotron Open Model License and Nemotron 3 Ultra's OpenMDW-1.1 license both permit commercial use.

This guide covers choosing between Nemotron-2 and Nemotron-3, the quantization options, how NVIDIA's own deployment tools compare to generic serving engines, and a step-by-step path to a working local deployment.

Can You Run Nemotron Locally?

Yes — Nemotron-2 runs on a single workstation-class GPU, and Nemotron 3 Super runs on a single B200 or DGX Spark in NVFP4, or a multi-GPU server at higher precision. Both are realistic self-hosting targets for teams that already have NVIDIA hardware.

The key advantage is first-party tooling. NVIDIA provides TensorRT-LLM for optimized inference, NIM containers for one-command deployment, and integration with its own hardware profiling tools. No other model family gets this level of vendor-side deployment support on NVIDIA GPUs.

If you have NVIDIA GPUs and want the most optimized local inference, Nemotron is the model family to evaluate first. If you do not have NVIDIA hardware, other open-weight models with broader hardware support may be a better fit.

• Nemotron-2 (~15B) — runs on a single workstation GPU with 24+ GB VRAM.
• Nemotron 3 Super (120B total, 12B active) needs 8x H100-80GB at BF16, 2x H100-80GB at FP8, or a single B200 or DGX Spark at NVFP4.
• First-party NVIDIA tooling — TensorRT-LLM, NIM containers, hardware profiling.

Fix a GGUF That Will Not Load

Most detection failures happen when a GGUF file sits in the wrong folder. LM Studio expects files inside models/publisher/model/file.gguf. A loose file in the root models directory may not show up, with no error. Move it into its own subfolder or import it with the lms import command. If it is in the right place and still will not load, update LM Studio's runtime.

• Move the file into publisher and model subfolders. LM Studio stores models in ~/.lmstudio/models/ with publisher/ and model/ subfolders ending in the .gguf file to preserve the download structure from Hugging Face. A GitHub issue reports that placing a file directly in the models folder leaves it undetected, with no error or hint until it is placed in a subfolder. The Unsloth guide suggests checking that the file sits at ~/.lmstudio/models/publisher/model/model-file.gguf.
• Import the model using the command line. Documentation from LM Studio recommends running lms import <path/to/model.gguf> to position the file automatically. The command provides a dry-run option. It also includes a symbolic link option (lms import /path/to/model.gguf --symbolic-link) for loading large files stored on a secondary drive.
• Update the runtime engine. A model in the right folder can still fail to load when the installed runtime is out of date; updating LM Studio fixed the same load error for other architectures. Newly released model GGUF files have also failed on a current runtime while earlier versions worked. Update LM Studio's runtime, then try loading the file again.
• Check runtime support for Nemotron builds. A very new Nemotron release may need a newer runtime than the one installed. As a suggestion (not a documented LM Studio procedure), check the model card on Hugging Face for the minimum runtime version.

Picking the Right Nemotron Size for Your Hardware

Nemotron-2 and Nemotron-3 serve different hardware tiers. Pick the one that matches what you already own, then size up only if your task needs the larger model's capability.

Nemotron-2 (roughly 15B parameters) fits on a single NVIDIA GPU with 24 GB or more of VRAM — an RTX 4090, an A5000, or an L4. This is the right starting point for most teams, because it runs on hardware that a single workstation provides and still delivers strong performance on coding, reasoning, and instruction-following tasks.

Nemotron 3 Super (120B total, 12B active) needs at least 8x H100-80GB at BF16, 2x H100-80GB at FP8, or a single B200 or DGX Spark at NVFP4, per NVIDIA's model cards. Nemotron 3 Ultra (550B total, 55B active) needs at least 8x B200 or 16x H100 at BF16, or 4x B200 or 8x H100 at NVFP4. These tiers are for teams that need the strongest capability and already have server-class NVIDIA hardware or a cloud budget for it. For the GPU counts NVIDIA lists per precision, see what hardware Nemotron 3 Super needs and how many GPUs Nemotron 3 Ultra takes.

• Nemotron-2 (~15B) — single GPU, 24+ GB VRAM (RTX 4090, A5000, L4).
• Nemotron 3 Super (120B): 8x H100 at BF16, 2x H100 at FP8, or 1 B200 / DGX Spark at NVFP4.
• Nemotron 3 Ultra (550B): 8x B200 or 16x H100 at BF16, or 4x B200 or 8x H100 at NVFP4.
• Start with Nemotron-2 unless your task clearly needs the larger model.

Quantization: Running Nemotron on Less Hardware

Quantization shrinks Nemotron's memory footprint by storing weights at lower numeric precision. For Nemotron 3 Super, NVFP4 is the difference between needing eight H100s at BF16 and running on a single B200 or DGX Spark.

NVIDIA's own TensorRT-LLM supports INT8 and FP8 quantization natively, with minimal quality loss on most tasks. This is a significant advantage over generic quantization tools: TensorRT-LLM calibrates quantization against NVIDIA hardware specifically, producing better speed-to-quality ratios than a generic GGUF conversion.

Community GGUF quantized builds also appear on Hugging Face for Nemotron models, which work with Ollama and llama.cpp. These are a good fit for quick prototyping, but for production serving on NVIDIA GPUs, TensorRT-LLM quantization is the better path.

• TensorRT-LLM — native INT8 and FP8 quantization, optimized for NVIDIA GPUs.
• GGUF (community) — works with Ollama and llama.cpp for quick prototyping.
• NVIDIA's NVFP4 checkpoint of Nemotron 3 Super runs on a single B200 or DGX Spark.
• TensorRT-LLM calibration produces better quality-speed ratios than generic tools.

Ollama, VLLM, and SGLang: Which Tool?

For Nemotron, NVIDIA's own tools are the strongest option, but the generic ecosystem works too. The choice depends on whether you want maximum performance or maximum simplicity.

TensorRT-LLM with NIM containers is the first-party path. NVIDIA's build portal provides pre-built NIM container images for Nemotron that handle model loading, quantization, and serving in a single Docker container. This is the fastest path to production-grade inference on NVIDIA hardware.

Ollama supports Nemotron models and is the easiest way to try them locally. vLLM and SGLang also serve Nemotron well for production workloads, especially if your infrastructure already uses those tools for other models and you want a single serving stack.

• Maximum NVIDIA performance — TensorRT-LLM + NIM containers from build.nvidia.com.
• Quick local start — Ollama, one command to pull and run Nemotron.
• Existing infrastructure — vLLM or SGLang, if you already run other models on them.

A Practical Setup Outline

Getting Nemotron running locally follows two paths: the NVIDIA-native path (TensorRT-LLM and NIM) for production, or the Ollama path for quick prototyping. Both work, but the NVIDIA path extracts more performance from the same hardware.

Start with Nemotron-2 on Ollama to validate that the model fits your task. If it does, move to TensorRT-LLM for production serving. If Nemotron-2 falls short, evaluate Nemotron-3 before jumping to a different model family.

Keep a test harness of your own real prompts from day one. This tells you whether the model, the quantization level, and the serving config meet your quality and latency bar.

• Step 1 — Confirm you have NVIDIA GPUs with sufficient VRAM (24+ GB for Nemotron-2; for Nemotron 3 Super, at least 2x H100-80GB at FP8 or one B200 or DGX Spark at NVFP4).
• Step 2 — Quick test: ollama pull nemotron and run it to validate the model fits your task.
• Step 3 — For production: pull the NIM container from build.nvidia.com and launch with Docker.
• Step 4 — Choose quantization (FP8 or INT8 via TensorRT-LLM) if you need to fit on fewer GPUs.
• Step 5 — The NIM container exposes an OpenAI-compatible endpoint. Point your app at it.
• Step 6 — Run your eval harness and monitor latency, throughput, and quality.

Licensing and When to Self-Host vs Use the API

Nemotron licenses vary by model. Nemotron 3 Super ships under the NVIDIA Nemotron Open Model License, and Nemotron 3 Ultra ships under OpenMDW-1.1, which NVIDIA describes as a Linux Foundation permissive license that Nemotron releases are moving to. Both permit commercial use, but neither is MIT or Apache 2.0, so read the terms on each model card before you redistribute a modified version.

Self-host Nemotron when you have NVIDIA GPUs, need data to stay on your hardware, or want the TensorRT-LLM performance advantage. The combination of first-party tooling and hardware optimization makes Nemotron one of the smoothest self-hosting experiences for teams already invested in NVIDIA infrastructure.

Use a hosted NVIDIA API (via build.nvidia.com or cloud partners) when you want to start fast without provisioning hardware, or when your usage is too low to justify dedicated GPUs. The hosted option uses the same NIM serving stack, so migration to self-hosted later is straightforward.

• Licenses vary by model: NVIDIA Nemotron Open Model License (Nemotron 3 Super) or OpenMDW-1.1 (Nemotron 3 Ultra); both allow commercial use. Review redistribution terms.
• Self-host — data residency, TensorRT-LLM optimization, existing NVIDIA hardware.
• Hosted API (build.nvidia.com) — no hardware, same NIM stack, easy migration to self-hosted later.

Running the New Nemotron 3.5 Lightning Locally

Nemotron 3.5 Lightning is NVIDIA's newest open-weight option to run locally, released August 11, 2026. It is a 30B mixture-of-experts model that activates only 3 billion parameters per token (a "30B-A3B" design), so it runs far lighter than its total size suggests.

The architecture interleaves Mamba-2 layers, MoE layers, and select attention layers, and it supports up to a 1 million token context window. NVIDIA ships two checkpoints: an NVFP4 (4-bit) build for the smallest memory footprint, and a BF16 build for full precision. The NVFP4 checkpoint is the practical choice for single-GPU local use.

Because only 3B parameters are active per token, Nemotron 3.5 Lightning targets fast, long-running agent workloads on RTX and DGX hardware. Serve it with vLLM or an NVIDIA NIM container, and pull the weights from Hugging Face or a hosted endpoint on build.nvidia.com. It is licensed under OpenMDW-1.1, which opens the weights, data, and training recipes.

• Nemotron 3.5 Lightning — 30B total, 3B active per token, 1M context.
• NVFP4 (4-bit) checkpoint for single-GPU local use; BF16 for full precision.
• Serve with vLLM or NVIDIA NIM on RTX or DGX hardware.

What you need to run Nemotron yourself

Nemotron needs real memory, but it is within reach of a high-end workstation or a couple of professional GPUs — and many teams simply rent instead of buying. Match the path below to whether you want to own the hardware or pay by the hour.

To put Nemotron to work once it is live, connect a coding client like Cursor (via OpenRouter) or a local runner such as Ollama.

Apple Mac Studio (M4 Max, 128GB)

A single quiet on-prem box

View on Amazon →

NVIDIA RTX 6000 Ada (48GB)

Power users who want hardware they own

View on Amazon →