How to Run Nemotron 3.5 Lightning Locally

You can run Nemotron 3.5 Lightning on your own hardware. The weights are open under the OpenMDW-1.1 license and hosted on Hugging Face. Self-hosting costs $0 in license fees.

The model is a 30B total, 3B active Mixture-of-Experts. That split matters for hardware. You must fit all 30B in memory, even though only 3B run per token.

This guide covers the choices that decide your setup. You will pick a checkpoint, a GPU path, and a serving stack, then bring the model up step by step.

The VRAM Reality of a 30B-A3B Model

The key rule is simple: you must hold all 30B parameters in memory, not just the 3B active. The router can call any expert at any token, so every weight must be loaded.

The checkpoint format sets the size. A 4-bit checkpoint needs far less memory than a full 16-bit one. The context window you use adds more memory on top for the key-value cache.

A 1 million token context can use a large amount of cache memory. Start with a smaller context to fit on one GPU, then scale up as your hardware allows.

NVFP4 vs BF16: Which Checkpoint to Pick

Pick NVFP4 for the lightest footprint. NVFP4 is a 4-bit format, so it holds the 30B weights in roughly a quarter of the memory of BF16. It is the practical choice for a single consumer GPU.

Pick BF16 for the full-precision reference. BF16 is the 16-bit checkpoint and matches NVIDIA's published benchmark numbers. It needs the most memory and the most capable hardware.

The tradeoff is memory versus fidelity. NVFP4 may show tiny quality shifts on hard tasks, but it fits where BF16 will not. Both checkpoints are on Hugging Face.

RTX vs DGX: Which Hardware Path

Choose the RTX path for a local workstation. NVIDIA highlights RTX as a supported target for this model. The NVFP4 checkpoint is what makes a single RTX card viable.

Choose the DGX path for scale and the full BF16 model. DGX systems carry far more memory and can serve the full-precision checkpoint with a large context window.

Match the path to the job. RTX suits development, testing, and light local serving. DGX suits production traffic, big batches, and the maximum context window.

The Serving Stack: VLLM or NVIDIA NIM

Pick vLLM for an open-source server. vLLM loads the Hugging Face weights and exposes an API endpoint your apps can call. It handles batching and the key-value cache for you.

Pick NVIDIA NIM for a packaged, supported container. NIM wraps the model in a ready-to-run microservice with an OpenAI-style API. It trades some flexibility for a smoother setup.

Both give you a local endpoint. Your agents and analysis tools then call that endpoint instead of a hosted API. The choice is open-source control versus a managed container.

Step by Step: Bring the Model Up

First, confirm your GPU and driver stack. Check that your card has enough memory for your chosen checkpoint plus context cache. NVFP4 on RTX is the easiest start.

Second, download the weights from Hugging Face. Choose the NVFP4 or BF16 repository that matches your hardware. Accept the OpenMDW-1.1 license terms as prompted.

Third, install your serving stack. For vLLM, install the server and point it at the downloaded model id. For NIM, pull and run the container image.

Fourth, start with a modest context length. Bring the server up, send one test prompt, and confirm the response. Then raise the context window toward 1 million tokens as memory allows.

In our work evaluating open-weight models for clients, the recurring failure mode is planning for active parameters instead of total. Budget memory for all 30B, and the rest of the setup is routine.

How the KV Cache Grows with Context

The weights are only half of your memory budget; the KV cache is the other half. The cache holds the key and value tensors for every token in the context. It grows as the context grows.

A short prompt uses a small cache. A 1 million token context uses a large one. That cache sits in GPU memory on top of the 30B weights you already loaded.

This is why context length drives your hardware, not just the checkpoint size. Two setups with the same weights can need very different memory if their context windows differ.

The hybrid design helps here. Interleaved Mamba-2 layers carry long sequences at lower memory cost than pure attention. That is part of how the model reaches a 1 million token window.

Plan the cache and the weights together. Budget memory for all 30B parameters, then add headroom for the context cache. Start small and raise the window as memory allows.

Serving Config: Quantization and Context Settings

Set two things first in your serving config: the checkpoint and the max context length. Those two choices decide whether the model fits your GPU.

Match the checkpoint to your memory. NVFP4 is the 4-bit format and holds the 30B weights in roughly a quarter of the BF16 footprint. Pick NVFP4 for a single consumer GPU.

Pick BF16 when you have the memory to spare. The 16-bit checkpoint matches NVIDIA's published benchmark numbers and suits DGX-class systems. It needs the most memory of the two.

Then set the max context length to what you actually need. A lower limit shrinks the KV cache and frees memory. Raise it only when your task truly needs the extra tokens.

Leave batching to the server. A stack like vLLM manages batching and the KV cache for you. Start with defaults, confirm one prompt works, then tune for your traffic.

Troubleshooting a Local Nemotron Setup

Most local setup problems trace back to memory. Three symptoms cover the common cases, and each has a direct fix.

An out-of-memory error at load usually means the checkpoint is too big for your GPU. Switch from BF16 to the NVFP4 4-bit checkpoint. That alone frees most of the space you need.

An out-of-memory error partway through means the KV cache overran. Lower the max context length in your serving config. The cache shrinks, and the run completes.

Very slow output often means the model spilled onto system memory or disk. Fit the whole model in GPU memory instead. A model that stays on the GPU serves tokens far faster.

A checkpoint that will not load may need a newer serving stack. Update vLLM or your NIM container so it understands the NVFP4 format and the hybrid architecture.

In our work evaluating open-weight models for clients, the recurring failure mode is sizing memory for the 3B active count instead of the full 30B. Budget for the total, and most errors disappear.