How to Use Nemotron 3.5 Lightning

You can use Nemotron 3.5 Lightning in three ways: the hosted API on build.nvidia.com, the open weights on Hugging Face, or a self-hosted server with a stack like vLLM. Pick the path that matches your control and budget needs.

The model is a 30B-total, 3B-active Mixture-of-Experts with a context window up to 1 million tokens. NVIDIA built it for long-running agents that execute specialized tasks.

This guide walks each access path. It also covers picking the NVFP4 or BF16 checkpoint and connecting the model to an agent loop.

Three Ways to Access the Model

There are three main paths to run Nemotron 3.5 Lightning. Hosted API, downloaded weights, or your own served endpoint.

The hosted API on build.nvidia.com is fastest to start. You call an endpoint and pay per token, with no hardware to manage.

The Hugging Face weights let you download the model and run it yourself. This is free to use, though you supply the GPU and operations.

A self-hosted server with vLLM sits in the middle. You host the weights but expose them as an API your apps can call.

Path 1: The Hosted API on Build.nvidia.com

The hosted API is the quickest way to send your first request. Sign in at build.nvidia.com and open the model page.

Create an API key from your account. Keep it secret and store it as an environment variable, not in your code.

The endpoint follows a standard chat completion format. You send messages and a model id, then read the response text.

This path suits teams that want results today. You skip GPU setup and let NVIDIA handle the serving.

The endpoint follows a common chat format, so most existing client libraries work with a small config change. Point the base URL at build.nvidia.com and set the model id.

Your First Request, Step by Step

Your first request needs three things: a key, a model id, and a prompt. Start with a short test prompt to confirm the wiring.

First, set your API key as an environment variable in your shell. This keeps it out of your source files.

Second, point the client at the build.nvidia.com endpoint and pass the model id in the form NVIDIA-Nemotron-3.5-Lightning-30B-A3B. Send a single user message.

Third, read the reply and check the token counts. Once a small prompt works, scale up to real tasks and longer context.

Path 2 and 3: Weights and Self-Hosting

To self-host, download the weights from Hugging Face and serve them with a stack like vLLM. This gives you full control and no per-token bill.

The model runs on NVIDIA hardware from RTX workstations up to DGX systems. Your VRAM and throughput needs decide the tier.

vLLM exposes an OpenAI-style API on your own machine. Your apps then call localhost instead of a vendor endpoint.

Self-hosting fits teams with strict data rules or steady high volume. You trade setup work for privacy and predictable cost.

Pick your checkpoint to match your GPU. NVFP4 fits smaller cards, while BF16 needs more memory but gives you the reference quality.

NVFP4 vs BF16: Which Checkpoint to Pick

Pick NVFP4 for speed and lower memory, and BF16 for maximum quality. Both checkpoints live on Hugging Face.

NVFP4 is a 4-bit format. It shrinks the model and speeds up inference, which helps on smaller GPUs and high-volume serving.

BF16 keeps 16-bit precision. NVIDIA measured its published benchmarks on this checkpoint, so it is the reference for quality.

A common plan is to prototype on BF16, then test NVFP4 for production. If NVFP4 holds your accuracy bar, you gain speed and save memory.

Wiring It into an Agent Workflow

To use the model in an agent, expose your tools, pass them to the model, and run a loop that executes tool calls. The model was built for exactly this.

Define each tool with a name, a description, and a schema. The model picks a tool and returns arguments for it.

Your code runs the tool, feeds the result back, and lets the model decide the next step. The 1 million token context lets long runs keep their history.

Across the model launches we track, the teams that win start with a narrow agent and a tight tool set. In our client work, the recurring failure mode is handing a new model ten tools on day one and losing the ability to debug. Add tools once the base loop is stable.

Log every tool call and result while you build. When an agent goes off track, that trace shows you the exact step that failed.

Using the 1 Million Token Context

The 1 million token context lets a single session hold huge inputs, like a whole codebase or a long task history. Feed it what the task truly needs.

Long context is powerful, but it is not free on a hosted endpoint. Every extra token you send is billed and adds latency.

A good rule is to include the files, logs, or history the model must see, and drop the rest. Precision beats stuffing the window full.

For agents, the large window means fewer restarts. The model can keep its plan and past steps in view across a long run.

Test with a realistic history size early. That tells you how the model behaves near the context limits before real users push it there.

Where NeMo Switchyard Fits

NeMo Switchyard is a separate model router that shipped alongside Nemotron 3.5 Lightning. It is not the model, and you do not need it to use the model.

A router sends each request to the best model for the job. That can pair a small fast model with a larger one for hard cases.

You can run Nemotron 3.5 Lightning on its own with no router at all. Start simple, then add routing only if you run several models.

Keep the two ideas separate in your plans. The model executes tasks, while the router decides which model handles each request.

If you already run a fleet of models, a router can trim cost by sending easy work to the cheapest capable model. For a single-model setup, skip it and add it later if your needs grow.

Sensible Next Steps

Start on the hosted API, confirm the model fits your task, then decide whether to self-host. This order lowers risk and cost.

Keep a small test set of real prompts. Run it on both checkpoints so you can see the speed and quality tradeoff for yourself.

Log tokens and latency from day one. Those numbers guide your checkpoint choice and your hosting decision.

When your workload is steady and private, move to self-hosted vLLM. Until then, the hosted API keeps you moving fast.