Nemotron 3.5 Lightning Limits

Nemotron 3.5 Lightning has a context limit of up to 1 million tokens, and its other limits depend on how you run it. Hosted endpoints add rate limits; self-hosting removes per-call caps but adds hardware limits.

Every model has a shape it fits and a shape it does not. A 30B model with only 3B active parameters is fast, but that trade sets real bounds.

This guide covers the context window, hosted quotas, self-host limits, and the tasks this model is not the best pick for. Where numbers vary by vendor, we tell you to verify rather than guess.

The 1 Million Token Context Limit

Nemotron 3.5 Lightning supports up to 1 million tokens of context. That is the hard ceiling on how much you can feed one request.

One million tokens is large, enough for whole codebases or long document sets. Most tasks never come close to filling it.

A big window is not free. On a hosted endpoint you pay for every input token, so a full window costs real money. On self-host, a full window uses more memory and slows the response.

Quality can also drift across very long inputs. Test recall at the length you actually use, rather than assuming the full window works perfectly.

Facts buried deep in a long input can be missed. Place the most important details near the start or end of your prompt.

Treat the full window as a ceiling, not a target. Most tasks work better with a focused, shorter prompt.

Hosted Endpoint Rate Limits and Quotas

Hosted endpoints add rate limits and quotas that self-hosting does not have. These cap how many requests or tokens you can use in a window.

On build.nvidia.com or a third-party provider, expect caps on requests per minute and total tokens per period. Higher tiers usually raise those caps.

The exact numbers vary by provider and plan, and they change. Do not treat any figure in a guide as current. Check the live limits on build.nvidia.com or your provider before you scale.

Rate limits matter most for agents. A long-running agent can fire many calls fast, and a low limit will throttle it mid-task.

Self-host Limits: No Caps, but Hardware Bounds

Self-hosting has no per-call rate limits or quotas. You can send as many requests as your hardware can handle.

The real limit becomes your GPUs. Throughput depends on how much memory and compute you own.

The NVFP4 4-bit checkpoint lowers the memory bar, so it runs on smaller hardware. The BF16 checkpoint needs more memory but keeps full precision.

Serving stacks like vLLM help you batch requests and use hardware well. Your ceiling is set by your fleet, not by a vendor plan.

You also control uptime and versioning. Nothing changes under you without your say, which matters for stable production agents.

The trade is responsibility. No vendor absorbs a hardware failure for you, so plan for redundancy and monitoring.

The Limits of a 3B-active MoE

A 3B-active model is fast, but it thinks with fewer active parameters per token than a large dense model. That trade has real limits.

The model holds 30 billion total parameters, so its knowledge is broad. Only 3 billion fire per token, which favors speed over deep, single-pass reasoning.

NVIDIA tuned it for specialized task execution in agents, not for the hardest open-ended reasoning. For the toughest research or nuanced writing, a larger dense model may reason better.

In our work evaluating open-weight models for clients, the common mistake is pointing a fast task model at problems that need deep reasoning, then blaming the model for slow, shaky answers. Match the tool to the job.

What Nemotron 3.5 Lightning Is Not For

This model is not the best pick for every job. Knowing its weak spots saves you a bad deployment.

It is not built to be a cheap, tiny chat model for light use. Its value shows up in high-volume, multi-step agent work.

It is not a guaranteed win on the hardest reasoning tasks. Larger dense models can still edge it out where deep, single-shot reasoning matters most.

It is not a fit for tiny edge devices with little memory. Even the 4-bit checkpoint needs real GPU capacity to run well.

It is not a router. NVIDIA ships NeMo Switchyard separately for routing between models. Do not expect the model itself to choose models for you.

It is not a drop-in replacement for a large dense model on every task. Match it to fast, repeatable work, and keep a bigger model for the hardest jobs.

It is not a set-and-forget hosted service if you self-host. You own the uptime, the updates, and the scaling.

The Practical Limit of Long Context

The 1 million token window has a practical limit that is not about the number itself. It is about cost and speed at that length.

Every token you load costs compute. On a hosted endpoint, a near-full window can make a single call expensive.

Long inputs also add latency. The model reads more before it answers, so response time grows with context length.

Use the window when the task truly needs it, like whole-codebase work. For short tasks, a smaller prompt is faster and cheaper.

How Limits Hit Long-running Agents

Rate limits bite hardest in agent workloads. A long-running agent can send bursts of calls in seconds.

If a hosted plan caps requests per minute, the agent stalls when it hits the cap. That breaks the flow of a multi-step task.

Self-hosting removes that cap, but the hardware becomes the bottleneck. An overloaded GPU queues requests and slows every step.

Plan for peak load, not average load. Size your plan or your fleet for the busiest moment of an agent run.

Batching and retries help smooth the load. Good serving stacks queue and group requests so a burst does not fail outright.

Watch for silent slowdowns too. A throttled agent may not error, it may just crawl, which is harder to spot in production.

How to Verify Current Limits

Always confirm limits on the official pages before you build. Numbers move faster than guides do.

For context length and benchmarks, read the model card on build.nvidia.com and the weights pages on Hugging Face.

For rate limits and quotas, check your hosted provider's plan page. If you self-host, test throughput on your own hardware.

Treat published figures as a starting point, then measure against your real workload. Your traffic pattern is the true test.

Recheck the limits on a schedule, not just at launch. Vendors adjust quotas and pricing over time, and your own usage grows with your product.