LLM Tokens per Second Calculator for Inference Speed
Estimate generation speed ceilings across graphics processing units and Apple silicon using memory bandwidth and model precision.
Decode speed ceiling in tokens per second roughly equals memory bandwidth divided by bytes read from memory per token, and this tokens per second calculator runs that calculation across hardware configurations. Generating text with an 8-billion parameter model at 4-bit precision on an NVIDIA GeForce RTX 4090 with 1,008 gigabytes per second (GB/s) of memory bandwidth yields a ceiling of 252 tokens per second, or 126 tokens per second at 50% bandwidth utilization.
Large language model (LLM) text generation reads model parameters from memory to the processor for every generated token. Because this decode phase is memory-bound rather than compute-bound, raw memory bandwidth sets the upper limit on how many tokens an engine can produce each second.
Select your hardware preset or enter custom bandwidth values to compare generation ceilings across model parameter sizes, quantization levels, and active key-value (KV) cache footprints before purchasing hardware or deploying an inference workload.
LLM Tokens per Second Calculator
Tokens per second ≈ 1008 GB/s ÷ 3.7 GB per token, for one request on a dense model, following Databricks' memory-bandwidth method, whose worked example runs at 50% of peak bandwidth. Bandwidth from the vendor's specifications. Prompt processing (time to first token) is not included.
Reading Your Speed Estimate
The calculator displays speed projections as soon as you select a device preset or enter architecture parameters. Outputs update live without requiring a button press, separating hardware throughput ceilings from active memory requirements.
Read from memory per token shows the total memory traffic required to output a single token, combining model weights and the optional key-value cache. For an 8-billion parameter model at 4-bit precision with no key-value cache, this figure equals 3.7 gigabytes (GB).
Tokens per second, bandwidth ceiling is the speed if the model used 100% of the device's peak memory bandwidth. At 50% bandwidth utilization shows the speed at half that ceiling, the utilization in Databricks' worked example, not a measured speed for any device.
When model weights plus key-value cache exceed Device memory (GB), the calculator displays an error warning indicating that the model does not fit and the throughput estimate does not apply. You can evaluate hardware capacity using the What LLM Can I Run tool or click Copy result to copy plain text specifications directly.
Why Memory Bandwidth Governs Decode Speed
Databricks writes that generation speed depends on how quickly model parameters load from GPU memory to local caches and registers. Databricks defines model bandwidth utilization (MBU) as achieved memory bandwidth divided by peak theoretical memory bandwidth, where achieved bandwidth equals total model parameter size plus key-value cache size divided by time per output token (TPOT).
In the Databricks worked example, serving a 7-billion parameter model at 16-bit floating-point (FP16) precision with a 14-millisecond time per output token moves 14 GB of parameter data in 14 milliseconds, which equates to roughly 1 terabyte per second (TB/s) of memory traffic. On hardware providing 2 TB/s of peak theoretical bandwidth, that performance represents an MBU of 50%.
Rearranging this relationship demonstrates that generation speed ceiling equals peak memory bandwidth divided by bytes read per token. Larger models require transferring more parameters per token, which reduces throughput on identical memory subsystems.
Impact of Weight Precision and Parameter Count
Lowering weight precision cuts the total volume of memory traffic required for each output token, directly raising generation speed ceilings.
Evaluating an 8-billion parameter model at FP16 precision requires reading 14.9 GB of weight data per token. On an NVIDIA GeForce RTX 4090 with 1,008 GB/s bandwidth, the ceiling reaches 63 tokens per second, or 31.5 tokens per second at 50% utilization. Switching that same 8B model to 4-bit precision reduces memory traffic to 3.7 GB per token, quadrupling throughput to a 252 tokens per second ceiling.
A 32-billion parameter model quantized to 4-bit precision requires 14.9 GB of memory reads per token, matching the exact memory footprint of an 8B model at FP16 precision. Consequently, both configurations share an identical throughput ceiling of 63 tokens per second on an RTX 4090.
A 70-billion parameter model at 4-bit precision requires reading 32.6 GB per token. On an NVIDIA GeForce RTX 5090 providing 1,792 GB/s bandwidth, this workload produces a calculated ceiling of 51.2 tokens per second, or 25.6 tokens per second at 50% utilization. However, because 32.6 GB exceeds the RTX 5090's 32 GB of physical memory, the calculator triggers a warning that the model does not fit in device memory.
Comparing Discrete GPUs and Apple Silicon Architectures
Memory bandwidth dictates raw throughput ceilings across both dedicated graphics processing units (GPUs) and unified-memory computing architectures.
Evaluating an 8-billion parameter model at 4-bit precision (3.7 GB per token) illustrates these hardware differences. The NVIDIA GeForce RTX 5060 Ti with 448 GB/s provides a ceiling of 112 tokens per second (56 at 50% utilization). The NVIDIA RTX 6000 Ada provides 960 GB/s bandwidth and 48 GB of memory. An NVIDIA H100 SXM with 3.35 TB/s peak bandwidth delivers a ceiling of 837.5 tokens per second (418.8 at 50% utilization).
A Mac shares one pool of unified memory between CPU and GPU, so the bandwidth Apple lists for each chip sets its ceiling. The Apple Mac mini M4 Pro delivers 273 GB/s bandwidth, producing an 8B 4-bit ceiling of 68.2 tokens per second (34.1 at 50%). An Apple Mac Studio M3 Ultra provides 819 GB/s bandwidth, yielding 204.8 tokens per second (102.4 at 50%). For a 70B model at 4-bit precision (32.6 GB read per token), the Mac Studio M3 Ultra maintains a ceiling of 23.4 tokens per second, holding the complete parameter weights entirely in unified memory.
Incorporating Key-Value Cache Memory Overhead
Active key-value cache memory increases the total bytes transferred across the memory bus during each decode step, slowing generation speeds over extended contexts.
The calculator includes an optional input field labeled KV cache (GB, optional), which accepts values calculated using the companion KV Cache Calculator. By default, this value is set to 0 GB, reflecting a stateless baseline calculation.
Entering cache requirements shows the exact throughput penalty caused by context length. Running an 8-billion parameter model at 4-bit precision requires 3.7 GB of parameter data. Adding 1.75 GB of key-value cache data (the memory required for a Qwen2.5 7B model at 32,768 tokens) increases total memory traffic to 5.5 GB per token.
On an NVIDIA GeForce RTX 4090 with 1,008 GB/s bandwidth, adding this 1.75 GB cache reduces the generation ceiling from 252 tokens per second down to 171.5 tokens per second. At 50% utilization, throughput drops from 126 tokens per second to 85.7 tokens per second.
Operational Limits of Bandwidth Speed Estimates
Calculated speed ceilings isolate theoretical hardware memory transfer limits and omit external serving constraints encountered in production environments.
This calculator assumes a batch size of 1, representing a single isolated request. Serving concurrent users batches memory transfers across requests, amortizing weight retrieval costs and altering observed throughput per token.
The mathematical model applies strictly to dense neural architectures. Mixture-of-experts (MoE) architectures activate only a fraction of their total parameters for any single token, meaning this formula overstates bytes read per token for sparse models.
The formula models only the token decode phase and excludes prompt processing (prefill) and time to first token (TTFT). Real speed depends on the runtime and its kernels; the 50% figure is Databricks' worked example, not a measurement of these devices. If total model weights exceed available hardware memory, layers spill to slower memory, invalidating these memory bandwidth calculations.
For complete system sizing, review our Local AI Hardware Calculator, explore memory requirements in Best Local LLMs, or evaluate desktop systems in Best AI Workstations.
Frequently Asked Questions
- To calculate the decode ceiling in tokens per second, divide hardware memory bandwidth (in gigabytes per second) by the total gigabytes read from memory per token. The total bytes read per token equals the model weight size plus active key-value cache memory.
- There is no single standard figure for what constitutes good speed, as the LLM Tokens per Second Calculator gives a ceiling rather than a measured benchmark. Instead, users can compare theoretical ceilings and 50% utilization reference points across hardware options.
- Yes. Lowering precision shrinks weight sizes and reduces the memory traffic needed per token. For example, an 8-billion parameter model requires reading 14.9 GB per token at FP16 precision compared to 3.7 GB per token at 4-bit precision, quadrupling the theoretical bandwidth speed ceiling.
- Generation speed is bound by memory bandwidth. An NVIDIA GeForce RTX 5090 provides 1,792 GB/s of bandwidth, whereas an Apple Mac mini M4 Pro offers 273 GB/s and an M3 Ultra provides 819 GB/s. Higher memory bandwidth transfers weights faster during token generation.
- Yes. Extended context lengths enlarge the key-value cache footprint. Because the processor must retrieve keys and values for all past tokens alongside static weights, higher memory traffic per step reduces generation speeds.
Need help sizing inference hardware for your models?
Selecting GPUs or unified memory workstations depends heavily on parameter scale, quantization precision, and context length requirements. Layer3Labs reviews model architecture, concurrency targets, and memory bandwidth requirements to ensure workloads run efficiently on target hardware.
Book a Consultation