Groq Explained: The Fast-Inference API Built on LPU Chips

Groq is an inference platform that runs open-source AI models on custom chips called LPUs instead of GPUs. That swap is the whole story: Groq does not train its own models, it serves other labs' open-weight models - Llama, Qwen, GPT-OSS, and more - at speeds GPU-based APIs cannot match.

The company is easy to confuse with Grok, xAI's chatbot. They are unrelated. Groq sells inference infrastructure through GroqCloud; Grok is a consumer chat product built by a different company.

This guide covers how the LPU architecture creates the speed gap, which models Groq hosts, what the benchmarks actually show against OpenAI and Anthropic-hosted endpoints, and the specific situations where routing a workload to Groq is the correct engineering call - and where it is not.

What Is Groq?

Groq is a cloud inference provider that runs large language models on its own chip design, the LPU, instead of renting Nvidia GPUs like most competitors.

The company was founded by engineers who previously built Google's Tensor Processing Unit, then set out to design a chip purpose-built for one job: running a trained model's inference step as fast as physically possible.

GroqCloud is the product that exposes this hardware over a standard API. Developers send it prompts the same way they would call OpenAI or Anthropic, and it returns completions from models like Llama, Qwen, and GPT-OSS - just materially faster.

• Groq does not train foundation models - it hosts open-weight models from other labs
• The hardware is called an LPU (Language Processing Unit), not a GPU
• GroqCloud is the developer-facing API and console at console.groq.com

LPU vs GPU: Why Groq Is Faster

Groq is faster than GPU-based inference because its LPU chip removes the memory bottleneck that slows every GPU down mid-generation.

A GPU was designed for parallel math across many workload types - graphics, training, inference, simulation. During inference, a GPU spends much of its time waiting on data to move between its processor and its high-bandwidth memory (HBM), because token generation is a sequential, one-step-at-a-time process, not the wide parallel math GPUs excel at.

The LPU flips the design. It keeps the entire model's working weights in fast on-chip SRAM instead of external HBM, and it executes on a deterministic schedule - every clock cycle does a known, pre-planned operation instead of a GPU's dynamic scheduling. That removes the memory-wait stalls almost entirely.

The tradeoff is real, not free. SRAM is far more expensive per gigabyte than HBM, so Groq needs many chips wired together to hold a large model, and it cannot offer the massive per-chip memory GPUs use for training. Groq built a chip that wins at one job - serving inference fast - by giving up flexibility everywhere else.

• GPUs stall waiting on HBM memory during token-by-token generation; LPUs keep weights in on-chip SRAM
• LPU execution is deterministic (fixed schedule), GPU execution is dynamically scheduled
• SRAM costs more per gigabyte than HBM, so Groq needs more chips per model - it trades cost-per-chip for speed-per-chip

What Models Does Groq Host?

Groq hosts open-weight models from Meta, OpenAI's open releases, Alibaba's Qwen, Moonshot's Kimi, and its own agentic system called Groq Compound - it does not run closed models like GPT-4-class endpoints, Claude, or Gemini.

The catalog changes as new open models ship, since Groq's business is being first to serve them fast, not building its own. Llama 4 Scout and Llama 4 Maverick landed on GroqCloud on the same day Meta released them, which is typical of how Groq operates.

Reported throughput figures move with hardware and model updates, so treat any specific tokens-per-second number as a snapshot, not a guarantee.

• Llama 3.1 8B Instant - Meta's small, fast model, commonly the cheapest option on the platform
• Llama 3.3 70B Versatile - Meta's larger general-purpose model
• GPT OSS 20B and GPT OSS 120B - OpenAI's open-weight releases, hosted day-zero on Groq
• Qwen and Kimi models - Alibaba's and Moonshot's open releases
• Groq Compound - an agentic system with built-in web search and code execution
• Whisper - OpenAI's speech-to-text model, billed per audio hour instead of per token

How Much Does Groq Cost?

Groq prices per million tokens, with rates that run well below equivalent closed-model APIs because it is serving open-weight models, not training and amortizing its own frontier model.

Entry-level models like Llama 3.1 8B price near the bottom of the market, while larger models like Llama 3.3 70B or Qwen's bigger variants cost more per million tokens but still generate faster than a comparable GPU-hosted endpoint.

Exact rates change as Groq adds and retires models, so treat any dollar figure here as directional. For a full model-by-model rate card, see our dedicated Groq pricing guide and verify current numbers on Groq's own pricing page before committing spend.

Groq vs OpenAI and Anthropic: How Much Faster Is It?

Groq's LPU-hosted models generate tokens noticeably faster than GPU-hosted endpoints from OpenAI and Anthropic, often by a factor of several times on comparable model sizes.

The comparison is not apples-to-apples on model quality: Groq is racing open-weight models like Llama and Qwen against GPT and Claude's closed models, so a fast Groq response and a fast OpenAI response are not always answering with equal reasoning depth. Speed and capability are separate variables.

The gap matters most for time-to-first-token and total generation time on long outputs - a chatbot streaming a long answer, or a pipeline generating thousands of tokens per call, feels the difference. A single short completion where the model itself takes a second to "think" before answering will not show Groq's advantage as clearly, because reasoning latency and token-generation latency are different bottlenecks.

• Streaming a long response: Groq's raw tokens-per-second advantage is most visible here
• Short factual lookups: the gap narrows because network round-trip and model "thinking" time dominate over generation speed
• Reasoning-heavy tasks: a slower, more capable closed model can still finish faster in wall-clock time if it needs fewer follow-up turns to get a correct answer

When Should You Route Traffic to Groq?

Route a workload to Groq when the task needs an open-weight model's capability level and speed is the deciding constraint - voice agents, live chat, and real-time agent loops are the clearest fits.

Groq is the correct choice for voice AI pipelines, where every added millisecond of model latency shows up as an awkward pause in conversation. It also fits high-volume batch jobs where the model quality bar is met by an open-weight model and the API bill scales with speed.

Groq is the wrong choice when the task needs frontier reasoning that only a closed model reliably delivers - complex code generation, nuanced writing, or multi-step agentic planning where Claude or GPT-class models still outperform current open-weight options. Routing purely on speed and ignoring the capability gap is a common failure mode: teams see the tokens-per-second number, switch a reasoning-heavy workflow to Groq, and then spend more on retries and follow-up prompts than they saved on inference cost.

• Good fit: voice agents, live chat, high-volume classification, real-time agent tool loops
• Good fit: any workload already validated on an open-weight model where cost and latency matter more than marginal quality gains
• Poor fit: complex multi-step coding or planning tasks where a closed frontier model's accuracy reduces total retries
• Poor fit: workloads that require a specific closed model's behavior (e.g., a fine-tuned Claude or GPT deployment)

Groq vs Together AI vs Fireworks AI: How Do the Fast-Inference APIs Differ?

Groq, Together AI, and Fireworks AI all serve open-weight models over an API, but they differ on hardware and specialization.

Groq is the speed specialist: its custom LPU hardware exists for one purpose, and that shows up in raw tokens-per-second on supported models. Together AI and Fireworks AI run on GPU infrastructure but compete on breadth - larger model catalogs, fine-tuning support, and more flexible deployment options like dedicated instances.

A common pattern is using an aggregator like OpenRouter in front of all three, routing each request to whichever backend has the best combination of price, speed, and uptime for that specific model at that moment.

• Groq: fastest raw generation speed via LPU hardware, narrower model catalog
• Together AI: broader model catalog, fine-tuning and dedicated deployments
• Fireworks AI: similar breadth play, strong on custom model hosting