NPU vs GPU for AI Workloads on Every Form Factor

NPUs and GPUs handle different kinds of AI work. At Layer3Labs, we build and run AI systems inside our clients' businesses, and the hardware question comes up whenever a client wants to keep a model on their own machines.

An NPU, or neural processing unit, is a low-power chip designed for smaller AI features built into an operating system, things like background blur, live captions, and call noise removal. A GPU, or graphics processing unit, handles large language models (LLMs). To run, the model must fit in the GPU's own memory, known as VRAM (video RAM), or in the shared unified memory available on Apple silicon.

So NPU vs. GPU isn't really a contest. Both can sit in the same machine, with the operating system routing each task to the chip best suited for it. Microsoft explains this process in its Copilot+ PC developer guide. The guide also notes that Windows ML falls back to the GPU or CPU (central processing unit) when the preferred NPU execution provider isn't available.

That makes the buying decision simpler than the chip names might suggest: memory capacity determines whether a model can run at all. Our local AI hardware calculator can size those requirements for a specific model.

NPU vs. GPU: Side-by-Side

Suggest a correction — if you work at one of the products above and something here is out of date, tell us and we'll fix it.

NPU vs GPU in Plain Terms

An NPU is a small, power-efficient accelerator built into the same package as the CPU, and it exists so AI features can run constantly without draining a battery. Microsoft's support page on neural processing units states that an NPU uses less power and is more efficient at AI tasks than a CPU or GPU. It also frees those chips for other work.

A GPU is a much larger parallel processor with its own pool of fast memory. It was built for graphics, and the same math turns out to run neural networks well, which is why every serious local AI tool targets it. On Apple silicon there is no separate card, and the GPU shares one large unified memory pool with the CPU.

Vendors rate NPU speed in TOPS, or trillions of operations per second. Microsoft requires an NPU rated at 40 TOPS or more before a machine can carry the Copilot+ PC badge, which gates features such as Windows Studio Effects and live caption translation. A TOPS number says nothing about whether a machine can hold a 30-billion-parameter model, so it is the wrong figure to shop on for local LLM work.

• NPU: fixed-function, low power, always available, tuned for small models that ship with the operating system
• GPU: large parallel compute plus dedicated memory, the target of nearly every open-source AI runtime
• CPU: runs the surrounding work, and runs models itself when nothing faster is available, slowly

Which Chip Runs Which AI Workload

Small, always-on AI features run on the NPU, and any job built on a model with billions of parameters runs on the GPU or on unified memory. The list below covers the AI jobs people expect from a new machine, with the chip that carries each one and the reason it lands there.

• Background noise suppression on calls: NPU. The model is tiny, it runs for the length of the meeting, and the NPU keeps the fan off and the battery alive.
• Live captions and on-device translation: NPU. Speech models are small and latency-sensitive, and Microsoft ships them as Copilot+ PC features built for that chip.
• Background blur and webcam framing: NPU. These run for hours, so power draw matters more than raw throughput.
• Photo cleanup and small on-device image edits: NPU on a phone or a thin laptop, GPU on a desktop. Either finishes the job, and the NPU just costs less battery.
• Local LLM chat with Ollama or LM Studio: GPU or unified memory. The model has to sit in memory to answer at a readable speed, and neither app targets an NPU.
• Retrieval and embeddings over a document set: GPU. The embedding pass is short but heavy, and it reruns every time the document set changes.
• Image and video generation: GPU, with more VRAM than a chat model of similar size needs. Video generation is the workload most likely to run out of memory mid-render.
• Fine-tuning a model on company data: GPU, and usually a rented cloud one rather than a laptop. Training holds the model plus gradients and optimizer state in memory at once.
• Full training from scratch: neither, in any consumer machine. That workload belongs on server accelerators.

NPU vs GPU by Form Factor

The NPU matters most in the machines with the least room for a GPU. A phone has no discrete graphics card and no thermal headroom. Apple, Samsung, Qualcomm, and Google therefore put on-device AI features on a dedicated neural block. Phone AI is a set of built-in features rather than an open model runtime. Nothing about a phone NPU means a handset can run a full local LLM.

In a thin laptop the NPU carries the always-on features while the integrated GPU handles anything larger, and total memory sets the ceiling on both. Machines built around a discrete NVIDIA card change that balance, since a laptop GPU with real VRAM can run models the NPU will never touch. The best AI laptops roundup covers which configurations reach which model sizes.

On the desktop side the NPU stops being interesting at all. A tower with a discrete card has far more GPU memory and throughput. Its NPU handles the same small camera and microphone features it would on a laptop, and nothing more. Workstations and towers are covered in the best AI workstations roundup, and the best computers for AI page routes between form factors.

• Phone: NPU only. Built-in assistant features work well, and open local model runtimes do not apply.
• Thin laptop: NPU for battery-friendly features, integrated GPU and total memory for anything bigger.
• Mini PC: memory capacity decides everything, as the best AI mini PCs roundup sets out in detail.
• Desktop and workstation: the discrete GPU does the AI work, and the NPU is close to irrelevant.

Memory Capacity Decides Which Models Run

A local model either fits in memory or it does not run well, and no amount of NPU performance changes that. The practical bands have held steady across chip generations, and they are listed below. Memory bandwidth then sets how fast tokens come back, once the model fits.

Unified memory changes the shape of the decision. Apple's own M5 announcement says a unified memory architecture lets a machine run larger AI models entirely on device. Mac configurations reach memory sizes that consumer graphics cards do not offer. NVIDIA cards answer faster per token and own the broader software ecosystem.

Neither approach is the automatic pick. Model size per dollar tends to favor unified memory. Ecosystem breadth and raw speed favor a discrete card. A high-end NVIDIA tower can also draw close to a kilowatt under load, and it is loud.

• 12 to 16GB: 7 to 13 billion parameter models
• 24 to 32GB: quantized 30 to 70 billion parameter models
• Above that: professional GPUs, high-memory unified machines, or DGX-class boxes
• Bandwidth sets token speed, and capacity sets whether the model loads at all

Training vs Inference

Inference means running a finished model to get an answer, and training means building or adjusting the model's weights in the first place. Almost everything a business does locally is inference: a chat assistant, a summarizer, a document search, a transcription pass. Training and fine-tuning are far rarer, and far heavier.

An NPU handles inference only, and only for models converted to run on it. A GPU handles both, which is why a fine-tune that would be impossible on an NPU is merely slow on a mid-range card and comfortable on a rented cloud one.

The memory gap between the two jobs surprises most buyers. Fine-tuning holds the model weights, the gradients, and the optimizer state in memory at once. It needs several times the memory that inference on the same model takes. A machine sized for local chat is rarely sized for local fine-tuning.

• Inference: running a trained model. Fits the NPU for small built-in features, the GPU for anything larger.
• Fine-tuning: adjusting an existing model on company data. GPU only, and memory-hungry beyond the inference numbers.
• Training from scratch: server accelerators, rented by the hour. No consumer machine is the right tool.

Why a Business Buys the Chip at All

The NPU and the GPU both answer one business question from opposite ends: which work has to stay inside the building. A local model on a GPU keeps prompts, client files, and draft output on the machine. Teams accept the hardware bill for that reason rather than sending the same work to a hosted API. The NPU covers the smaller version of that promise, keeping call audio and captions on the laptop instead of streaming them to a meeting service.

A compliance or legal reviewer can act on that boundary. A statement that a named workload runs on hardware the company owns, with no outbound request, is a far shorter approval conversation than a description of somebody else's data-handling terms. In the implementations we run for clients, the workloads that clear review first are the ones where nothing leaves the device at all.

The cost side runs on how often the model is used. Light or occasional use never reaches break-even against a machine with enough memory to hold the model, while steady daily use does, and so does any workload the data rules forbid sending out. The best AI laptops page works through where that break-even sits.

• Best AI laptops works through the buy-against-rent arithmetic in full.
• Pay for the NPU when the always-on features are the point and the fleet lives on battery
• Count the seats before the memory tier, because a team rarely needs a large-memory machine per person, as the best computers for AI page sets out

Why Local LLM Apps Leave the NPU Idle

The most common local AI tools do not use the NPU at all, and their own documentation says so. Ollama's hardware support page lists NVIDIA GPUs, AMD GPUs through ROCm, Apple Metal, and Vulkan, with no NPU among them. LM Studio's system requirements ask for Apple silicon or a Windows machine with dedicated VRAM, and mention no NPU either.

The reason is a software gap rather than a marketing one. Microsoft's developer guide notes that many NPU devices support only lower-bit integer math such as INT8. A model must be converted into an NPU-ready format first, through Windows ML and a vendor execution provider. The general-purpose model files people download for local chat are not in that format.

That produces a failure mode that is invisible from the outside. A buyer picks a laptop for its NPU rating, installs a local model runner, and the NPU sits at zero percent. The integrated GPU and the CPU do the work instead, slowly. Ollama and LM Studio never ask for the NPU in the first place, so nothing warns the buyer, and the only signal is the NPU column in Task Manager staying flat. Checking that column during a trial run shows which chip a machine is using.

• Ollama documents NVIDIA, AMD, Metal, and Vulkan acceleration. No NPU.
• LM Studio documents Apple silicon and dedicated VRAM. No NPU.
• NPU execution needs a converted, quantized model plus a vendor runtime, which downloaded general-purpose model files are not.
• Windows ML falls back to GPU or CPU without warning, and runtimes that skip Windows ML never touch the NPU at all, so an idle NPU looks like a slow machine rather than a routing problem.

How to Check What Is in a Machine before Buying

Three numbers settle whether a machine suits the intended AI work, and none of them is the marketing badge on the lid. Read the memory figure first, since it sets the model ceiling. Then confirm the graphics part, and only then look at the NPU rating.

On Windows, Intel documents how to check whether a processor includes an NPU through its product specification pages, where the feature is listed as Intel AI Boost. AMD lists the same information under its Ryzen AI processor specifications. Task Manager shows an NPU tab on machines that have one, which is the quickest live confirmation.

On a Mac there is no separate figure to hunt for, because the unified memory number covers it. Apple publishes the memory configurations for each machine on its Mac Studio tech specs and equivalent support pages, and that number is the model ceiling. The what is an AI PC explainer covers what the badge itself does and does not promise.

• Memory capacity: VRAM on a discrete card, or unified memory on Apple silicon. This decides which models load.
• Graphics part: discrete card, integrated graphics, or Apple silicon GPU. This decides how fast answers arrive.
• NPU rating: only matters for built-in operating system features and battery life on those features.
• Upgrade path: a tower takes a bigger card later. A phone, a thin laptop, and most mini PCs do not.

Who Should Not Buy for the NPU

Anyone whose plan is to run a local LLM should not pay extra for a higher NPU rating. That buyer wants memory capacity, and the money is better spent on a machine with more VRAM or more unified memory, even if its NPU rating is lower or absent. The best computers for AI page sorts machines by that constraint rather than by badge.

A team whose AI work happens in a browser should also skip the premium. Cloud assistants run in a data center, so a laptop chosen for them needs a good screen, a good keyboard, and battery life, none of which the NPU rating predicts.

The NPU is worth paying for in one clear case. That case is a fleet of thin laptops running meeting effects, captions, and on-device search all day, where battery life is the complaint IT hears most. The chip was designed for exactly that work. It runs without touching a discrete card.

• Skip the NPU premium: local LLM users, anyone fine-tuning, anyone generating images or video at length.
• Pay for the NPU: laptop fleets that live in video calls and want longer battery on those features.
• Neither applies: browser-based cloud AI users, who should buy on screen, keyboard, and battery instead.

What Would Change This Verdict

This routing holds because of a software gap, and a software change would close it. If Ollama, LM Studio, or llama.cpp shipped native NPU execution across Intel, AMD, and Qualcomm parts, a mid-range laptop could offload real local models to a chip that sips power. The NPU would stop being only a battery feature. Each project documents its supported accelerators, so a change would show up in Ollama's hardware support page before it shows up in any review.

A hardware change would do it too. NPUs today borrow a slice of system RAM and are tuned for small converted models. Give one a large dedicated memory pool, plus support for the floating-point formats local models ship in, and the split changes outright.

Until one of those lands, the split stays where it is. Read the Ollama and LM Studio hardware pages rather than the badge on the lid, because these capabilities change quietly between chip generations.

The Verdict

There is no winner between NPU and GPU, because they are aimed at different work. The NPU runs the small AI features an operating system ships with, at a power level that keeps a laptop alive all day. The GPU, or a large unified memory pool, runs the models a person downloads and points at their own data.

Buy for the NPU when the AI work is meeting effects, captions, transcription, and on-device search across a fleet of portable machines. The 40 TOPS Copilot+ PC threshold gates exactly those features, and they drain a battery fast when the CPU has to run them instead.

Buy for the GPU, or for a high-memory Mac, when the plan is local model work of any real size. Memory capacity decides it, and every band above follows from capacity rather than from a chip rating.

The inversion case is simple. A laptop with a strong NPU and 16GB of shared memory loses to an older machine with a 24GB graphics card at every local LLM task. It wins comfortably on battery life during a day of video calls.

Anyone weighing NPU vs GPU should size the model they intend to run with the local AI hardware calculator, then shortlist only the machines whose memory holds it.