Huawei MACaQ Is an Open-Source Model Quantizer

MACaQ is an open-source toolkit from the huawei-csl organization that compresses text, image, video and audio models so they run in less memory. MACaQ is short for Model Agnostic Calibrated Quantization.

The MACaQ repository lists the Apache License 2.0 and no price, while bundled third-party components retain their own licenses. MACaQ provides code to quantize artificial intelligence (AI) models across multiple modalities rather than focusing exclusively on language processing.

As of 12 November 2026, the README outlines MACaQ's software requirements, target inference runtimes and claimed performance for teams that operate models locally.

What Huawei MACaQ Is

Huawei MACaQ is an open-source toolkit that quantizes multimodal AI models for local execution. In AI engineering, quantizing a model compresses its numbers into fewer bits so the model needs less memory to run.

The repository states that MACaQ produces high-quality GGUF files (the model file format llama.cpp runs) for local inference and supports model-agnostic quantization beyond text models. MACaQ quantizes text, image, video and audio models, and you can then run them across several independent runtimes.

MACaQ is published under the Apache License 2.0 with no purchase price listed, though bundled third-party components retain their respective licenses. The paper is on arXiv as 2611.10649.

MACaQ targets four runtime environments: llama.cpp, vLLM, Diffusers, and PyTorch. With these four systems, MACaQ lets engineers quantize text models to GGUF, save them to disk and generate outputs with an included runtime.

As its headline claim, the repository reports that MACaQ can quantize a 122B-parameter model in 64 minutes using 32.5 gigabytes (GB) of device memory. The README also claims lower perplexity than MR-GPTQ and DuQuant++, which are two other quantization methods it names. It says users can match their best Llama-3-8B perplexity with 11 times less quantization time.

MACaQ fits an engineering team that already manages open-weights models on internal infrastructure and needs to reduce memory consumption. It does not fit an organization that relies exclusively on managed commercial application programming interface (API) endpoints.

What You Need to Run MACaQ

Running MACaQ requires Python 3.12 and an isolated software environment for each chosen backend.

The README instructs users to create a separate environment for each backend.

Beyond software dependencies, the front page lists no other hardware or software requirement. The README's only hardware figure is the 32.5 GB of device memory in its 122B-parameter example.

For execution, the documentation notes that the llama.cpp backend supports Central Processing Unit (CPU) execution.

The vLLM README says its backend needs Linux and Python 3.12. Installing it registers the MACaQ plugin in worker processes. It supports text models such as Qwen3.8-27B, including dense Qwen3 and Qwen3.5-family models, and text mixture-of-experts (MoE) models. W4A4 (4-bit weights and 4-bit activations) is the default. Calibration needs representative UTF-8 text. The saved checkpoint holds packed weights, scales, transforms, the tokenizer and source provenance, and loading or serving it does not repeat quantization.

The llama.cpp README says quantizing needs disk space for an intermediate full-precision GGUF, and CPU quantization takes longer. Raising --vq-sweeps from its default of 2 to 4 or 8 can improve quality. More passes take longer, and the file size stays the same. For the Flash-Next example, the vLLM README reports about 360 GB of disk space for the original source and about 178 GB for the serving checkpoint. Its large embedding table stays on the CPU and needs about 102 GB before framework and loading overhead.

Each supported backend folder contains its own dedicated guide with distinct installation, quantization, and generation steps:

• diffusers/README.md covers image, video, and audio generation pipelines.
• llamacpp/README.md handles text generation and GGUF quantization workflows.
• vllm/README.md manages text generation and serving setups.
• pytorch/README.md details speech generation scripts.

MACaQ Backends

The MACaQ README lists four backends, and it points to Ascend kernels in two pull requests. The README says W4A4 (4-bit weights and 4-bit activations) is the default inference mode for vLLM.

In a collapsed section titled Qwen3.8-27B quality and speed, the README provides comparative figures for the vLLM backend using original BF16 as a 1× baseline. The README reports that original BF16 (bfloat16) had a perplexity of 2.8463 with 1.00× prefill and 1.00× decode speeds. MACaQ W4A4 had a perplexity of 2.8994 with 2.33× prefill speed and 2.58× decode speed. The published table does not list the underlying hardware used to collect these measurements.

The vLLM README says the backend can serve an HTTP (Hypertext Transfer Protocol) API on 127.0.0.1 port 8000 at /v1/chat/completions. Its current command-line tool processes one sequence at a time. In the MoE adapter, routed expert projections use W4A4, and attention, shared experts, routers, norms, embeddings and the output head keep their original precision. The adapter supports W4A4 only. Weight-only mode, chosen with --precision w4a16, is currently slower than W4A4. Checkpoints from other quantization implementations are not interchangeable with MACaQ checkpoints.

The PyTorch README says the Qwen3-TTS and Chatterbox Turbo adapters both quantize to W4A4, save a checkpoint and generate WAV files. Saved checkpoints load without the original weights or a second calibration pass. Chatterbox Turbo generates English speech and retains its watermark. The default runtime processes one utterance at a time. The Diffusers README says MACaQ quantizes transformer projections to W4A4 and leaves text encoders, the variational autoencoder (VAE) and untargeted layers at their source precision. It supports only single-transformer Wan pipelines. It also says the Stable Audio checkpoint is a community checkpoint whose upstream card does not identify the original model or state a weight license.

MACaQ-R for Existing GGUF Files

MACaQ-R is a specialized component within the repository designed to refine the weight codes of existing GGUF files without breaking compatibility. The README explains that MACaQ-R preserves native GGUF formats and block scales, which ensures that refined model files continue to run directly inside standard llama.cpp installations.

The repository states that MACaQ-R was released to improve the output quality of existing quantized models and adapt them to a workload.

To quantify this quality improvement, the README compares Kullback-Leibler (KL) divergence against 16-bit floating-point (FP16) baseline outputs. KL divergence measures how far a quantized model's output probability distribution drifts from the original model, where a lower number indicates closer alignment to original output behavior.

For the Qwen3.6-35B-A3B model at a file size of 22.36 GB, the README reports a KL divergence of 0.015731 for an Unsloth UD-Q4_K_XL file. A MACaQ-R Q4_K_XL file at the same 22.36 GB size reports a lower KL divergence of 0.013367.

The llama.cpp README says MACaQ-R refines an existing native K-quant mixture-of-experts GGUF. It updates Q4_K, Q5_K and Q6_K weight codes and copies other tensors unchanged. Block scales, tensor formats and the original activation basis stay as they were. The defaults are 64 refinement iterations and 131,072 calibration tokens, and --resume continues an interrupted run.

The repository outlines the command syntax used to execute this refinement step: llamacpp/macaq.py refine --model models/model-q4.gguf --hf models/original --teacher models/original-f16.gguf --calibration data/calibration.txt --output models/model-macaq-r.gguf --work-dir work/refinement.

Engineers who want to examine a pre-built refined artifact can download the official MACaQ-R Qwen3.6-35B-A3B GGUF hosted on Hugging Face.

Who Should Skip MACaQ

MACaQ is not a general-purpose product for every business exploring artificial intelligence. Three kinds of team should skip it:

• Teams that rely entirely on hosted third-party model APIs have nothing to compress. They should keep their existing API contracts, or read our guide on running quantized models with Ollama for an approachable starting point for internal testing.
• Companies without software engineers who can run a Python quantization pipeline will struggle with this project. MACaQ is a GitHub code repository. It is not a packaged commercial desktop application or a turnkey service.
• Teams that need a packaged app rather than a repository should read how to run Llama locally instead.

Our Takeaway on Huawei MACaQ

On our reading of the MACaQ README as of 12 November 2026, MACaQ suits an engineering team that already runs open-weights models on internal infrastructure. That team also needs developers who can operate a Python quantization pipeline. The repository lists the Apache License 2.0 and no purchase price. It is a code repository and not a packaged app, and it offers no benefit to businesses that call only hosted APIs.

Our perspective would change if independent testing challenges the repository's stated perplexity and speed claims, or if MACaQ's maintainers package it into an automated utility.

To evaluate the pipeline on your own systems, open the MACaQ repository and follow the llama.cpp instructions for Qwen3-4B to verify what Huawei MACaQ outputs on a smaller model.