How to Run Kolibri

You can run Kolibri on your own hardware by downloading the open weights and serving them with vLLM. At Layer3Labs, we build and operate AI systems inside other people's businesses. Kolibri is an open-weight Mixture-of-Experts model from Aleph Alpha. It gives organizations an on-premise alternative for private text processing.

The model features 78.1B total parameters and activates 3.46B parameters per token. Aleph Alpha released these weights under the Apache 2.0 license on 3 October 2026. This structure allows teams to run Kolibri on premise without sending confidential records outside. You retain full control over your computational infrastructure.

Serving the architecture requires an inference plugin provided by Aleph Alpha. Hardware sizing depends directly on your context length and concurrency targets. Test the model thoroughly before deploying it into production.

How to Run Kolibri on Your Own Hardware

Running Kolibri requires downloading the weights from Hugging Face and serving them using the Aleph Alpha vLLM plugin. Aleph Alpha distributes the weights directly on Hugging Face. The Apache 2.0 license is a permissive open-source license. You can read the complete license text on the Kolibri Hugging Face repository.

Aleph Alpha provides the aleph-alpha-inference package, which supplies the Kolibri vLLM plugin. You can install it with pip or pull the container image.

• 1. Obtain weights: Download the model files from the official repository on Hugging Face.
• 2. Prepare environment: Install the plugin with pip install "aleph-alpha-inference>=1.0" or pull ghcr.io/aleph-alpha/aleph-alpha-inference.
• 3. Launch inference: Run vllm serve Aleph-Alpha/Kolibri-1 --kv-cache-dtype fp8 --reasoning-parser kolibri1 --tool-call-parser kolibri1 --enable-auto-tool-choice.
• 4. Set sampling values: Configure your inference requests using temperature=1.0, top_p=0.97, and top_k=128.
• 5. Extend context length: For requests beyond 262,144 tokens, add --max-model-len 1048576 and --hf-overrides '{"max_position_embeddings": 1048576}'.

Sizing GPUs for Kolibri

Hardware sizing depends on the 78.1B total parameters plus the key-value cache for your concurrency and context length. Aleph Alpha publishes no minimum VRAM requirement in the launch post. Engineering teams must test their own workloads or contact Aleph Alpha directly. The model activates 3.46B parameters per token across 6 active experts out of 384.

Aleph Alpha's launch post compares Kolibri against a 123B model. That test ran on two NVIDIA H100 GPUs at a 256k context window. The 123B model managed only 3 concurrent queries. In contrast, Kolibri handled 18 concurrent queries and decoded 28 percent faster.

The official serving command includes the --kv-cache-dtype fp8 flag. This flag reduces memory consumption in the key-value cache during active processing. Large context prompts will still expand GPU memory usage rapidly. You should size your memory pool to accommodate peak concurrent tokens.

• Active parameter load: Activating 3.46B parameters per token lowers computational overhead during inference.
• Hardware benchmark: Aleph Alpha reports 18 concurrent 256k queries on two NVIDIA H100 GPUs.
• Decoding rate: Aleph Alpha reports a 28 percent decoding speed advantage over the 123B comparison model.
• Cache management: The fp8 flag helps manage key-value cache memory on supported accelerators.

Reasoning Effort Levels and Selection

Kolibri provides four reasoning effort levels designed to balance generation speed against output accuracy. These four levels are none, low, medium, and high. Aleph Alpha explains that higher reasoning effort trades cost and latency for answer quality. The request parameter that sets the level is not in the launch material we reviewed.

Engineers should check the Hugging Face model card for current request parameters. We suggest starting your evaluation at the lowest reasoning effort level. Verify your task accuracy with reasoning set to none or low. Step up to higher levels only if difficult questions fail.

• Reasoning settings: Select between none, low, medium, and high reasoning effort.
• Latency impact: Higher effort levels trade cost and latency for answer quality, per Aleph Alpha.
• Request parameters: Check the Hugging Face repository for current API parameters.
• Recommended baseline: Start testing with lower reasoning settings before allocating extra compute.

Long-Context Limits and Serving Flags

Kolibri supports context windows up to 1M tokens when launched with explicit runtime override parameters. The longest native training length for the model is 262,144 tokens. Operating past 256k tokens requires adding the --max-model-len 1048576 flag. You must also pass the --hf-overrides flag with max_position_embeddings set to 1048576.

Long inputs demand vast memory allocations for the key-value cache. Serving prompts at 1M tokens reduces the number of concurrent queries a server can process. Extensive context windows also increase generation latency. Test your retrieval accuracy across long documents before putting extended windows into production.

• Native context: The model underwent native long-context pre-training up to 262,144 tokens.
• Extended limits: Reach 1,048,576 tokens by supplying explicit CLI override flags.
• Hardware demands: Long context windows multiply memory requirements and lower total concurrency.
• Validation needed: Measure accuracy degradation on long prompts prior to operational rollout.

On-Premise Deployment Versus Hosted Options

Choosing an on-premise installation over a managed endpoint depends on privacy requirements and operational capacity. The model weights carry no cost under the Apache 2.0 license. Your primary expenses are enterprise GPUs, power, cooling, and staff. Operating dedicated servers requires specialized engineering time.

Aleph Alpha has not published hosted API pricing on its website or launch announcement. Organizations wanting managed access or specialized models must contact Aleph Alpha sales directly. Self-hosting keeps all document data inside your corporate firewall. Teams lacking hardware may prefer evaluating external APIs before purchasing servers.

• Zero weight cost: Apache 2.0 allows downloading and running the model without licensing fees.
• Operating overhead: Running private hardware requires ongoing investments in servers, power, and engineers.
• Unpublished pricing: Aleph Alpha lists no public API rates and directs enterprise inquiries to sales.
• Data security: Aleph Alpha says on-premise serving lets you avoid sending internal data to third-party inference services.

Production Testing and Validation Guidelines

Production testing for Kolibri should rely on internal validation benchmarks rather than self-reported scores. Aleph Alpha positions Kolibri specifically for public administration, industrials, and aerospace. These categories represent target markets rather than proven case studies. Start your testing by evaluating question answering across your internal German-language documents.

Aleph Alpha published several benchmark figures in its announcement materials. It reports 96.9 on AIME 2025, 84.3 on GPQA diamond, and 85.9 on LiveCodeBench v6. On the AA-Omniscience Index, Kolibri scored -32.8 points. That score trailed behind Mistral Small 4 at -24.0 and Qwen3.6-35B-A3B at -15.3, where higher values represent better performance. Aleph Alpha also publishes a tech report.

Kolibri includes a training protocol that Aleph Alpha calls Merlin-Arthur. This protocol trains the model on abstention data to prevent invented answers. The model should reply with 'I do not know' when documents lack the answer. Test this abstention behavior thoroughly using negative test cases.

• Positioning targets: Aleph Alpha identifies public administration, industrials, and aerospace as intended sectors.
• Self-reported scores: Aleph Alpha reports 96.9 on AIME 2025 and 84.3 on GPQA diamond.
• Omniscience benchmark: Kolibri scored -32.8, performing below Mistral Small 4 and Qwen3.6-35B-A3B.
• Hallucination controls: The Merlin-Arthur protocol trains the model to abstain when context is insufficient.

Evaluating Private Deployment for Your Team

Deciding to host Kolibri privately requires evaluating compliance obligations against the total cost of ownership. Procuring enterprise accelerators involves substantial capital outlays and facility planning. Organizations with strict regulatory mandates often require on-premise execution. Teams without strict isolation policies may find hosted APIs more cost effective.

Our free AI workflow audit helps organizations decide between self-hosting and hosted APIs. We evaluate your regulatory constraints, query volume, and document security requirements. This analysis clarifies whether on-premise clusters justify their operational expense.

What you need to run Kolibri yourself

Kolibri needs real memory, but it is within reach of a high-end workstation or a couple of professional GPUs — and many teams simply rent instead of buying. Match the path below to whether you want to own the hardware or pay by the hour.

To put Kolibri to work once it is live, connect a coding client like Cursor (via OpenRouter) or a local runner such as Ollama.

Apple Mac Studio (M4 Max, 128GB)

A single quiet on-prem box

View on Amazon →

NVIDIA RTX 6000 Ada (48GB)

Power users who want hardware they own

View on Amazon →