MoE Inference Engine Colibri Runs Frontier Models Locally
Tiny engine, immense model: Colibri runs huge MoE models on ordinary hardware.
Why MoE models are a natural fit for tiered storage
Cold experts: the main trade-off
Who should try colibri
Colibri is a hierarchical inference engine built specifically for mixture-of-experts (MoE) models. Instead of trying to load a whole 200B-parameter model into VRAM, it keeps only the most active experts on the GPU (via CUDA or Vulkan), stores frequently used ones in system RAM, and streams rarer experts straight from an NVMe SSD. Because an MoE model activates a fraction of its experts per token, the engine can serve models that would otherwise require datacenter-class GPUs.
## Why MoE models are a natural fit for tiered storage
The trick works because frontier MoE models are sparsely activated. A large language model may have dozens or hundreds of experts, but each token routes to only a handful. Colibri exploits this by maintaining a hot/warm/cold memory hierarchy: hot experts live on the GPU where compute happens, warm ones stay in RAM for quick fallback, and cold ones remain on NVMe. Writing the engine in C allows tight control over memory layout and transfer behavior, while supporting both CUDA and Vulkan broadens the range of GPUs that can participate.
## Cold experts: the main trade-off
The obvious risk is latency. If the router suddenly calls an expert that has been swapped out, the engine must pull it from NVMe into RAM and then onto the GPU before the next token can finish. NVMe is fast, but it is not GPU memory, so the ordering and frequency of expert loads matter. That makes colibri a performance-sensitive, early-stage project rather than a zero-configuration tool. It will appeal most to people who are comfortable building C projects, configuring Vulkan or CUDA, and profiling their own bottlenecks.
## Who should try colibri
Colibri is aimed at self-hosters, researchers, and AI engineers who want to run frontier-scale MoE models on hardware they already own. If you have a decent GPU, enough system RAM, and an NVMe drive, it offers a path to private, low-cost inference without renting cloud instances. It is still an emerging project, so check the official website, releases, and community channels before committing it to production.
The right idea at the right time: colibri makes giant MoE models viable on modest local hardware.