Colibrì: Frontier AI Model Running on Just 25 GB of RAM

What Happened?

Italian engineer Vincenzo (known as JustVugg) has published a project on GitHub called Colibrì that allows running the GLM-5.2 language model (744 billion parameters, 1.5 TB) on a system with only 25 GB of RAM and a virtual NVMe SSD at 1 GB/s. GLM-5.2 is a Mixture-of-Experts (MoE) model that rivals GPT-4 and Claude in performance. Colibrì loads the model in slices from disk to RAM as needed, leveraging the fact that in MoE models only a subset of experts is activated per token. The result is an inference rate of 0.05 to 0.1 tokens per second, far from the 20-30 needed for interactive use, but enough to prove that frontier models can run on consumer hardware.

The technique, known as memory swapping or intelligent paging, is not new in AI: projects like llama.cpp and Ollama already use similar strategies for smaller models. However, Colibrì takes this idea to the extreme by handling a 1.5 TB model, which would normally require multiple GPUs with HBM memory (like NVIDIA H100 with 80 GB each) or even an entire cluster. According to Tom's Hardware, Vincenzo achieved this with a modest CPU and a virtualized NVMe SSD at 1 GB/s, highlighting that the innovation lies in software, not hardware.

The historical context is relevant: in 2023, running a 70B parameter model on a single consumer GPU (RTX 4090) was an achievement. Now, Colibrì shows that even 744B models can run without a GPU, albeit at extremely low speeds. This recalls the early days of personal computing, when programs were loaded from magnetic tapes; speed was minimal, but it demonstrated feasibility.

Why Is This Important?

Until now, running state-of-the-art AI models required multiple high-end GPUs (like H100) or expensive clusters. Colibrì demonstrates that with efficient software engineering, access to frontier models can be democratized for enthusiasts, researchers, and small businesses that prioritize data privacy. It also highlights the potential of MoE models to reduce active memory footprint, as only relevant experts need to be loaded at each step.

The market impact could be significant. Subscriptions to AI services like ChatGPT Plus or Claude Pro cost around $20 per month, but involve sending data to the cloud. For sectors like healthcare, finance, or defense, where privacy is critical, local execution is a necessity. Colibrì, though slow, offers a way for these sectors to evaluate frontier models without exposing sensitive data. Additionally, it could pressure hardware manufacturers (like Samsung and SK Hynix) to optimize bandwidth between storage and memory, as noted by Tom's Hardware in its analysis of AI-driven memory shortages.

Compared to other democratization efforts, such as the TinyML project or model quantizations (e.g., GPTQ or GGUF), Colibrì focuses on full model execution without compression, preserving original quality. However, the current speed is 200-300 times slower than needed for real-time interaction, limiting practical use to batch tasks like document processing or overnight report generation.

Consequences and Outlook

In the short term, Colibrì is a proof of concept with speed limitations. However, it points to a future where high-level local AI is viable with affordable hardware. This could drive applications in sensitive sectors (healthcare, finance, defense) where data cannot be sent to the cloud. It could also incentivize hardware manufacturers to optimize storage-to-memory bandwidth, and model developers to design even more efficient architectures.

An interesting perspective is the potential evolution toward faster memories, such as CXL (Compute Express Link), which would allow low-latency memory sharing between CPU and GPU. If Colibrì adapts to these technologies, speed could improve dramatically. Additionally, the swapping technique could be combined with quantization (e.g., FP8 or INT4) to further reduce model size on disk, though this would degrade quality.

In market terms, projects like Colibrì could accelerate the adoption of specialized hardware for local AI, such as NPUs (Neural Processing Units) integrated into modern CPUs (AMD Ryzen AI, Intel Meteor Lake). If these NPUs can handle swapping efficiently, we could see an ecosystem of local AI applications competing with cloud services. However, the current speed of 0.05 tokens/second is too low for interactive applications; for reference, a human reads about 5-6 tokens per second, making Colibrì 100 times slower than human reading.

What Should Readers Know?

• Current speed: 0.05-0.1 tokens/second, unusable for real-time chat, but viable for batch processing or tasks that don't require immediate response.
• Required hardware: 25 GB of RAM, a modern CPU, and a fast NVMe SSD (1 GB/s). No GPU needed.
• Model used: GLM-5.2, a 744B parameter MoE, comparable to GPT-4 in quality.
• Future: If storage bandwidth is optimized or faster memories (like CXL) are used, speed could improve significantly.
• Limitations: Inference is extremely slow; not suitable for real-time applications. Additionally, the GLM-5.2 model is not officially available for public download, limiting reproducibility of the experiment.

Colibrì demonstrates that the barrier to entry for frontier AI is not just hardware, but also software engineering. It is a step toward data sovereignty and local computing.

In summary, Colibrì is a technical milestone showing the potential of software engineering to overcome hardware limitations. Although its practical application is limited today, it lays the groundwork for a future where high-performance AI is accessible locally, with profound implications for privacy, data sovereignty, and democratization of artificial intelligence.