Inkling-Small: High-Performance AI in a Compact Format

What happened: the birth of Inkling-Small

Two weeks after launching its first open-source language model, Thinking Machines, the startup led by former OpenAI CTO Mira Murati, has introduced Inkling-Small, a multimodal reasoning model with 276 billion total parameters and 12 billion active parameters per token. This new model, available under the permissive Apache 2.0 license, accepts text, image, and audio inputs, and generates text, with a context window of up to one million tokens.

Most strikingly, according to the Artificial Analysis Intelligence Index, Inkling-Small scores 40, just one point below its predecessor's score of 41, Inkling, which has 975 billion total parameters and 41 billion active parameters. In other words, nearly the same performance with a quarter of the size.

Why it matters: efficiency without sacrificing performance

This release is significant because it addresses one of the biggest challenges in generative AI: the high cost of deployment and operation. By reducing computational and memory requirements, Inkling-Small allows companies to run high-performance models with less infrastructure, translating into lower inference costs and greater viability for organizations with limited resources.

Furthermore, Thinking Machines has published the full weights on Hugging Face and integrated support for fine-tuning through its Tinker API. This openness allows developers to customize the model to their needs, fostering innovation and adoption in the community.

Performance: nearly on par, sometimes superior

In several evaluations, Inkling-Small outperforms its larger sibling. For example, on SWE-bench Verified, it scores 80.2% versus Inkling's 77.6%, and on Terminal Bench 2.1, 64.7% versus 63.8%. It also improves on SciCode, Humanity's Last Exam, GPQA Diamond, and CritPt. However, not everything is advantageous: Inkling retains a clear superiority in factual knowledge and some agentic tasks, as shown by its 23.7% score on τ³-Banking versus Inkling-Small's 15.5%, and its omniscience index (AA Omniscience) is negative, indicating weaker factual coverage.

This balance between performance and efficiency is crucial for businesses. As VentureBeat notes, "the appeal is not simply that Inkling-Small is smaller, but that developers give up relatively little capability while reducing compute requirements, inference costs, and deployment footprint".

Pricing and accessibility

To incentivize adoption, Thinking Machines is offering a 50% discount for a limited time on API prices. The standard 64K context model costs $0.58 per million prefill (input) tokens, $1.44 per million sampling (output) tokens, and $1.73 per million training tokens, with cached prefill tokens at $0.116 per million. There is also a 256K context variant at higher prices.

These competitive prices, combined with the open-source nature, could democratize access to advanced AI, allowing startups and SMBs to experiment and deploy solutions that were previously only within reach of large corporations.

Consequences: market and industry impact

The release of Inkling-Small could intensify competition in the open-source AI model market, where giants like Meta (Llama), Mistral, and Alibaba (Qwen) already compete. Thinking Machines' strategy of offering highly efficient models could pressure other players to optimize their own models to reduce costs, benefiting end users.

For businesses, the ability to deploy a 276-billion-parameter model on their own hardware with fewer GPUs is attractive. Although it is still too large for a conventional laptop, it is much more manageable than the 975-billion flagship model, making it a viable option for organizations with some compute capacity.

However, limitations must be considered. The weaker factual coverage can be an issue for applications requiring accuracy in general knowledge, such as virtual assistants or information retrieval systems. Companies will need to carefully evaluate whether the balance between performance and cost fits their specific use cases.

What readers should know

• Nearly equivalent performance: Inkling-Small is just one point behind its predecessor on the Artificial Analysis intelligence index, and in some tests even surpasses it.
• Significant efficiency: With 276 billion total parameters and 12 billion active, it reduces inference and deployment costs compared to the flagship model.
• Open source and customizable: Available under Apache 2.0 with full weights on Hugging Face and support for fine-tuning via the Tinker API.
• Promotional pricing: 50% discount on API rates for a limited time, with prices ranging from $0.58 to $1.73 per million tokens depending on type.
• Factual limitations: Its general knowledge coverage is weaker than Inkling's, which may affect applications requiring precise factual information.

"Efficiency is the new battlefield in AI. Inkling-Small demonstrates that high performance is achievable without massive infrastructure, which could accelerate enterprise adoption and lower the barrier to entry."

Historical context and comparisons

This move by Thinking Machines comes at a time when the AI industry is seeking more efficient models. We have already seen examples like Inkling, which focused on low cost and resistance to censorship, and now with Inkling-Small, the trend toward optimization is reinforced.

Compared to other open-source models, such as Meta's Llama 3.1, which has 405 billion parameters, Inkling-Small offers superior performance on some reasoning tasks but with a smaller footprint. This could change expectations about what is needed to achieve state-of-the-art results.

In summary, Inkling-Small is a step forward in democratizing AI, offering an attractive balance between performance and cost. Companies looking to implement generative AI solutions should seriously consider this option, especially if they have reasoning and multimodality needs without requiring exhaustive factual knowledge.