The True Cost of AI: Completion Rate vs. Tokens

Databricks' recent internal study has called into question the widely used cost-per-token metric for comparing language models (LLMs). The company evaluated several models and code assistants using real software engineering tasks, finding that the price per completed task is a much more reliable indicator. For example, Z.ai's open model GLM 5.2 matched Anthropic Opus 4.8 in quality, but at a cost of $1.28 per task versus $1.94 for Opus. In contrast, Anthropic's Sonnet 5, cheaper per token than Opus, turned out to be more expensive per task ($2.09) due to a lower completion rate (81% vs. 87%) and higher token consumption.

This finding is not isolated. An academic study from March 2026 already pointed out that in one-third of comparisons, the model with the lowest listed price ended up costing more. For instance, Gemini 3 Flash, 80% cheaper than GPT-5.4 in price per token, turned out to be 38% more expensive in real cost per task.

Historical Context and the Dominant Metric

Since the explosion of LLMs in 2022-2023, the cost per token has become the standard metric for comparing models. Companies like OpenAI, Anthropic, and Google publish prices per input and output token, and analysts use them to calculate operational costs. However, this metric ignores the real efficiency in task completion. The Databricks study, led by CTO Matei Zaharia, demonstrates that completion rates and token consumption vary greatly between models. For example, Opus 4.8 completed 87% of tasks, while Sonnet 5 only 81%, and consumed more tokens. This is due to differences in model architecture, training, and the ability to follow complex instructions.

The problem is not new. In 2024, academic studies already warned that benchmarks like SWE-Bench were "broken" (according to OpenAI), as models were tuned to them but did not reflect performance on real tasks. Databricks created its own benchmark based on real tasks from its engineers, providing a more accurate view of performance.

Impact on Companies and Users

For companies integrating AI into their processes, choosing the cheapest model per token can lead to higher operational costs and lower productivity. For example, if a company uses Sonnet 5 for coding tasks, it will pay $2.09 per task instead of $1.94 with Opus 4.8, even though Sonnet is cheaper per token. At scale, this can mean thousands of additional dollars per month. Additionally, the lower completion rate means developers must spend time correcting or completing tasks, reducing productivity.

The study also highlights the role of the 'harness' or intermediary software. The harness can inflate the context (prompt + history) by up to 3 times, multiplying the cost without improving the success rate. For example, with Claude Code as the harness, Opus 4.8 consumed 742,000 tokens per task, while with the Pi harness (minimalist) only 236,999 tokens, achieving the same success rate. This means companies can significantly reduce costs by choosing an optimized harness. Databricks launched Omnigent, a harness that promises to reduce costs without sacrificing performance.

Comparison with Previous Events

This paradigm shift recalls the transition in cloud storage costs. Initially, companies compared prices per gigabyte, but later discovered that transfer and operations costs (IOPS) were more important. Similarly, the cost per token is only part of the equation. Another parallel is with database engines: the price per query was the key metric, but later it was understood that performance per transaction was more relevant.

In the AI field, there are already precedents. In 2024, startup Together AI promoted cost per task for open-source models, but failed to establish it as a standard. Now, with the backing of Databricks and academic studies, it is likely that the industry will adopt more realistic metrics.

Implications for the Market

This analysis forces a rethink of AI adoption strategies. Companies will need to conduct internal evaluations with their own tasks to determine the real cost per completed task, rather than relying on token price tables. It also opens an opportunity for startups developing optimized harnesses, like Pi or Databricks' new Omnigent, which promise to reduce costs without sacrificing performance. Model providers, for their part, will need to be transparent not only about price per token but also about efficiency metrics per task. In the long term, an industry standard based on 'cost per completion' could emerge.

Furthermore, the study shows that open models like GLM 5.2 are competitive with frontier models. This could accelerate the adoption of open-weight models, especially in companies looking to reduce costs and avoid dependence on a single provider. However, the quality of the harness remains critical: an excellent model with an inefficient harness can be more expensive than a worse model with a lightweight harness.

What Should Readers Know?

• Don't rely solely on price per token: A model cheaper per token can be up to 38% more expensive per task if it has a low completion rate or consumes many tokens.
• The harness matters: The software that orchestrates interaction with the LLM can double or triple the cost. Opt for lightweight, minimalist harnesses.
• Evaluate with your data: Generic benchmarks like SWE-Bench may be 'broken,' according to OpenAI. Conduct tests with real tasks from your domain.
• Open models compete: Z.ai's GLM 5.2 matched Opus 4.8 in quality with a 34% lower cost per task. Don't rule out open-weight options.

"Cheaper per-token does not imply cheaper per-task" — Matei Zaharia, CTO of Databricks.