Running LLMs Locally: The Real Cost per Token on a GPU

What Happened?

A recent article on Towards Data Science quantified the real electricity consumption of eight large language models (LLMs) run locally on an NVIDIA RTX 3090 GPU, calculating the cost in euros per million generated tokens. This study, led by a data engineer, directly measures the power consumed by the GPU during inference using a physical meter connected to the system. The evaluated models range from small options like Phi-2 (2.7B parameters) to medium-sized models like Llama 2 13B and Mistral 7B, including quantized versions. The goal is to provide a practical metric for those considering local inference versus cloud APIs, a debate that has gained momentum with the democratization of LLMs.

Why Is This Important?

With the proliferation of open-source models like Llama, Mistral, and Gemma, more companies and developers are weighing running models locally to reduce costs, improve privacy, or avoid reliance on external providers. However, the energy cost of high-consumption GPUs like the RTX 3090 with a TDP of 350W is often underestimated or ignored in feasibility analyses. This study fills a gap in concrete data, allowing decision-makers to compare the real cost of local inference with API prices from OpenAI (GPT-4: ~€0.03 per 1K output tokens) or Anthropic (Claude 3: ~€0.015 per 1K tokens). Moreover, in a context where the energy sustainability of AI is increasingly questioned, measuring real consumption is a step toward more responsible adoption.

Key Results

The results reveal a cost range from less than €0.01 per million tokens to over €0.10, depending on the model and its configuration. Surprisingly, the cheapest model was not the smallest: Phi-2 (2.7B) cost €0.008/M tokens, while Mistral 7B quantized to 4 bits reached €0.006/M tokens, being the most efficient. At the other end, Llama 2 13B (unquantized) came to €0.12/M tokens. This demonstrates that architectural efficiency and software optimization (such as quantization and efficient kernels) can drastically reduce energy consumption, outperforming smaller but less optimized models. The RTX 3090, with measured consumption between 250W and 350W during inference, serves as a reference for high-end consumer GPUs; results can be extrapolated to other cards by adjusting consumption and relative performance.

Comparison with Cloud APIs

To put the data in perspective, OpenAI APIs charge per token: GPT-4 costs ~€0.03 per 1K output tokens (€30 per million tokens), while GPT-3.5 Turbo costs ~€0.002 per 1K tokens (€2/M tokens). On the local side, even the most expensive model (Llama 2 13B) costs €0.12/M tokens in electricity, assuming a price of €0.30/kWh (average European rate). This means that for high volumes (e.g., more than 10 million tokens per month), local execution can be significantly cheaper, especially when using efficient models. However, hardware cost must be considered: an RTX 3090 costs around €1,500, and its lifespan degrades with continuous use. For sporadic use, the cloud remains more competitive as it requires no upfront investment or maintenance. Additionally, the cloud offers instant scalability and access to larger models without memory limitations.

Market Implications

This type of analysis helps demystify the idea that local is always cheaper, and puts pressure on hardware manufacturers and software optimizers to improve energy efficiency. NVIDIA is already working on more efficient GPUs like the RTX 4090 (TDP of 450W but higher performance per watt) and upcoming Blackwell architectures. Frameworks like llama.cpp, TensorRT-LLM, and vLLM reduce consumption through quantization, pruning, and optimized kernels. Companies like Groq and Cerebras are betting on specialized hardware that promises much lower consumption per token. Additionally, cloud providers are offering instances with more efficient GPUs (such as A10G or L4) that can compete on total cost. The study also underscores the importance of measuring rather than assuming: “The cheapest model was not the smallest, nor the most expensive the largest,” the author notes, highlighting that software optimization can drastically alter the equation.

What Readers Should Know

• The real cost of running a local LLM includes electricity, hardware wear, setup time, and maintenance. The RTX 3090 has an estimated lifespan of 3-5 years under continuous load.
• For continuous use (more than ~10 million tokens per month), a local GPU can be cost-effective, especially with quantized and efficient models. Below that threshold, the cloud is more economical.
• Software optimization (4-bit quantization, kernels like FlashAttention, JIT compilation) can reduce consumption by up to 50% without significant quality loss.
• Electricity prices vary by region: in Europe the average is ~€0.30/kWh, in the US ~€0.13/kWh, and in countries with low rates like China (~€0.08/kWh), local is even more advantageous.
• The study does not include the cost of RAM, storage, or additional cooling, which can add 10-20% to total consumption.

Conclusion

Running LLMs locally has a tangible cost that can be precisely measured, and this study provides a solid foundation for companies and enthusiasts to compare options and optimize their deployments. Transparency in energy costs is a step toward more sustainable adoption of generative AI. As models become more efficient and hardware improves, the gap between local and cloud will narrow, but for now the decision depends on usage volume, privacy needs, and tolerance for obsolescence risk. In any case, measuring is the first step to optimizing.