AWS cuts log costs with new analytical engine for OpenSearch
The new engine promises to reduce storage by up to 70% and improve performance in AI and agent environments.
July 2, 2026 · 3 min read
TL;DR: AWS presents an optimized analytical engine for logs in OpenSearch that reduces storage costs by up to 70%. It responds to the explosive growth of AI telemetry, which has led companies to discard 86% of logs to control expenses.
What happened?
AWS has announced a new analytical engine optimized for logs and telemetry within its managed service Amazon OpenSearch Service, designed to reduce storage costs by up to 70% and improve price performance compared to the general-purpose engine. This engine, currently in preview, uses Apache Parquet for columnar storage, maintains Lucene indexes for search fields, and employs Apache Calcite for query optimization, routing analytical operations to Apache DataFusion and search predicates to Lucene. It supports SQL and Piped Processing Language (PPL). The announcement was made during the AWS re:Invent 2024 conference, in a context where telemetry generated by AI applications and agents is growing exponentially.
Why is it important?
The growth of AI applications and agents is generating unprecedented volumes of telemetry. A Dynatrace survey revealed that AI workloads increased log volume by 93% year-over-year, leading companies to exclude an average of 86% of log data to control costs. This leaves organizations blind to security and compliance incidents. The new engine allows keeping more data for longer without skyrocketing costs, addressing a critical problem of modern observability. According to the Dynatrace report, companies implementing generative AI report an average 93% increase in log volume, but only 14% of those logs are retained for analysis. This means 86% of data is discarded, which can hide error patterns, security breaches, or performance issues. AWS's new engine promises to change this dynamic by offering a drastic reduction in storage costs, enabling companies to retain more data for longer periods.
Consequences for businesses and users
According to Ashish Chaturvedi of HFS Research, companies often reduce retention windows or sample logs, losing crucial data for unforeseen incidents. The new engine could change this. Shashi Bellamkonda of Info-Tech Research Group notes that AI agents broke OpenSearch's cost model: constant agent queries did not fit the original engine's assumptions. The new engine could help contain the proliferation of observability tools, reducing fragmentation and its hidden costs. Stephanie Walter of HyperFrame Research highlights that even if only a portion of the promised gains is achieved, lower storage costs translate into better compliance and more complete incident investigations. Additionally, the optimized engine could reduce the need for external storage solutions like Amazon S3 or data lake systems, simplifying the observability architecture. However, the challenge remains that companies must evaluate whether the 70% reduction holds for their specific usage patterns, as actual savings depend on data compressibility and query frequency.
What readers should know
The engine is available in preview for Amazon OpenSearch Service customers. AWS has not detailed specific pricing but promises a 70% reduction in storage costs. Companies should evaluate whether the reduction holds for their usage patterns. The engine maintains the same console, API, security model, and network configuration, easing migration. However, it is advisable to test with real workloads before adopting it in production. AWS has indicated the engine is compatible with existing indexes, but gradual data migration is recommended. Additionally, the engine supports hybrid indexes that combine columnar and search storage, enabling analytical and full-text queries in a single step. This contrasts with previous approaches where separate engines were required for logs and search. For companies already using Elasticsearch or open-source solutions, integration with OpenSearch can be a key factor. In terms of competition, this move by AWS pressures vendors like Elastic, Datadog, and Splunk, which offer observability solutions with high costs. If AWS delivers on the 70% reduction promise, it could significantly disrupt the observability market, especially for AI workloads. However, analysts warn that actual savings will depend on the engine's efficiency in real-world scenarios, and companies should conduct proof-of-concept tests before committing.