The relentless rise in observability costs is a growing concern for engineering teams across the tech landscape, pushing leadership for answers and placing immense pressure on platform teams. While the immediate instinct is to point fingers at vendor pricing or implement tactical optimizations like sampling and log filtering, a deeper investigation into production telemetry reveals a more fundamental issue: a pervasive governance gap in how data is generated, managed, and understood. This pervasive challenge, impacting organizations of all sizes, highlights the critical need to shift focus from pipeline-level fixes to a proactive, source-level approach to data quality and management.
Over the past year, conversations with dozens of engineering teams and direct analysis of their production telemetry data have illuminated a recurring pattern. For instance, a prominent payments company, boasting thousands of engineers, is currently grappling with an observability tooling bill exceeding $200,000 per month. Their dedicated 18-person platform team dedicates daily efforts to monitoring cost estimates. When spikes occur, the process involves a laborious manual chase to identify the offending team and address the issue. While automated anomaly alerts were implemented to streamline this, their calibration remains a significant hurdle, leading to false positives and missed detections.
To rein in these spiraling expenses, the company has resorted to drastic measures. Debug logs are entirely excluded, and an astonishing 99% of unformatted logs are dropped before reaching the indexing stage. Certain categories of informational logs have also been curtailed. The financial strain has already prompted a migration from one observability vendor to another, and a secondary, self-hosted stack is now under development as a cost-hedging strategy. This scenario is not an isolated incident but rather a widely recognized "cost spiral" that organizations with substantial observability deployments frequently encounter. The underlying instrumentation, however, has remained largely unchanged, a testament to the superficiality of these cost-cutting efforts.
The Illusion of Vendor Solutions: Addressing Symptoms, Not Causes
The prevalent strategy of treating observability costs as a vendor-specific problem, by evaluating cheaper backends, implementing sampling, or routing data to less expensive tiers, represents a reactive approach that addresses symptoms rather than the root cause. While these optimizations may offer temporary relief, they fail to tackle the fundamental issues driving the escalating expenditure.
The core questions that organizations must confront are far more profound: What telemetry data are we actually generating? Who is accountable for each distinct signal? And critically, is any of this data truly providing actionable insights?
The Governance Gap: A Chasm in Data Accountability and Security
An in-depth analysis of production telemetry from a financial institution, operating over 4,700 services, revealed a startling statistic: a staggering 82% of all data points lacked a crucial service.name attribute. This meant millions of data points, generated every five minutes, were effectively ownerless. Without this essential identifier, attributing these metrics to a specific team, product area, or cost center became impossible. Despite the existence of instrumentation guidelines, their enforcement at scale was conspicuously absent.
The same analysis unearthed an even more alarming finding: 1,300 services were inadvertently leaking sensitive data into their observability pipelines. This was not a result of isolated developer errors but rather systemic issues. A framework-level Java Virtual Machine (JVM) truststore password, for example, was exposed in 352 services due to the framework automatically capturing it as a system property. Similarly, Kafka configuration logging led to credential exposure in 479 services. Tax IDs, bank account numbers, and authorization headers were also found across hundreds of other services. While security teams had established policies, automated enforcement mechanisms were missing.
These are not vendor-specific problems. Migrating to a different backend will not rectify instrumentation that captures sensitive credentials. Sampling will not magically imbue missing service.name attributes, nor will a cheaper data tier eliminate 75% log duplication.
A Case Study in Unchecked Trace Sampling and Data Leakage
At another financial operations (FinOps) review, held bi-monthly, a concerning pattern emerged. The initial trace implementation had defaulted to 100% sampling across all services. Years later, this setting had remained unchanged, fueled by a cultural expectation of having all traces readily available. This unchecked sampling, however, led to a critical issue: user IDs and offer IDs were being leaked into metric attributes. This created high-cardinality explosions, driving costs upward with every new customer interaction.
The Problem with Pipeline-Level Solutions: Operating Too Late
Solutions that focus on the observability pipeline, such as sampling, filtering, routing, and data dropping, aim to reduce data volume but fail to improve the inherent quality of the signal. These methods operate too late in the process. By the time telemetry reaches the pipeline, the cost of its generation has already been incurred, and any sensitive information has already been serialized and transmitted.
In a production environment supporting between 4,000 and 6,000 services, a significant reliance on auto-generated instrumentation was observed. This resulted in traces containing over 2,000 spans, largely composed of internal framework calls that provided no operational value. As one engineer succinctly put it, "We almost exclusively have auto-instrumentation and we see a lot of garbage and technical debt as a result." While sampling can reduce the volume of these traces, it cannot imbue them with meaning or operational relevance.
Shifting Focus to Source Quality: The Foundation of Effective Observability
The fundamental principle of improving quality at the source dictates that every metric, span, and log must exist for a clearly defined and stated reason. This involves adhering to established instrumentation conventions, ensuring attributes carry the necessary metadata for accurate attribution, and preventing sensitive data from ever entering the observability pipeline. This represents a paradigm shift from treating observability as an afterthought that happens to code, to actively designing it into the very fabric of the code.
Further analysis of a production environment revealed that 81% of all trace data consisted of Redis PING commands and health check endpoints. In another organization, an astonishing 75% of logs were exact duplicates, including repetitive Kafka client chatter, recurring Kubernetes event warnings, and one service emitting nearly 2,000 identical log lines within a five-minute window. While pipeline deduplication can compress this data, the underlying question remains: why is the application generating this redundant information in the first place?
Defining Effective Telemetry Governance: Practical Capabilities
Telemetry governance is the disciplined practice of understanding precisely what data is being collected, who owns it, and whether it serves a demonstrable purpose. Based on observed patterns across numerous organizations, effective telemetry governance encompasses several key capabilities:
Instrumentation Scoring: Quantifying Data Quality
Instrumentation scoring provides each service with a quantitative baseline for data quality. Instead of a generalized inquiry of "Is our instrumentation good?", the focus shifts to actionable questions like "Which services dropped below our quality threshold this week, and what were the contributing factors?". This makes data quality measurable, trackable, and subject to continuous improvement.
Automated Review: Proactive Problem Detection
Automated review processes are crucial for identifying and rectifying issues before they manifest in production. When a developer introduces a span attribute containing a user ID or a metric with unbounded cardinality, immediate feedback should be provided. This immediate feedback loop is far more effective than waiting for the next FinOps review or the arrival of an exorbitant bill.
Fleet-Wide Visibility: Ensuring Scalable Governance
Comprehensive fleet-wide visibility into SDK versions, configuration drift, and compliance with semantic conventions is essential for scaling governance effectively. When numerous teams onboard to a centralized platform, each with their own interpretation of guidelines, manual enforcement becomes an untenable strategy.
PII Detection: Safeguarding Sensitive Information
Automated detection of Personally Identifiable Information (PII) within telemetry data, not just within application logic, is vital. This capability can catch sensitive data leaks that might be missed during code reviews, including framework-level exposures, auto-instrumentation capturing headers, and business logic inadvertently logging transaction details into span attributes.
The Productive Question: Shifting from Cost to Quality
The next time the question of "Why is our observability bill so high?" arises, the most productive response is not to suggest switching vendors. Instead, the focus should pivot to a fundamental inquiry: "Let’s understand precisely what telemetry we are generating and why."
This reframing of the conversation shifts the objective from mere cost reduction to genuine quality improvement. It moves the discussion from the periphery of the data pipeline to its very source. Ultimately, improvements in data quality naturally lead to cost savings, as purposeful instrumentation results in less data to collect, store, and process, thereby reducing associated expenses.
The organizations that currently bear the heaviest observability costs are not necessarily those with the largest number of services. Instead, they are often the ones that have never adequately questioned the quality and purpose of their generated telemetry.
This analysis precedes KubeCon + CloudNativeCon Europe, the Cloud Native Computing Foundation’s flagship conference, scheduled to convene adopters and technologists from leading open-source and cloud-native communities in Amsterdam, the Netherlands, from March 23-26, 2026. The event serves as a critical forum for addressing such pressing challenges in the cloud-native ecosystem.