In the world of cloud-native systems, observability isn’t a luxury; it’s a necessity. And for many organizations, Datadog is the undisputed king of observability platforms. It offers a “single pane of glass” that promises to unify metrics, traces, and logs, providing unparalleled insight into complex applications.

The initial sales pitch is compelling: install an agent, enable a few integrations, and voilà—your dashboards light up with data. But as many engineering teams have discovered, the initial “sticker price” is just the beginning. The real financial story unfolds weeks or months later, with a bill that can be orders of magnitude higher than anticipated.

This isn’t just about underestimating log volume or the number of hosts. The most significant and often unforeseen costs are baked into the very design of the platform, where features are deeply coupled, creating a powerful—and expensive—snowball effect.

The Gateway Drug: Application Performance Monitoring (APM)

The journey into cost overruns often starts with a simple, sensible goal: “We need to understand our application’s performance.” The natural answer within the Datadog ecosystem is to enable APM, their distributed tracing solution.

You instrument your services, and almost instantly, you get beautiful flame graphs and service maps. You can trace a request from the front end all the way down to the database. It feels like magic.

But what you’ve actually done is opened a firehose of data, and Datadog is waiting with a dozen different buckets to catch it, each with its own price tag.

How Tracing Becomes the Center of Your Bill

Enabling APM doesn’t just generate traces. It acts as a catalyst that pulls other, separately priced features into its orbit.

  1. Indexed Spans: The first “gotcha.” The traces themselves are one cost, but to search and analyze them (which is the whole point), you need to index spans. Datadog charges per million indexed spans. For a high-throughput service, this number grows incredibly fast.

  2. Log Ingestion and Indexing: To get the full value of a trace, you need to see the logs associated with each span. So, you configure your loggers to inject trace IDs and send them to Datadog. Now you’re paying for:

    • Log Ingestion: A fee for every gigabyte of logs sent. (trace ids increase the avg log size)
    • Log Indexing: A separate, much higher fee for the logs you want to make searchable. Injecting trace IDs also increases the size of every single log line, subtly driving up both ingestion and indexing costs across the board. While cheaper tiers like Flex Logging exist, they come with limitations on searchability and alerting.

  3. Custom Metrics from Spans: Your traces contain a wealth of information. You might think, “I’ll create a metric to track the latency of a specific span tag.” You’ve just created a custom metric from a span. Datadog’s custom metric pricing is based on the number of distinct metric-and-tag combinations (cardinality). A single, poorly-designed metric can generate thousands of time series, adding a significant, recurring cost.

  4. Infrastructure Monitoring: Your traces show a service is slow, but is it the code or the underlying host? To find out, you need the Datadog Infrastructure Agent running on your hosts or containers. That’s another per-host or per-container fee.

  5. The Upsell Pile-On: Once you’re sending traces, the path to enabling other products becomes frictionless—and costly:

    • Application Security Management (ASM): Hooks directly into your APM tracer to find vulnerabilities. It’s a great product, but it’s another line item on your bill, priced per service.
    • Universal Service Monitoring (USM): Provides service-to-service communication maps without code changes, but it’s yet another cost layer.
    • Data Streams Monitoring: If you use Kafka or RabbitMQ, this product offers deep insights into your message queues. It’s incredibly useful, but it’s also another per-host charge that can add up quickly in a distributed streaming architecture.

What started as a single decision to enable tracing has now cascaded into six or more distinct, billable data streams.

The “Datadog Tax”: When Coupling Creates Cost

This tight coupling is both Datadog’s greatest strength and its biggest financial trap. The platform is designed so that each feature is more valuable when used with others. This creates a powerful incentive to turn everything on.

The problem is that the pricing model treats them as à la carte services. It’s like going to a restaurant where the burger is $5, but the bun, the patty, the lettuce, and the plate are all sold separately. You don’t realize the full cost until the check arrives.

This “Datadog Tax” is the hidden premium you pay for the convenience of an integrated ecosystem.

Strategies for Taming the Datadog Beast

Complaining about the cost is easy. Controlling it is harder, but not impossible. It requires discipline, governance, and a shift in mindset from “observe everything” to “observe what matters.”

1. Be Ruthless with Sampling

Do you really need to trace 100% of requests for every service? For most use cases, the answer is no.

  • Head-based Sampling: Configure the Datadog agent to sample a percentage of traces at the source. Start aggressively (e.g., 5-10%) and adjust as needed.
  • Ingestion Controls: Use Datadog’s Ingestion Controls to set rules that retain traces based on specific criteria (e.g., keep all traces with errors, but only 1% of successful ones).

2. Govern Your Metrics

Custom metrics are a budget black hole.

  • Establish a Process: Don’t let every engineer create custom metrics on the fly. Create a process for proposing and reviewing new metrics.
  • Watch Cardinality: Educate your team about cardinality. Use fixed, low-cardinality tags. Avoid using things like user IDs, request IDs, or container IDs as tags.
  • Use Logs-to-Metrics: For many use cases, you can generate metrics from your indexed logs instead of creating a custom metric, which is often cheaper.

3. Tier Your Logs

Not all logs are created equal. You don’t need to index every DEBUG message.

  • Index What’s Critical: Be selective about which logs get indexed. Index error logs and key application events.
  • Rehydrate on Demand: Send everything else to a cheaper archival tier (like Datadog’s own or S3). You can “rehydrate” them back into Datadog for analysis if an incident occurs. It’s cheaper than paying for constant indexing.

4. Start Small and Justify Expansion

Resist the urge to enable a new Datadog product for your entire organization at once.

  • Pilot Programs: Roll out new features (like ASM or USM) to a single, critical service first.
  • Measure ROI: Before expanding, ask: “Did this new feature provide value proportional to its cost?” If you can’t answer that, don’t roll it out further.

5. Break Vendor Lock-in with an Observability Pipeline

This is an advanced but powerful strategy. Relying solely on Datadog’s proprietary SDKs and agents creates deep vendor lock-in. Breaking free gives you ultimate control and bargaining power.

  • Adopt OpenTelemetry (OTel): Instead of using Datadog’s SDKs, instrument your applications with OpenTelemetry, the vendor-neutral open standard for telemetry data. This ensures your instrumentation is portable.
  • Implement an Observability Pipeline: Deploy a tool like the OpenTelemetry Collector or Vector between your applications and Datadog. This pipeline acts as a central control plane for all your telemetry data.

With a pipeline, you can:

  • Centrally manage sampling: Enforce sampling rules for all services in one place.
  • Filter and scrub data: Remove sensitive information or noisy, low-value data before it ever reaches Datadog, saving you money.
  • Route data intelligently: Send 100% of logs to a cheap object store like S3 for compliance, while sending only critical, indexed logs to Datadog.
  • Gain Bargaining Power: When your contract is up for renewal, you can easily dual-send your data to a competitor for evaluation. This ability to switch vendors without re-instrumenting your entire codebase is a massive advantage in negotiations.

Conclusion: Observability with Intention

Datadog is an exceptional tool, and for many, it’s worth the price. But going in blind is a recipe for financial disaster. The platform’s interconnected nature means that small decisions can have huge financial consequences.

By understanding how features like APM act as a cost-multiplier and by implementing strict governance around data ingestion, you can harness the power of Datadog without letting its costs spiral out of control.

Treat observability as you would any other part of your architecture: with intention, discipline, and a constant eye on the cost-to-value ratio. Your CFO and FinOps team will thank you.

✏️ Personal Notes

  • This article reflects a common experience in the industry, but every organization’s usage patterns are different. Your mileage may vary.
  • The goal isn’t to bash Datadog—it’s a best-in-class product for a reason. The goal is to encourage financial awareness and deliberate implementation.
  • Always, always read the pricing page. Then read it again. And model your expected costs before you enable anything.