In the last post, we established a foundational principle: one application, one key. This practice gives you critical application-level identity, saving you from the “Four Horsemen” of shared-key chaos: security breaches, rate-limiting famines, observability black holes, and debugging nightmares.

Your services are now distinct entities. Your billing-service has its own key, and your analytics-worker has another. You can sleep a little better.

But a new question emerges, especially for any SaaS platform that acts as a proxy between its users and a third-party API (like OpenAI), Stripe, or Twilio:

All the requests from my api-gateway use a single, secure key. But how do I know which of my users is responsible for which call?

This is where the next layer of identity comes into play: the end-user ID. Many modern APIs allow you to pass a unique identifier representing the person who initiated the action. It’s a powerful feature that, when combined with internal identifiers like a source-service-id, unlocks multiple layers of observability. However, it also walks a fine line between visibility and privacy.

The Power: Why You Need End-User IDs

Imagine you run a SaaS product that helps users write marketing copy using an AI model. All API calls to the AI provider go through your backend, using your backend’s API key. Without an end-user ID, your provider sees a single, monolithic stream of traffic. With it, you unlock several superpowers.

Pro 1: Surgical Abuse Mitigation

This is the most compelling reason. A malicious user signs up for your service and starts running abusive prompts, attempting to jailbreak the model or generate harmful content.

  • Without End-User ID: The AI provider detects the abuse and rate-limits or blocks your entire application key. All of your customers are now offline because of one bad actor. You are blind.
  • With End-User ID: The provider can pinpoint the abuse to user-id-1a2b3c. They can notify you, and you can immediately disable that specific user’s account. The provider might even be able to block that user ID on their end. The blast radius is contained to a single user, not your entire platform.

Pro 2: Granular Cost Attribution and FinOps

Your monthly bill from the AI provider arrives, and it’s doubled. Where did the cost come from?

  • Without End-User ID: You have no idea. Was it one power user? A new feature taking off? A bug causing runaway calls? You can’t attribute costs to customers, making it impossible to implement fair usage policies or tiered pricing.
  • With End-User ID: You can now slice and dice your usage data. You can see that 80% of your costs are coming from 5% of your users. This insight is gold for your product and finance teams (FinOps). You can build dashboards showing customers their own usage and justify pricing tiers.

Pro 3: Enhanced Debugging and Customer Support

A customer writes in: “The AI feature is giving me weird results.”

  • Without End-User ID: Your support team is back to correlating timestamps. “What time did this happen? What was the exact input?” The process is slow and frustrating.
  • With End-User ID: You can immediately look up all API calls made by that user’s ID in your logs or your provider’s dashboard. You see the exact inputs, outputs, and any errors, allowing you to resolve the issue in minutes.

Pro 4: Lightweight Internal Observability with Source Service IDs

While the previous points focus on third-party APIs, the principle of propagating identity internally is a game-changer. Full distributed tracing is the gold standard, but it can be complex and costly. A powerful, budget-friendly alternative is to propagate a source-service-id.

The source-service-id is the name of the service that initiated a request chain. For example, if a customer request hits your api-gateway, that gateway injects a header like X-Source-Service-ID: api-gateway into all downstream calls it makes to other services. If a batch job in your reporting-service kicks off a task, it injects X-Source-Service-ID: reporting-service.

This simple header allows you to answer critical questions without a full tracing system:

  • Dependency Mapping: “Which services are calling our payments-service? Is it just the checkout-api, or is the marketing-service also making calls?”
  • Resource Attribution: “How much of the load on the user-database is coming from real-time user traffic (api-gateway) versus background processing (analytics-worker)?”

This approach provides a significant portion of the benefits of distributed tracing for a fraction of the implementation cost. You still pass the end-user-id in logs for deep debugging, but you use the low-cardinality source-service-id for your metrics.

A Concrete Cost Comparison: Custom Metrics vs. Full APM

Let’s make this more concrete by comparing the cost of this lightweight approach to a full-featured APM (Application Performance Monitoring) solution like Datadog APM.

  • Full APM & Distributed Tracing (e.g., Datadog): The cost model is typically multi-faceted. You are often charged per host running the tracing agent, plus a fee for the volume of trace data (spans) ingested. For a system with dozens of microservices running on many hosts, this can quickly escalate to thousands of dollars per month. The value is immense—you get automatic service maps, detailed flame graphs, and out-of-the-box latency analysis—but the price tag is significant.

  • Lightweight Approach (Logs + Custom Metrics): With this method, your primary cost is log ingestion and storage, which you are likely already paying for. The additional cost comes from generating custom metrics from those logs. Platforms like Datadog or Splunk typically charge per 100 custom metrics.

The key difference is one of scale and efficiency. A single user request can generate thousands of trace spans in a deep microservices architecture, leading to high data volume. In contrast, you might only generate a handful of highly relevant, low-cardinality custom metrics from your logs for that same request (e.g., request.success.count and request.latency.ms, both tagged by source-service-id).

While not a full replacement for APM, this layered approach allows you to gain critical, queryable, segmented visibility at what is often an order of magnitude lower cost. It’s a pragmatic trade-off that delivers significant value for a fraction of the price, making it a perfect starting point for observability.

The Peril: The Privacy Minefield

The benefits are clear, but passing user data to a third party is fraught with risk. The primary danger is the potential leakage of Personally Identifiable Information (PII).

Con 1: PII Exposure and Regulatory Risk

What are you using for the user parameter?

  • A user’s email (user: "[email protected]"): You have just sent PII to a third party. This could violate privacy regulations like GDPR or CCPA and break the trust of your users.
  • A user’s internal database ID (user: "42"): This is better, as it’s pseudonymized. However, if that third party ever has a data breach, and an attacker also breaches your database, they can now link the two datasets together, re-identifying the users.

Con 2: The Hashing Fallacy

A common but flawed mitigation is to hash the identifier. For example, sending sha256("[email protected]") as the user ID.

This feels secure, but it’s often ineffective. Emails are not high-entropy secrets. An attacker with a list of common email addresses (or all the emails from a previous data breach) can pre-compute the hashes and compare them against a leaked dataset from the third party. This is known as a rainbow table attack. While a salted hash is much stronger, it complicates your ability to consistently generate the same ID for the same user across all systems.

Con 3: Vendor Lock-in and Data Portability

Once you start relying on a third party’s ability to track your per-user data, your systems become more tightly coupled with that vendor. Migrating to a new provider becomes harder because you risk losing the historical data and observability you’ve built around their specific implementation of user tracking.

The Balanced Approach: A Guideline for Implementation

So, how do you reap the benefits without falling into the privacy traps?

  1. Never Use Raw PII: Do not use emails, usernames, or real names as end-user IDs.
  2. Use Stable, Opaque, and Anonymous Identifiers: The best practice is to generate a separate, random, and persistent identifier for each user specifically for this purpose. Use a UUID (e.g., f47ac10b-58cc-4372-a567-0e02b2c3d479) that is stored in your user database. This ID has no connection to the user’s PII and is not your internal primary key.
  3. Consult Your Legal Team: Before sending any user-related data to a third party, review your Data Processing Agreements (DPAs) and consult with legal counsel to ensure you are compliant with all relevant privacy laws.
  4. Be Transparent with Users: Your privacy policy should clearly state that you share pseudonymized identifiers with third-party service providers for the purposes of security, monitoring, and service operation.

Conclusion: Identity is a Layered Concern

Just as unique API keys provide identity for your applications, end-user IDs and source-service IDs provide vital, more granular layers of identity for your users and internal request flows.

Ignoring end-user IDs leaves you blind to user-specific behavior, making you vulnerable to abuse and unable to answer critical business questions. Implementing them carelessly creates significant privacy risks. And neglecting internal identifiers like source-service-id leaves you with observability gaps that are expensive to fill with more complex tooling.

By using stable, opaque identifiers, adopting a privacy-first mindset, and distinguishing between high- and low-cardinality data, you can unlock immense observability while protecting your most valuable asset: your users’ trust.

✏️ Personal Notes

  • I want to be clear: I’m not against distributed tracing. Full APM solutions are incredibly powerful. However, their cost can be prohibitive for many businesses, especially smaller teams or new projects. The source-service-id approach is a pragmatic first step that delivers immense value without the hefty price tag. It’s about choosing the right tool for your current scale and budget.
  • The tension between observability and privacy can be a challenging aspects of software engineering. The “right” answer often involves legal and product decisions, not just technical ones. The best technical solution is one that gives your business options while defaulting to protecting user data.