Many articles focus on the benefits of microservice architecture such as the possiblity of independent horizontal scaling of components or reduced cognitive load, due to compact codebase aligned with a business capability. Yet those benefits are paired with downsides. One of them is necessity to trace request across multiple microservices when debugging a production issue.

To combat this problem, the concept of correlation id was invented. Correlation id is a single indentifier that is propagated during request allowing to establish a single context across multiple microservices. This allows us to understand how a request flows through multiple services.

Enter OpenTelemetry

While it is possible to come up with your in-house solution, for the majority of cases, it would be more cost-efficient to use tools that are proven to be as industry standart at this point in time. OpenTelemetry is an open-source observability framework that provides a unified API, SDK, and tooling to instrument, generate, collect, and export telemetry data (traces, metrics, and logs).

Core Concepts

  • Activity: A unit of work that represents an operation (e.g., handling an HTTP request, processing a message). Each activity has a unique ID and can have parent-child relationships.
  • Trace: A collection of activities that represent the lifecycle of a request as it flows through a system.
  • Span: Another term for an activity, representing a single operation within a trace.
  • Baggage: Key-value pairs that propagate alongside the trace context, allowing you to attach custom metadata (like user IDs, correlation IDs) to every span in a trace.
  • ActivitySource: The factory that creates activities. Think of it as the entry point for instrumentation.

Setting Up OpenTelemetry Tracing

In an ASP.NET Core application, OpenTelemetry tracing is typically configured in the service collection:

services.AddOpenTelemetryTracing(builder =>
{
    builder.AddSource("API");
});

This tells OpenTelemetry to listen for activities created by the "API" activity source. You can register multiple sources for different parts of your application.

Activity Listener

To ensure all activities are sampled (recorded), an ActivityListener is configured:

ActivitySource.AddActivityListener(
    new ActivityListener
    {
        ShouldListenTo = _ => true,
        Sample = (ref _) => ActivitySamplingResult.AllData
    });

This listener accepts all activities and samples them with full data, ensuring no trace information is lost during development or in production.

Logging with Trace Context

One of the most powerful features of distributed tracing is the ability to correlate log entries with trace spans. This is achieved through Serilog enrichers.

Let’s have a look at how appsettings.json is configured

{
  "Serilog": {
    "Enrich": [
      "FromLogContext",
      "WithThreadId",
      "WithSpan"
    ],
    "WriteTo": [
      {
        "Name": "Console",
        "Args": {
          "outputTemplate": "[{Timestampmm:ss} {Level:u3} {SourceContext}] [{Properties}] {Message:lj}{NewLine}{Exception}"
        }
      }
    ]
  }
}

The WithSpan enricher automatically injects the current trace and span IDs into the log context. This means every log entry written while an activity is active will include:

  • TraceId: The unique identifier for the entire trace
  • SpanId: The unique identifier for the current operation

When you search logs by a TraceId, you can reconstruct the entire journey of a request across all services.

Context Propagation in REST APIs

When a request arrives at a REST API, the tracing context needs to be extracted from the HTTP headers. OpenTelemetry provides the DistributedContextPropagator for this purpose.

How It Works

  1. Incoming Request: The propagator extracts trace context from standard W3C Trace Context headers (traceparent, tracestate) or other proprietary headers.
  2. Activity Creation: A new activity (span) is started as a child of the extracted parent context.
  3. Outgoing Request: When the service calls another service, the current activity’s context is injected into the outgoing HTTP headers.

Activity Creation Pattern

using (var activity = ActivitySources.ApiActivitySource
    .StartActivity("Operation Name", ActivityKind.Server, parentContext.ActivityContext))
{
    // Work is done here, all logs will be correlated
}

The ActivityKind enum indicates the role of the activity: - Server: Handling an incoming request - Client: Making an outgoing request - Producer: Publishing a message - Consumer: Consuming a message

parentContext.ActivityContext parameter is worth noting as it allows seamlessly propagating context from the parent activity when ASP.NET Core is used. This, in turn, is what allows us to trace the path of the request across multiple services, preserving all the information we need.

Context Propagation in RabbitMQ

Propagating tracing context through message queues requires custom injectors and extractors, as RabbitMQ doesn’t natively support W3C Trace Context headers.

Injecting Context (Producer Side)

When publishing a message, the current activity’s trace context is serialized and added to the message headers:

public static void Inject(object carrier, string fieldName, string fieldValue)
{
    var props = carrier as IBasicProperties;
    props.Headers ??= new Dictionary<string, object>();
    props.Headers.Add(fieldName, fieldValue);
}

The DistributedContextPropagator.Current.Inject() method calls this injector for each trace context field, embedding the trace and span IDs into the RabbitMQ message properties.

Extracting Context (Consumer Side)

When consuming a message, the trace context is extracted from the headers:

public static IEnumerable<string> ExtractTraceContextFromHeaders(
    IDictionary<string, object> headers, string key)
{
    if (headers.TryGetValue(key, out object value))
    {
        var bytes = value as byte[];
        return new string[] { Encoding.UTF8.GetString(bytes) };
    }
    return Array.Empty<string>();
}

The consumer then starts a new activity linked to the parent context:

var parentContext = _propagator.Extract(
    default, headers, DistributedContextExtractor.ExtractTraceContextFromHeaders);

using (var activity = ActivitySources.ApiActivitySource
    .StartActivity("consume", ActivityKind.Consumer, parentContext.ActivityContext))
{
    // Process message
}

Propagating Baggage

Baggage allows you to pass custom metadata through the distributed trace. The ActivityPopulator utility class helps transfer baggage from a parent activity to a child activity:

public static void PopulateBaggageFromParentActivity(Activity parentActivity, Activity activity)
{
    if (parentActivity is null || activity is null) return;

    foreach (var item in parentActivity.Baggage)
    {
        activity.SetBaggage(item.Key, item.Value);
    }
}

This ensures that custom correlation data (like material IDs, user IDs, or request identifiers) flows through the entire trace chain.

Here’s how it used 

var parentContext = _propagator.Extract(default, headers, DistributedContextExtractor.ExtractTraceContextFromHeaders);
var baggage = DistributedContextPropagator.Current.ExtractBaggage(headers, DistributedContextExtractor.ExtractBaggageFromHeaders);

using (var activity = ActivitySources.IisApiActivitySource.StartActivity($"{ApplicationName} consume", ActivityKind.Consumer, parentContext.ActivityContext))
{
    ActivityPopulator.PopulateBaggage(baggage, activity);
    // further processing
}

Further work

At this point, we can dissect logs in our setup quering it by TraceId and visualising the path of a complex multi-service request. As a further step, we can visualise those paths using Jaeger.

Conclusion

In the current landscape of multi-microservices architectures ability to understand the path of each request is crucial for observability and debugging purposes. Opentelemetry grants us this posilibity allowing us to almost seamlessly enrich our logs with trace information as well as visualising it with modern tools.