OpenTelemetry Instrumentation

Hello everyone, Welcome to our blog!

Today we will see in detail how to instrument OpenTelemtry with

1. Manual Instrumentation
2. Automatic Instrumentation

To learn about opentelemetry read my previous blog here

Lets get started

Manual Instrumentation

Libraries that want to use OpenTelemetry to export telemetry data should only use the opentelemetry-api package, never configuring or relying on the OpenTelemetry SDK. Applications that depend on the opentelemetry-sdk package, or any other implementation of the OpenTelemetry API, must supply the SDK configuration. Libraries will only get a genuine implementation if the user application is configured for it in this way. Check out the Library Guidelines for further information.

Set up

The first step is to get a handle to an instance of the OpenTelemetry interface.

If you are an application developer, you need to configure an instance of the OpenTelemetrySdk as early as possible in your application. This can be done using the OpenTelemetrySdk.builder() method.

For example:

SdkTracerProvider sdkTracerProvider = SdkTracerProvider.builder()
.addSpanProcessor(BatchSpanProcessor.builder(OtlpGrpcSpanExporter.builder().build()).build())
    .build();
OpenTelemetry openTelemetry = OpenTelemetrySdk.builder()
    .setTracerProvider(sdkTracerProvider)
 .setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance()))
    .buildAndRegisterGlobal();

As an aside, if you are writing library instrumentation, it is strongly recommended that you provide your users the ability to inject an instance of OpenTelemetry into your instrumentation code. If this is not possible for some reason, you can fall back to using an instance from the GlobalOpenTelemetry class. Note that you can’t force end-users to configure the global, so this is the most brittle option for library instrumentation.

Tracing

In the following, we present how to trace code using the OpenTelemetry API.

First, a Tracer must be acquired, which is responsible for creating spans and interacting with the Context. A tracer is acquired by using the OpenTelemetry API specifying the name and version of the library instrumenting the instrumented library or application to be monitored. More information is available in the specification chapter Obtaining a Tracer.

Tracer tracer =
    openTelemetry.getTracer("instrumentation-library-name", "1.0.0");

Important: the “name” and optional version of the tracer are purely informational. All Tracers that are created by a single OpenTelemetry instance will interoperate, regardless of name.

Create a basic Span

To create a basic span, you only need to specify the name of the span. The start and end time of the span is automatically set by the OpenTelemetry SDK.

Span span = tracer.spanBuilder("my span").startSpan();
// put the span into the current Context
try {
  // do something...
} finally {
    span.end();
}

Create nested Spans

Most of the time, we want to correlate spans for nested operations. OpenTelemetry supports tracing within processes and across remote processes. For more details how to share context between remote processes, see Context Propagation.

For a method a calling a method b, the spans could be manually linked in the following way:

void parentOne() {
  Span parentSpan = tracer.spanBuilder("parent").startSpan();
  try {
    childOne(parentSpan);
  } finally {
    parentSpan.end();
  }
}

void childOne(Span parentSpan) {
  Span childSpan = tracer.spanBuilder("child")
        .setParent(Context.current().with(parentSpan))
        .startSpan();
  try {
    // do stuff
  } finally {
    childSpan.end();
  }
}

The OpenTelemetry API offers also an automated way to propagate the parent span on the current thread:

void parentTwo() {
  Span parentSpan = tracer.spanBuilder("parent").startSpan();
  try(Scope scope = parentSpan.makeCurrent()) {
    childTwo();
  } finally {
    parentSpan.end();
  }
}
void childTwo() {
  Span childSpan = tracer.spanBuilder("child")
    // NOTE: setParent(...) is not required;
    // `Span.current()` is automatically added as the parent
    .startSpan();
  try(Scope scope = childSpan.makeCurrent()) {
    // do stuff
  } finally {
    childSpan.end();
  }
}

To link spans from remote processes, it is sufficient to set the Remote Context as parent.

Span childRemoteParent = tracer.spanBuilder("Child").setParent(remoteContext).startSpan();

Get the current span

Sometimes it’s helpful to do something with the current/active span at a particular point in program execution.

Span span = Span.current()

And if you want the current span for a particular Context object:

Span span = Span.fromContext(context)

Span Attributes

In OpenTelemetry spans can be created freely and it’s up to the implementor to annotate them with attributes specific to the represented operation. Attributes provide additional context on a span about the specific operation it tracks, such as results or operation properties.

Span span = tracer.spanBuilder("/resource/path").setSpanKind(SpanKind.CLIENT).startSpan();
span.setAttribute("http.method", "GET");
span.setAttribute("http.url", url.toString());

Some of these operations represent calls that use well-known protocols like HTTP or database calls. For these, OpenTelemetry requires specific attributes to be set. The full attribute list is available in the Semantic Conventions in the cross-language specification.

Create Spans with events

Spans can be annotated with named events that can carry zero or more Span Attributes, each of which is itself a key:value map paired automatically with a timestamp.

span.addEvent("Init");
...
span.addEvent("End");

Attributes eventAttributes = Attributes.of(
    AttributeKey.stringKey("key"), "value",
    AttributeKey.longKey("result"), 0L);

span.addEvent("End Computation", eventAttributes);

Create Spans with links

A Span may be linked to zero or more other Spans that are causally related. Links can be used to represent batched operations where a Span was initiated by multiple initiating Spans, each representing a single incoming item being processed in the batch.

Span child = tracer.spanBuilder("childWithLink")
        .addLink(parentSpan1.getSpanContext())
        .addLink(parentSpan2.getSpanContext())
        .addLink(parentSpan3.getSpanContext())
        .addLink(remoteSpanContext)
    .startSpan();

For more details how to read context from remote processes, see Context Propagation.

Set span status

A status can be set on a span, typically used to specify that a span has not completed successfully – SpanStatus.Error. In rare scenarios, you could override the Error status with OK, but don’t set OK on successfully-completed spans.

The status can be set at any time before the span is finished:

Span span = tracer.spanBuilder("my span").startSpan();
// put the span into the current Context
try (Scope scope = span.makeCurrent()) {
	// do something
} catch (Throwable t) {
  span.setStatus(StatusCode.ERROR, "Something bad happened!");
} finally {
  span.end(); // Cannot set a span after this call
}

Record exceptions in spans

It can be a good idea to record exceptions when they happen. It’s recommended to do this in conjunction with setting span status.

Span span = tracer.spanBuilder("my span").startSpan();
// put the span into the current Context
try (Scope scope = span.makeCurrent()) {
	// do something
} catch (Throwable throwable) {
  span.setStatus(StatusCode.ERROR, "Something bad happened!");
  span.recordException(throwable)
} finally {
  span.end(); // Cannot set a span after this call
}

This will capture things like the current stack trace in the span.

Context Propagation

OpenTelemetry provides a text-based approach to propagate context to remote services using the W3C Trace Context HTTP headers.

The following presents an example of an outgoing HTTP request using HttpURLConnection.

// Tell OpenTelemetry to inject the context in the HTTP headers
TextMapSetter<HttpURLConnection> setter =
  new TextMapSetter<HttpURLConnection>() {
    @Override
    public void set(HttpURLConnection carrier, String key, String value) {
        // Insert the context as Header
        carrier.setRequestProperty(key, value);
    }
};

URL url = new URL("http://127.0.0.1:8080/resource");
Span outGoing = tracer.spanBuilder("/resource").setSpanKind(SpanKind.CLIENT).startSpan();
try (Scope scope = outGoing.makeCurrent()) {
  // Use the Semantic Conventions.
  // (Note that to set these, Span does not *need* to be the current instance in Context or Scope.)
  outGoing.setAttribute(SemanticAttributes.HTTP_METHOD, "GET");
  outGoing.setAttribute(SemanticAttributes.HTTP_URL, url.toString());
  HttpURLConnection transportLayer = (HttpURLConnection) url.openConnection();
  // Inject the request with the *current*  Context, which contains our current Span.
  openTelemetry.getPropagators().getTextMapPropagator().inject(Context.current(), transportLayer, setter);
  // Make outgoing call
} finally {
  outGoing.end();
}
...

Similarly, the text-based approach can be used to read the W3C Trace Context from incoming requests. The following presents an example of processing an incoming HTTP request using HttpExchange.

TextMapGetter<HttpExchange> getter =
  new TextMapGetter<>() {
    @Override
    public String get(HttpExchange carrier, String key) {
      if (carrier.getRequestHeaders().containsKey(key)) {
        return carrier.getRequestHeaders().get(key).get(0);
      }
      return null;
    }

   @Override
   public Iterable<String> keys(HttpExchange carrier) {
     return carrier.getRequestHeaders().keySet();
   }
};
...
public void handle(HttpExchange httpExchange) {
  // Extract the SpanContext and other elements from the request.
  Context extractedContext = openTelemetry.getPropagators().getTextMapPropagator()
        .extract(Context.current(), httpExchange, getter);
  try (Scope scope = extractedContext.makeCurrent()) {
    // Automatically use the extracted SpanContext as parent.
    Span serverSpan = tracer.spanBuilder("GET /resource")
        .setSpanKind(SpanKind.SERVER)
        .startSpan();
    try {
      // Add the attributes defined in the Semantic Conventions
      serverSpan.setAttribute(SemanticAttributes.HTTP_METHOD, "GET");
      serverSpan.setAttribute(SemanticAttributes.HTTP_SCHEME, "http");
      serverSpan.setAttribute(SemanticAttributes.HTTP_HOST, "localhost:8080");
      serverSpan.setAttribute(SemanticAttributes.HTTP_TARGET, "/resource");
      // Serve the request
      ...
    } finally {
      serverSpan.end();
    }
  }
}

Metrics

Spans are a great way to get detailed information about what your application is doing, but what about a more aggregated perspective? OpenTelemetry provides supports for metrics, a time series of numbers that might express things such as CPU utilization, request count for an HTTP server, or a business metric such as transactions.

Note The stability of Metrics in opentelemetry-java is mixed. The first stable metrics API was release in version 1.10.0; however, the metrics SDK is alpha and still subject to change. The metrics API is included in the opentelemetry-api module. In order to access the alpha metrics SDK library, you will need to explicitly depend on the opentelemetry-sdk-metrics module.

All metrics can be annotated with attributes: additional qualifiers that help describe what subdivision of the measurements the metric represents.

The following is an example of counter usage:

OpenTelemetry openTelemetry = // obtain instance of OpenTelemetry

// Gets or creates a named meter instance
Meter meter = openTelemetry.meterBuilder("instrumentation-library-name")
        .setInstrumentationVersion("1.0.0")
        .build();

// Build counter e.g. LongCounter
LongCounter counter = meter
      .counterBuilder("processed_jobs")
      .setDescription("Processed jobs")
      .setUnit("1")
      .build();

// It is recommended that the API user keep a reference to Attributes they will record against
Attributes attributes = Attributes.of(stringKey("Key"), "SomeWork");

// Record data
counter.add(123, attributes);

Asynchronous instruments support collecting metrics on demand, once per collection interval.

The following is an example of usage of an asynchronous instrument:

// Build an asynchronous instrument, e.g. Gauge
meter
  .gaugeBuilder("cpu_usage")
  .setDescription("CPU Usage")
  .setUnit("ms")
  .buildWithCallback(measurement -> {
    measurement.record(getCpuUsage(), Attributes.of(stringKey("Key"), "SomeWork"));
  });

Tracing SDK Configuration

The configuration examples reported in this document only apply to the SDK provided by opentelemetry-sdk. Other implementation of the API might provide different configuration mechanisms.

The application has to install a span processor with an exporter and may customize the behavior of the OpenTelemetry SDK.

For example, a basic configuration instantiates the SDK tracer provider and sets to export the traces to a logging stream.

SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
  .addSpanProcessor(BatchSpanProcessor.builder(new LoggingSpanExporter()).build())
  .build();

Sampler

It is not always feasible to trace and export every user request in an application. In order to strike a balance between observability and expenses, traces can be sampled.

The OpenTelemetry SDK offers four samplers out of the box:

• AlwaysOnSampler which samples every trace regardless of upstream sampling decisions.
• AlwaysOffSampler which doesn’t sample any trace, regardless of upstream sampling decisions.
• ParentBased which uses the parent span to make sampling decisions, if present.
• TraceIdRatioBased which samples a configurable percentage of traces, and additionally samples any trace that was sampled upstream.

Additional samplers can be provided by implementing the io.opentelemetry.sdk.trace.Sampler interface.

SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
  .setSampler(Sampler.alwaysOn())
  //or
  .setSampler(Sampler.alwaysOff())
  //or
  .setSampler(Sampler.traceIdRatioBased(0.5))
  .build();

Span Processor

Different Span processors are offered by OpenTelemetry. The SimpleSpanProcessor immediately forwards ended spans to the exporter, while the BatchSpanProcessor batches them and sends them in bulk. Multiple Span processors can be configured to be active at the same time using the MultiSpanProcessor.

SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
  .addSpanProcessor(SimpleSpanProcessor.create(new LoggingSpanExporter()))
  .addSpanProcessor(BatchSpanProcessor.builder(new LoggingSpanExporter()).build())
  .build();

Exporter

Span processors are initialized with an exporter which is responsible for sending the telemetry data a particular backend. OpenTelemetry offers five exporters out of the box:

• In-Memory Exporter: keeps the data in memory, useful for debugging.
• Jaeger Exporter: prepares and sends the collected telemetry data to a Jaeger backend via gRPC.
• Zipkin Exporter: prepares and sends the collected telemetry data to a Zipkin backend via the Zipkin APIs.
• Logging Exporter: saves the telemetry data into log streams.
• OpenTelemetry Exporter: sends the data to the OpenTelemetry Collector.

ManagedChannel jaegerChannel = ManagedChannelBuilder.forAddress("localhost", 3336)
  .usePlaintext()
  .build();

JaegerGrpcSpanExporter jaegerExporter = JaegerGrpcSpanExporter.builder()
  .setEndpoint("localhost:3336")
  .setTimeout(30, TimeUnit.SECONDS)
  .build();

SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
  .addSpanProcessor(BatchSpanProcessor.builder(jaegerExporter).build())
  .build();

Auto Configuration

To configure the OpenTelemetry SDK based on the standard set of environment variables and system properties, you can use the opentelemetry-sdk-extension-autoconfigure module.

OpenTelemetrySdk sdk = OpenTelemetrySdkAutoConfiguration.initialize();

Logging and Error Handling

OpenTelemetry uses java.util.logging to log information about OpenTelemetry, including errors and warnings about misconfigurations or failures exporting data.

By default, log messages are handled by the root handler in your application. If you have not installed a custom root handler for your application, logs of level INFO or higher are sent to the console by default.

You may want to change the behavior of the logger for OpenTelemetry. For example, you can reduce the logging level to output additional information when debugging, increase the level for a particular class to ignore errors coming from that class, or install a custom handler or filter to run custom code whenever OpenTelemetry logs a particular message.

Examples

## Turn off all OpenTelemetry logging
io.opentelemetry.level = OFF

## Turn off logging for just the BatchSpanProcessor
io.opentelemetry.sdk.trace.export.BatchSpanProcessor.level = OFF

## Log "FINE" messages for help in debugging
io.opentelemetry.level = FINE

## Sets the default ConsoleHandler's logger's level
## Note this impacts the logging outside of OpenTelemetry as well
java.util.logging.ConsoleHandler.level = FINE

Tags: how to get telemetry dataopentelemetry automatic instrumentationopentelemetry how toopentelemetry javaopentelemetry manual instrumentationopentelemtry instrumentation