Introduction
Microservices have changed the way we build software, making applications more scalable and flexible. But with this shift comes new challenges—especially when it comes to monitoring and debugging. Traditional logging tools struggle to track requests as they move between services, making it harder to spot issues. That’s where distributed tracing helps, and OpenTelemetry has become a go-to solution for making it easier to see what’s happening across an entire system.
What is Distributed Tracing?
Distributed tracing is a technique of following requests as they pass through various services in a microservices-based application. Every request creates a distinct trace ID, which is propagated through services. Through the gathering and examination of these traces, developers are able to:
- Identify bottlenecks
- Debug latency issues
- Monitor system health
- Gain insights into service dependencies
Challenges in Microservices Observability
Microservices introduce several challenges:
- Multiple Services: An individual request might go through numerous services and prove challenging to track down errors in.
- Asynchronous Communication: With event-driven architectures and message queues, it is difficult to track interactions.
- High Volume of Data: Each of these services has logs and metrics, resulting in a very large volume of data.
Introduction to OpenTelemetry
OpenTelemetry is an open-source observability platform that offers APIs, SDKs, and tools to gather, process, and export telemetry data (traces, metrics, and logs). It is language-agnostic and can be used with various backends, including Jaeger, Zipkin, and Prometheus.
1. Instrumentation
OpenTelemetry offers both auto and manual instrumentation to record trace data. It has support for widely used frameworks such as Express.js, FastAPI, and Spring Boot, simplifying the integration with existing microservices.
2. Context Propagation
Context propagation guarantees that trace and span data are propagated across service boundaries. OpenTelemetry has built-in support for W3C Trace Context, which standardizes the passing of tracing metadata between services.
3. Tracing and Spans
A trace has several spans where each span equals a single unit of work performed in a service. OpenTelemetry allows engineers to build a custom span and add metadata so that there’s greater visibility.
4. Exporters and Backends
OpenTelemetry enables trace data export to other observability tools, including:
- Jaeger: Traces distributed calls and assists in analyzing performance problems.
- Zipkin: Gives insights about latency and service dependencies.
- Grafana Tempo: Interoperates with Prometheus for centralized observability.
Implementing OpenTelemetry in a Microservices Architecture
For a Node.js services, below are the following steps:
Step 1: Install OpenTelemetry SDKs
For a Node.js microservice, install the necessary packages:
npm install @opentelemetry/api @opentelemetry/auto-instrumentations-node
@opentelemetry/exporter-trace-otlp-grpc @opentelemetry/sdk-node
@opentelemetry/instrumentation-http
@opentelemetry/instrumentation-express
Step 2: Configure OpenTelemetry Tracing
Create a tracing.js file to set up tracing:
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const traceExporter = new OTLPTraceExporter({
url: '<YOUR_OTLP_EXPORTER_URL>',
});
const sdk = new NodeSDK({
traceExporter,
instrumentations: getNodeAutoInstrumentations(),
resource: new Resource({
[SemanticResourceAttributes.SERVICE_NAME]: '<YOUR_SERVICE_NAME>',
}),
});
sdk.start();
Step 3: Instrument Your Application
Modify your Express.js server to include tracing:
const express = require('express');
require('./tracing'); // Import tracing setup
const app = express();
app.get('/', (req, res) => {
res.send('Hello, OpenTelemetry!');
});
app.listen(3000, () => {
console.log('Server running on port 3000');
});
OpenTelemetry Integration with Kafka
Kafka is widely used in microservices for event-driven communication. OpenTelemetry provides instrumentation for Kafka producers and consumers to track message flow across services.
Node.js Implementation
Step 1: Install Required Packages
Install packages for the kafka , for node-rdkafka install the below package with the previously discussed packages.
npm install opentelemetry-instrumentation-node-rdkafka
Step 2: Configure OpenTelemetry for Kafka
Create a tracing.js file:
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { NodeRdKafkaInstrumentation } = require('opentelemetry-instrumentation-node-rdkafka');
const traceExporter = new OTLPTraceExporter({
url: '<YOUR_OTLP_EXPORTER_URL>',
});
const sdk = new NodeSDK({
traceExporter,
instrumentations: [...getNodeAutoInstrumentations(), new NodeRdKafkaInstrumentation({
// Producer and consumer hooks can be used for setting custom properties to the span
producerHook: (span, topic, message) => {
const parsedMessage = JSON.parse(message.value.toString());
span.setAttribute('kafka.producer.message', JSON.stringify(parsedMessage.properties));
span.setAttribute('kafka.producer.event', parsedMessage.event);
},
consumerHook: (span, topic, message) => {
const parsedMessage = JSON.parse(message.value.toString());
span.setAttribute('kafka.consumer.message', JSON.stringify(parsedMessage.properties));
span.setAttribute('kafka.consumer.event', parsedMessage.event);
}
})],
resource: new Resource({
[SemanticResourceAttributes.SERVICE_NAME]: '<YOUR_SERVICE_NAME>',
}),
});
sdk.start();
Step 3: Instrument Your Application
Modify your Express.js server to include tracing:
const express = require('express');
require('./tracing'); // Import tracing setup
const app = express();
app.get('/', (req, res) => {
res.send('Hello, OpenTelemetry!');
});
app.listen(3000, () => {
console.log('Server running on port 3000');
});
Conclusion
OpenTelemetry simplifies the tracking and understanding of how requests travel through various microservices. OpenTelemetry gives a common means of gathering and analyzing trace information, allowing teams to view what is going on within their systems. With OpenTelemetry, developers can identify bottlenecks, improve performance, and debug issues more easily.
