Building Custom Distributed Tracing Mechanism

Introduction
In today’s world of microservices environment we know how critical it is to have a tracing in place. We have many options today to implement distributed tracing in modern applications and they are very straightforward. But there are a lot of applications out there running in production servers since years which are legacy in nature. Companies are either facing issues or they don’t have priorities or budget to migrate such applications to a brand new tech stack and utilise all the new features it provides.
So, I will cover how we can implement such distributed tracing systems in such legacy apps. I am assuming here that applications are running in production and already have a log aggregation mechanism implemented.
Let’s get started!
Architecture
Let’s go through the architecture now.
Implementation
Usually applications get requests from API Gateway which sits between API consumer and application. Requests are routed by the Gateway to application.
Now, before this request is served by your application, we need to do some task. So, we will need a request filter. Capture the request in the filter and generate a new GUID and put that into a request header lets say
X-correlation-id=27118570-0bf7-474b-a10d-ce74d525a954
Before generating this new GUID, first it will check if the request already has this header present or not. If its present then just ignore and move forward. This way we will be preserving the old correlation id.
public class CustomTracingFilter implements Filter { ... @Override public void doFilter( ServletRequest request, ServletResponse response, FilterChain chain) { HttpServletRequest httpServletRequest = (HttpServletRequest) request; HttpServletResponse httpServletResponse = (HttpServletResponse) response; HttpHeaders headers = request.getHeaders(); if(httpServletRequest.getHeader(“X-correlation-id”) == null){ String id = “service-name-”.concat(UUID.randomUUID().toString()); MDC.put(“correlation-id”, id); headers.add(“X-correlation-id”, id); chain.doFilter(request, response); } }
Now we have the correlation id required for tracing, we need that to be logged in every log that is getting generated so that we can query over it.
To achieve this, we will just put correlation in MDC and update our log format to print this correlation id.
Slf4j and log4j APIs provides this MDC interface. In the filter you can put the correlation id in MDC
MDC.put("correlationId", correlationId);
In your logback.xml you can use this variable
<Configuration status="INFO"> <Appenders> <Console name="stdout" target="SYSTEM_OUT"> <PatternLayout pattern="%-4r [%t] %5p %c{1} - %m - correlationId=%X{correlationId}%n" /> </Console> </Appenders> <Loggers> <Logger name="com.aost.log4j2" level="TRACE" /> <AsyncRoot level="DEBUG"> <AppenderRef ref="stdout" /> </AsyncRoot> </Loggers> </Configuration>
Conclusion
That’s it. You have successfully configured correlation id in your system.
Right here is the perfect site for everyone who wants to find out about this topic. You realize so much its almost hard to argue with you (not that I really will need toÖHaHa). You definitely put a brand new spin on a topic which has been written about for decades. Great stuff, just wonderful!
Thank you for your kind words!