Beyond system.out.println: Applying a Debugging Mindset at Scale

Abstract

Debugging a consistently failing unit test is easy; you set a breakpoint and fix it. But how do you debug a "random" failure in a massive microservices landscape or a distributed monolith?

During an interview, I was once told, "You are more of a debugger than a coder." They were right. I thrive on the bugs that make others give up the nasty, intermittent issues hidden under the carpet.
But you cannot solve these issues with 'System.out.println()' or a simple debugger attached to localhost.

Debugging at scale requires a shift in mindset and tooling. It requires moving from the paradigm of "5 Whys" and 'logger.info' to a structured observational approach based on observability data.
Based on production outages, I'll share how I uncovered the problems, the lessons I learned and how you can combine a debugging mindset with observability tooling like OpenTelemetry and the Grafana toolset to gain deeper insights.