There's so much more to debugging than logging

Hi there ,

A few months ago one of my followers on twitter (X?) reached out to me to ask for some help figuring out an issue with their code. After exchanging a few messages, I suggested we have a call instead.

After some exchange of pleasantries (I am British, after all; it's essential that every conversation begins with a five-minute discussion about the weather), I asked him to show me the issue he was seeing. I was shocked at what happened next.

He added a bunch of logs to his application at various points and ran his application. The application had some concurrency involved (it was a web app) and so the ordering was a little unpredictable and hard to follow.

Me: “Why don’t we try setting a breakpoint on line 132”.

Him: “What’s a breakpoint?”.

Me: “You can just click in the gutter on line 132 and start the debugger”.

Him: “What’s the debugger?”

We spent the rest of the session learning the basics of the debugger and he was able to resolve his own issue (there was a value missing on a configuration struct).

This session really stuck with me because it made me reflect on my own ability to debug, and how I had learnt the things I had. None of it was taught to me at University or specifically on any training I had ever done; it was always learnt opportunistically in a situation like I described above. I believe being able to debug locally and in production is a critical skill for any engineer, so maybe we should make more effort to teach it. This encounter led to me creating The Ultimate Guide to Debugging With Go course.

If you are looking to get better at debugging, below are some topics to explore with a short explanation of why I think they are important. Each of the topics below is covered in more detail in my course, so if you want to learn more about them, or like my free content and want to support me, consider checking it out.

The Debugger

It will be no surprise from the story I have told that using the debugger is a skill I think every engineer should learn. Surprisingly (to me), this view is somewhat controversial; I wrote about that here.

If you do want to learn about the debugger, I find the easiest way to start is to use the IntelliJ IDEs (Goland, php storm, IDEA, etc). The Jetbrains site contains detailed guides for debugging for all of their IDEs. For most languages I have tried (except php) the experience is similar and easy to get going with.

Logging

It might seem from the story I shared that I am not a fan of logging. That couldn't be further from the truth. I do believe logging should be part of a bigger observability strategy, though.

Structured logging can really help make your logs useful, and I recommend using a logger that supports it. In Go you can use Slog.

Making a system easy to debug is a journey, and logging is an easy and understandable place to start. You may find you use logging less and less as you increase your observability. I have more to say on logging strategies and I will be writing further blogs about this.

Metrics

After you have a good logging strategy, it's time to start thinking about some pretty dashboards to show business metrics. Prometheus and Grafana seem to be the de-facto standard here, and I really like the look of this beginner’s guidewhich talks you through where these might be useful in a language agnostic fashion.

Prometheus publishes clients for some popular languages including Golang.

Tracing

Debugging a system isn’t always about finding a logic error, sometimes it can be about discovering inefficiencies that can impact user experience. Tracing is a great way to figure out where your system is spending the time.

Tracing is not just for distributed systems either, even in monolithic applications, instrumenting critical sections and communication with dependencies (such as your database) can be very valuable.

You have tons of options for doing this at various price points. Some of my favorite tools are Sentry, datadog, Jaeger and New Relic.

Wrapping Up

I hope this helps you start out on your journey to become better at debugging. If you have a good understanding of the above (as well as unit testing) you will be in a really strong position to be an excellent contributor at any company you might end up at.

If you enjoyed this blog and are interested in diving deeper into the topic covered here (plus more) in a lot more detail, consider purchasing my debugging course or debugging book.

You can click the links above to get 30% off, or use code NEWSLETTER1430

Happy Debugging!

- Matt