The ability to quickly and effectively find and resolve bugs in new and established systems is one of the most valuable engineering skills that you can develop. Since this skill enables the rapid development and maintenance of high-quality engineered systems, it is foundational for many technology companies, and is one of their most valued and sought-after skills. Nevertheless, this skill is rarely evaluated in coding interviews, and is often poorly understood and documented.
I have debugged and resolved many difficult problems in many different types of systems, including massively complex computer processors, multi-threaded servers and apps, and troubled individuals, families, and organizations. The optimal procedure for finding and fixing bugs is essentially the same across all domains. Surprisingly, many software engineers do not have a clear understanding of the process. I intend to address this deficit now. Here is my treatise on debugging.
Step 1: Determine what is working
When something is not working as expected, it’s easy to assume that everything is broken. Take the time to find the things that are working in the realm of the problem. This will help to circumscribe the problem and create a clear picture in your mind of its edges.
Step 2: Determine precisely what is not working
In the process of determining what is working, you will catalog a set of operations or behaviors that do not work. Spend time fleshing-out this list. Be clear on precisely how the system is not working as expected. Even though it might seem like these first steps are pointless because the problem is “obvious,” often jumping into solving the problem too soon can lead to wasted time and effort, and to a partial or non-optimal solution.
Step 3: Simplify the problem
Problem behavior that is discovered in a complex situation can be hard to reproduce or generalize, especially when there are non-deterministic or statistical effects. Any attempt to simplify the test case whilst retaining the problematic behavior is always time well spent.
For example, if the problematic behavior occurs when processing a very large dataset, you may want to try to reproduce the problem with increasingly smaller datasets. Of course, this example would not be possible if the problem is related to large datasets. In that case, creating a simple though still large dataset might make more sense.
By incrementally paring-down the situation where the problem arises, you not only increase your clarity about precisely what does and doesn’t work, but you also naturally start to construct hypotheses about what might be causing the problem.
Simple test cases are useful to precisely communicate the bug with others, to quickly test whether changes affect the bug, and may also become part of your anti-regression tests (see step 7). Since simple test cases can usually be run quickly, they also support hypothesis testing (see step 5).
Step 4: Generate hypotheses
You might arrive at this point after minutes, hours, days, or even weeks of work. No matter how you got here, or how long it took, you will now have data, and you will have learned something about the way that the problem manifests. This knowledge enables you to form hypotheses about what might be causing the problem. These are theories about what process inside (or even outside) the system might be leading to the observed problematic behavior.
Step 5: Test hypotheses using divide and conquer
Taking each hypothesis in turn, dive into the system and find a sub-unit where you believe that something may be going wrong. Then run your small test-case and look at the internal behavior before and after that sub-unit. If you find a problem before that sub-unit, then your hypothesis may have been wrong, and you at least know that you need to investigate further back towards the input of the system. If, on the other hand, the input to that part of the system seems correct, but the output seems incorrect, then you have support for your hypothesis, and you can zoom-in more closely.
At this point, if you are not fully clear on what the bug is, then loop back to step 1 on this identified sub-unit.
It’s possible at this point to apply divide and conquer very naively: split the system arbitrarily into two halves, look for a problem in each half, and then recursively zoom-in on the non-functional half. I don’t recommend this approach because it is usually very slow and cumbersome.
On the other hand, it’s possible to save a lot of time and effort by using hypothesis-driven divide and conquer, as described above. You still check whether behavior is as expected just before the sub-unit that is hypothesized to be broken, but, if things are functional there, you go straight to the output of that sub-unit. This enables very rapid zooming-in on the bug.
Only proceed to the next step once you’re clear about what the bug is.
Step 6: Think of other versions of this class of bug
Sometimes bugs are caused by simple typos, or one-off misunderstandings, and these kinds of bugs can just be fixed in isolation. However, it’s much more common for bugs to be representative of a much larger class of problems.
After spending the time and effort to get to this step, you will usually have an incredibly clear perception of the relevant parts of the system and of the problem. You will be the world-class expert on this bug. For this reason, now is the time to leverage all of that knowledge. In a month, you will no longer have this clarity of perception with respect to this specific problem.
So spend time now to fully leverage your investment. Think about and document the overall class of bug, and determine if the system will likely manifest other expressions of the underlying issues, whether or not those particular expressions have been manifesting for users.
We don’t want to stick a band-aid on a malignant tumor and send the patient home.
Step 7: Generate anti-regression tests
Even if you don’t design systems using test-driven development, I recommend that you use test-driven bug fixing.
Make sure to write unit-level and/or system-level tests that exercise as much of the bug class as possible. Make sure that the tests that you expect to fail do in fact fail. The main reason that the bug exists at all is that there were no tests to catch it. This means that there was a hole in the test suite. I often say that if something is not tested then it’s broken. This is because you have to assume that it’s either broken now or that it will get broken at some point in the future, and then the first person to discover that it’s broken will be a customer.
Since you have a broken system right now, now is a perfect opportunity to develop tests and ensure that they fail. These opportunities don’t arise often, so grasp them while they are available.
I like to call regression tests anti-regression tests, because they prevent the product from regressing to an earlier, broken state. Run your test suite with all of your tests before releasing new revisions of your product.
Step 8: Fix the bug(s)
If you have been diligent, fixing the bug(s) will now be extremely easy; it’s just a formality.
This kind of bug fixing can be performed very calmly and confidently. The fix is wrapped in a high-quality software engineering process, a process that informs and tests it. In contrast, I have witnessed engineers operating at the opposite end of the scale, changing code almost randomly in the hope that it will fix the overall problem. That kind of approach is more likely to introduce new bugs than to fix the existing ones.
While fixing the bugs, you might notice other problems. In that case, also loop back to an earlier step, such as step 6.
Step 9: Check that the tests now work
All of the new tests should now pass. If they don’t pass then you’ll need to loop back to an earlier step and resolve the issue.
Step 10: Check the original simple case
At this point, it should be possible to run the simple test cases that you developed in step 3, and they should be working properly. If not, then loop back to an earlier step to resolve the issue.
Step 11: Check the original issue
You should now be able to perform the behaviors originally reported to be problematic, and you should no longer see an issue. If you do see an issue, then return to an earlier step to resolve it.
Step 12: Document the fix
You have just performed an extremely high-quality set of engineering maneuvers. This is the stuff that legends are made of. It’s possible that you are the only person who is aware of your heroic actions. Write them down so that they can become a part of engineering lore. Document code, document the test-plan, document the test suite, write a wiki page or a blog post. Do something to capture the wisdom that you have developed and to make it available for others. Your documentation will also educate and mentor others. You will be setting a good example to other engineers, an example of both how to use resources effectively and efficiently, and also of how to execute challenging engineering work in a way that is deeply satisfying and nourishing to the soul.
Step 13: Note any other possible bug classes
During the time that you have been focusing your attention on resolving this particular issue, you may have noticed other potential classes of bug, and possibly other manifest classes of bug. File bug reports for issues that are manifest in dysfunctional behavior, or that you’re certain are lurking undetected. For other possible classes of bugs that may not be present but may also not currently be tested for, take whatever action is necessary to direct testing effort towards them. For example, you might update a test-plan ideas document.
Step 14: Release
Release your fix, either internally or externally, and make sure that everyone knows what you did. Summarize the problem and the solution succinctly, and include links to the documentation that you created.
Conclusion
You just did some awesome, high-quality engineering. Pat yourself on the back and head off to do something else that’s outstanding.