The Hidden Economy of Bugs

In many software development environments, there exists an underlying economy of bugs fueled by misguided incentives.

The Origins of the Cobra Effect

During the British colonization of India, the authorities faced a significant threat from poisonous cobras, capable of killing in under thirty minutes. To address this danger, they initiated a bounty program, promising one hundred rupees for every dead cobra brought in, hoping to eliminate the problem quickly. However, this initiative backfired, as individuals began to see a business opportunity. Rather than decreasing the cobra population, they started breeding cobras to profit from the bounty multiple times. When the British recognized that this approach was exacerbating the issue, they ended the program, prompting people to release their captive cobras. Consequently, the cobra population surged, illustrating what is now termed the "cobra effect," or the phenomenon of perverse incentives.

Meeting Arbitrary Deadlines in Software Development

I first witnessed this effect in action about a decade ago while working on a software project nearing the end of a sprint. I had just completed a feature, but another large task remained. When I informed my manager of the impossibility of delivering on time, he unexpectedly instructed me to create and push a placeholder file, allowing QA to find the bugs. At that moment, I realized that I could meet any deadline by delivering a sufficient number of bugs.

This insight echoed a sentiment shared by Joel Spolsky, who advocated for fixing bugs before writing new code, reflecting on his experiences with the Microsoft Word team, where developers relied on QA to expedite their work.

“The story goes that one programmer, who had to write the code to calculate the height of a line of text, simply wrote ‘return 12;’ and waited for the bug report to come in about how his function is not always correct.” — Joel Spolsky

Incentives for Developers

Consider the mindset of a developer. They know that QA will inevitably find bugs, regardless of severity. If no critical issues arise, QA will still look for minor flaws, as the human tendency is to compare. Consequently, developers may feel incentivized to rush their work, knowing that QA will catch any mistakes.

Note: If you seek a metric to assess your team’s quality improvement, avoid measuring the number of bugs, as not all bugs carry the same weight. Instead, focus on the time taken to resolve all issues.

Unfortunately, this can lead to developers intentionally cutting corners, generating numerous minor bugs that are hard to detect. As a result, they may be perceived as the fastest and most effective programmers, while the responsibility for fixing these issues falls on QA.

Thus, developers are effectively incentivized to produce bugs.

Incentives for the QA Team

On the flip side, we have the QA team. How do we measure their effectiveness? Common metrics include the severity of bugs found in production, the number of automated tests, and the total defect count. However, the severity of a bug is often subjective; for instance, a color mismatch might be considered severe by some. The quantity of automated tests is not a reliable measure either, as a high number of tests might not cover diverse scenarios. Lastly, the total number of defects is an arbitrary metric frequently used.

QA teams tend to feel accomplished when they uncover numerous defects. Picture a QA engineer receiving a new build and finding a hundred bugs. If only a handful of issues arise post-release, their performance might be questioned. Conversely, if many bugs are found before release, they are viewed more favorably. Thus, QA teams are incentivized to discover more bugs.

The Impact of Perverse Incentives in Software Development

This dynamic creates a perfect storm, leading to what can be termed a Defect Black Market. When QA acts as a safety net for developers, programmers are motivated to produce more bugs while QA is incentivized to find them. This situation results in an organizational culture that encourages defect creation.

“No word needs to be spoken. No agreement needs to be written. But both sides understand that they benefit from defects. A black market economy of defects emerges. This disease permeates many organizations, and if it’s not terminal, it’s certainly debilitating.” — Robert C. Martin

Finding a Solution

So, how do we address this issue?

The root cause lies in the safety net established for developers. It is imperative that developers deliver code free of defects, and QA should find nothing. QA must not be the final checkpoint. If developers believe there is a safety net to catch their mistakes, they will not prioritize quality in their coding.

“Inspection does not improve the quality, nor guarantee quality. Inspection is too late. The quality, good or bad, is already in the product.” As Harold F. Dodge stated, “You cannot inspect quality into a product.” — John Hunter, author of Management Matters: Building Enterprise Capability.

How can we ensure developers produce bug-free code? This is where Test-Driven Development (TDD) and the shift-left approach come into play.

QA should be involved before developers begin coding, not afterward. They cannot serve as a safety net. However, how can QA assess quality without the final product? This is the brilliance of TDD; it mandates that tests be created before the actual code, allowing for consideration of relevant use cases prior to development. While initial requirements come from the Product team, QA should also anticipate additional scenarios. Instead of conducting manual tests post-development, QA should outline cases in advance, allowing developers to create automated tests.

The New Incentive Structure

I urge you to pause and consider the implications of having QA write tests before code is developed. Firstly, it would become easier to gauge QA effectiveness; if they can foresee all edge cases and potential misunderstandings, they demonstrate competence. Any bug discovered post-delivery indicates a failure on QA's part. Secondly, this collaboration allows QA to engage with Product to agree on test cases, ensuring relevant scenarios are prioritized. Lastly, developers would have complete knowledge of all test cases before beginning their work, eliminating the need for rework due to misinterpretations.

As a result, programmers would be responsible for creating the tests, understanding that there is no safety net. They would translate human-written tests into code, ensuring reliability through TDD. Once all tests are implemented, and the code runs correctly, the job is complete. No further testing would be required, resulting in a finished product devoid of trivial bugs.

This approach, when executed effectively through TDD, can eradicate the defect black market stemming from perverse incentives.

Thanks for reading. I enjoy sharing stories that provoke thought about our understanding and application of software engineering, and how we can improve our practices. If you found this article insightful, explore my other popular stories on Medium. If you're already a member, thank you for your support; if not, consider supporting my writing and others through this Medium referral link.