I was working with a customer recently that develops software for use by their clients. They work in a DevOps style flow, regularly completing work and sending that to their clients. In one case, they released a hot fix quickly to resolve a bug in their software. In this case, the developers received the report, triage'd the bug, and released the fix in less than a week. Their clients were happy with the response and speed at which the fix was delivered.

Unfortunately, they also released another bug with the fix. The new bug was in an unrelated area of the software, created by separate changes to the code. Now they had a dilemma. The hot fix helped some customers, but the fix also broke some things that other customers cared about.

What do you do here? If you were in charge of this software team, how do you decide what to do? You can pull the patch, but that means some clients might have broken software. You can leave the patch out there, but then you risk other customers being upset by a new bug.

There isn't a good solution here, and the more clients you have, the more likely that you have a large split of customers wanting version 1 versus those wanting version 2. You could rebuild the hotfix to remove the new bug, or you could make another patch that just addresses the second bug. The latter idea is what DevOps would suggest, with a roll forward to a third version of the software. The idea being you keep improving things and fixing bugs as you continue to develop new features. You will release some bugs, but you can fix them quickly.

The downside of this is that if you release too many bugs, customers are loathe to upgrade. When this happens, you start to find yourself having to support many versions of your software. If you respond to customers with special patches that only fix bugs without adding features, you may end up with many more versions than you can track, which can be a real nightmare for developers to manage.

How do you deal with patches causing bugs? I recently re-read a piece by Eric Sink in which he recounts releasing three maintenance versions to his customers in a week. Embarrassing for a version control vendor. It's a good (but long) read, and one that might give you some perspective. We can't squash every bug, and we can't test for everything, but we can work to continue to improve, chasing the dream of bug free software. Or at least, regression bug free software.