5

I work primarily on small-scale (and large-scale) software projects. One of the questions that always needs answering is how do we decide how many bugs should we fix?

Realistically, software always has bugs. And these bugs are of different types; data corruption bugs are not the same as the-border-color-is-wrong bugs.

Some project management methodologies answer this question directly or indirectly; Six Sigma comes to mind (leave no more than one bug per one-point-something-million), as do Clean Room and Zero Defect methodologies (no bugs allowed, period).

But how do I decide how many bugs I should fix? What do I look at? Some metrics might be:

  • How many bugs do I have budget for fixing (fix as many as possible until launch)
  • How many customers are complaining about bugs (fix bugs that multiple people complain about)
  • Severity of bugs (fix data-corruption and software-halts bugs only)
  • ...

To summarize and clarify, I'm looking for a method or technique I can apply to come up with a set of guidelines on which bugs to fix and which to ignore.

Geo
  • 3,923
  • 1
  • 21
  • 33
ashes999
  • 5,318
  • 28
  • 43

9 Answers9

4

I always aim for zero.

While leaving cosmetic bugs in may seem acceptable, I would argue that they cost less to resolve than leave in. As long as they are left, they will take time to review every time someone goes through the bug list. They may also multiply through copy and paste. (Screen 1 has wrong border color. We copied it to build screens 5, 8, 12, and 17, now they have the wrong border color. Four new bugs to raise and fix.)

I might leave feature bugs in, especially if there is disagreement about the feature. (I worked on one project where we had a feature Monday, Wednesday, and Friday. It was taken out for Tuesday, Thursday, and the following Monday. What a productive week!!)

It tends to be cheaper to fix bugs as soon as possible. The responsible developer is likely still available and knows exactly how to fix it. Wait a couple months and the developer will likely have to spend more time figuring out what he did and how to fix it. Wait longer and the original developer may have moved on to another project or company. Also if developers spend too much time fixing their bugs they are likely to improve their quality or move on to greener pastures.

As others have mentioned, when the policy is not to fix bugs then workarounds are implemented. It costs time to implement and test the workaround. This cost will likely exceed the cost of fixing the bug. Even worse, the workarounds may break when the original bug is fixed, increasing the repair cost of the original bug.

Scope bugs should be left unfixed but should be documented as such or rejected. Make sure you can easily get scope bugs out of the regular bug review flow. (Bug: The clock radio doesn't turn on my coffee maker when the alarm goes off. Reject or ask for a scope change.) Man pages used to have a bugs section. The awk command had a documented bug that it didn't work for lines longer than 1024 characters.

BillThor
  • 1,566
  • 8
  • 7
3

I have spent time in countless meetings where more time was spent discussing a bug -- what is the bug, why is it there, can we just leave it in? -- than would have taken to just fix it. What if the team spent the energy they would have put into categorizing and ranking the bugs and put that energy into fixing the bugs?

There are times when you may decide to leave a bug in the product, but that should be rare. You should at least try to fix all the bugs in your product. Once you start leaving them in, it's a very slippery slope. One "acceptable" bug often begets others. Or the "we had to do it this way because of that bug we never fixed" effect on your overall design. Also, your developers will tend to be more sloppy once they know that some bugs are acceptable.

Marcie
  • 1,537
  • 12
  • 13
  • Not every bug is acceptable. See comment about "the border colour is wrong." I've seen bugs in enterprise system last years because this is how bugs are; some of them just never become high priority and deserving of a fix. – ashes999 Mar 02 '11 at 18:56
  • You mentioned you're a programmer that sometimes wears a PM hat. As a PM, your job isn't to fix bugs, as fun as it may sound. If we take that argument, then that means you should just do all the outstanding tasks in marketing, customer service, content writing, testing, as well as employee reviews the functional manager hasn't done yet. While you're at it, the plants need watering and the mail still needs to be sorted. As a PM, do you do your job or do you do everyone else's job? – jmort253 Mar 03 '11 at 03:34
  • One more thing to add: I do see your point that one can overthink things. That's equally bad. If the programmers are sitting around twiddling their fingers, then maybe they do just fix the bugs. But if they're really busy, then that buys you time for some analysis :) – jmort253 Mar 03 '11 at 03:59
  • @jmort, I didn't mean to imply that the PM should fix the bugs. I meant the team's energy could be better spent fixing bugs than categorizing and ranking, which I assume will require team meetings rather than having the PM do this alone. I'll update my answer :) – Marcie Mar 04 '11 at 17:18
  • Thanks for clarifying. I've definitely seen myself overthink things so I know what you mean. +1 – jmort253 Mar 05 '11 at 02:45
2

+1 for the comment: "you will always have bugs." - It does not matter the methodology, it is just plain impossible. Unless you are dealing with an extremely small team, with an extremely small application. Enough of the strong words.

What we do in our practice is the following:

  1. We finish dev and do our internal unit testing (in dev environment), after a few sprints of fixing those, we release the sw to QA team (another developer if you don't have QA personnel)
  2. Then we run test cases on the QA environment to find other defects.
  3. After a few iterations (depending on the project) you will get a few defects on each category (high, med, low).
  4. Get the QA person perspective in a document (QA Report, with go or no-go recommendation)
  5. Regardless of either go/no-go in the QA report. Take the report to the business(hopefully it will include the amount of defects left) and ask them if they feel comfortable with those bugs. If they say, yes let's go even if your QA say no, then you have your decision point.

Take the SW to UAT and you are done, until next time.

Geo
  • 3,923
  • 1
  • 21
  • 33
  • Too true - 100% elimination of bugs is impossible. You might be able to get rid of the ones you have logged, but there are always more out there. – Steven Mar 02 '11 at 18:05
  • To rephrase your answer, you're saying let the business side decide when defects fixed are sufficient; provide them with a QA report and get devs to fix easy bugs first. – ashes999 Mar 02 '11 at 18:32
  • @asshes999, I will agree with your statement, except for "fix easy bugs first". I will suggest to always focus on the business critical first, even if they are not easy. Sometimes, you are able to push some of those off, but not very likely. – Geo Mar 02 '11 at 19:35
  • Sorry, typo (from multitasking), that should be "fix bugs first" – ashes999 Mar 02 '11 at 19:58
  • The problem I have with letting the business side decide is that they don't know anything about details of bugs. They just see numbers, maybe severities. – ashes999 Mar 02 '11 at 20:06
2

Let me suggest an article written a year ago by our team: http://www.technoparkcorp.com/innovations/test-planning. The idea in the article is that there are two indicators: Quality of Implementation and Quality of Product.

Quality of Product (QoP) is how good your product is going to be, comparing to some ideal. To calculate it you need to know the anticipated size of the product in order to make an assumption of how many bugs you need to find. To find in order to reach "critical mass".

Quality of Implementation (QoI) is how good you managed to create the product so far, comparing to the planned QoP. This is where you are calculating the bugs actually found and fixed.

Having these two indicators in place you will be able to claim to your customer/sponsor, for example:

"Planned quality is 78%, while 55% of it is already achieved".
yegor256
  • 7,050
  • 4
  • 29
  • 50
  • Great article! I've seen the shops where the testers tried to dictate the direction of the project. It was 100% contrary to the goals of the organization. In short, your article sums it up best that quality is a judgement call by the PM. – jmort253 Mar 03 '11 at 04:18
  • @jmort253 indeed, testing as any other activity in a project, has to be managed by a project manager, and be included in WBS. – yegor256 Mar 03 '11 at 06:27
2

There's some additional info that would help with the response:

  • what are you 'contracted' to deliver?
  • is there a fixed budget that you CAN NOT exceed (seems to be by your question)
  • do you prioritize your bugs by impact and probability of occurrence
  • what if you don't fix ALL the bugs - what's the outcome? is there any liability?

Generally, you want to fix ALL known issues, you can never compromise on quality, there's a direct relationship between it and productivity. (Capers Jones, DeMarco, etc.)

Meade Rubenstein
  • 707
  • 3
  • 3
  • In this case, there is no contract or fixed budget; we prioritize bugs by impact; no liability if all the bugs are not fixed. – ashes999 Mar 02 '11 at 21:10
  • So - what is the impact if none of the bugs are fixed? There has to be some financial impact - otherwise when fix any in the sytem? – Meade Rubenstein Mar 03 '11 at 19:17
2

You can search for the "Good Enough Software" term. When I was in the university I did read a controversial book that talks about that, it was inspiring. I don't remember the name, but I trust in google can put you in the way. If anybody knows that book, please tell me the name.

Edit: It was an article from the IEEE Software journal (May 1995) called "When good enough software is best" by Edward Yourdon. Then, the "The Pragmatic Programmer" referenced to that one. Some link to the text: http://doi.ieeecomputersociety.org/10.1109/52.382191 http://portal.acm.org/citation.cfm?id=625475

Mauro Gagna
  • 21
  • 3
1

Number of unresolved bugs should be part of your acceptance criteria; the final word is always with your customer however. As a guideline we have the following classification:

  • Blocking: needs immediate resolution; number = 0 before go live, naturally
  • Workaround available: maximum 5 to 7 (but this depends upon the product!) and ofcourse need to be planned for the next update.
  • cosmetic (what you call "the border colour is wrong" type of bugs) - no limit

We classify ourselves, but the list is regularly discussed with the customer. Cosmetic bugs are resolved when time permits, prioritised by the customer. Now and then the list is cleaned out.

hope this helps

Stephan
  • 5,252
  • 1
  • 21
  • 31
1

I think looking at the number of bugs might be the wrong approach. I'm not saying it's a bad idea. But I want to point out two opposite albeit unlikely examples.

On one end of the spectrum, you could have 100 bugs that cripple the system, eat the client's data, sends all of the nasty emails in their Saved Drafts folder, burns their business to the ground, and urinates on it to try to put the fire out. These bugs are the nastiest of the nastiest bugs you've ever seen, and everything else -- other than that -- is perfect.

So do you fix all 100 or do you use some magic formula that tells you to just fix X of them?

On the other hand, you could have 1,000 "the-button-has-an-extra-pixel-in-it-omg!" type bugs that just don't matter to anyone except the designer who conceptualized the user interface. Do you again use the magic formula and go fix Y of them because that's what the formula says, or do you recognize that in all of your hallway usability tests, everyone loved the software, and not one person complained.

I believe that at some point, it's worth reading between the lines and using your judgement to determine what bugs affect the quality of the system, and your bottom line, and what bugs would just represent unnecessary costs.

If you're new to project management, a formula is a fine way to get started, but eventually after gaining some experience, you'll come to rely more on your judgement.

jmort253
  • 9,537
  • 7
  • 42
  • 88
1

The success of a deliverable will be defined by the quality of it. Its important to establish a process of assigning the severity of the bugs, normally by risk and impact, I don't think this should be overly complicated. Reverting back to a pre-defined acceptance criteria, the focus should be to clear all bugs of a certain severity (i.e. high, medium) and a percentage of the remaining. Ultimately, agreement with the stakeholders should guide your decision on what is acceptable or unacceptable.

Dave
  • 378
  • 1
  • 6