Fixes hard to verify because of poor quality bug reports



  • Working with programmers and customers I find it is sometimes harder for me to verify the fix for a defect, than to fix a defect for a developer.

    Just a simple example is when there is NullPointerException thrown by a method. A bug report contains no steps to reproduce the issue, just a stacktrace. Given such a poor report, a developer corrects the code so the method swallows such exceptions or re-write the method. Then he marks the defect as fixed.

    As a tester I usually care more about the context. Why null values arrived to the method? Which component produced it and why? So simply swallowing the exception is not a solution: it probably did't removed the error. Rather it masked it.

    This kind of problems usually happens, when someone else reported a bug (not me), for instance a customer representative, and I see the bug report only when verifying the fix.

    So it all boils down to the quality of bug report. If steps to reproduce weren't written before fixing the bug, they must be identified after the fix. And then it is often hard to do it, because I need to check-out/setup two system versions from SVN: one before the fix, and another after the fix was committed.

    So how can we improve the process?

    1. Shouldn't testers and devs complete the bug report with steps to reproduce before fixing it? I don't care who will do it, but I think such a step should be present in the bug lifecycle here.
    2. If devs and testers cannot collect information from customer to reproduce the bug, should they patch it (as in the example above) or reject a bug report as "cannot reproduce"?

    EDIT: Maybe I wasn't clear enough. I haven't asked how to write a good bug report. I'm rather looking for practical ways to make devs, customer representative, business analysts, me, etc. complete a bug report as it goes through our hands. Some solutions other have proposed so far:

    • introduce templates in bug reports
    • review good and bad reports with stakeholders to demonstrate how they help or hinder reproducing bugs and fixing them
    • gently ask customer for more details (talking to a customer seems a separate art to me), particularly for what they did before the failure occurred.

    Particularly, I haven't seen any answer what a dev should provide in a bug report after fixing the bug? Root cause? What areas were touched? How were they changed? In which revision change was committed?



  • First, I would like to say that often just taking care of exception is enough. Maybe it's a forgotten corner case (sort a list, but forget it might be empty). Of course you know better when you can trust the developers to make sure that fix is correct.

    Then to your questions:

    1. It would be really nice to have the steps in each bug. It would be even better to have a test case reproducing the bug before fixing it. That said, I also write bug reports that don't have exact steps but do explain the situation as well as possible. And sometimes I have to file a bug even though I don't even know what it exactly was that broke the system. Of course in these cases I help the developers with whatever questions they might have.

      Maybe the developer doesn't know how to reproduce it exactly either, but does a code review for the broken part and finds some problems. It's fully possible to know that something would be problematic without knowing how to actually get fully running system to that exact state.

      If the problem clearly is in the communication and not in the difficulties of reproducing the problem, then I can't suggest other than telling everyone why writing a good bug report matters. Point out that there are potentially lot of people reading the case is widely different times (developer fixing the bug, tester testing the fix, technical support helping customers, other people reporting bugs, product owner and other shareholders trying to understand state of the product, ...).

    2. I don't think it's fruitful to just reject the bugs. It may easily sound arrogant and only hinder discussion with the reporter. Leave the bug open, but ask clarifying questions or steps to reproduce and still be prepared to fix the problem without ever getting these. If there clearly is a problem, it should be fixed even if the person pointing it out can't tell how they got it exactly. They have seen it anyway. Maybe they don't even know the product well enough to understand what it was that caused the problem or are just too busy to come back to it. That's no good, but different people have different priorities and bug reporting may not be near the top for some.

    All in all I think we often have to live with less information than we would like to have. Take pride in being able to work also in those conditions but keep pushing people to better process.


Log in to reply
 

Suggested Topics

  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2