Wednesday, October 12, 2011

Anatomy of a good bug report

by Tim Heuer via Method ~ of ~ failed by Tim Heuer on 10/12/2011 4:43:04 PM

I was searching my archives for sending something to my team this morning after looking at various bugs logged from customers, internal partner teams and ourselves.  I had an old post from 2005 but it unfortunately used a Shrinkster URL that appears not to be working.  So here’s the mail I sent to my team this morning.  This is purely my opinion but things that I think make a great bug report for fastest resolution:

------

As we triage and such I know that there are likely times that you are frustrated with not enough detail to even determine if this is a bug or needs investigation. This leads to feature teams spending time doing unnecessary things getting more data…and the cycle of time eventually delays product quality. To that end, when *we* log bugs I’d ask that you take into account my opinion of the “anatomy of a good bug”:

• Title: Required. This is obvious. The title should be concise but describe the problem in one line so that someone can quickly glance at it and get a feel of the issue immediately. Something like “control doesn’t work” is not helpful. Something like “TreeView shows blank nodes when binding to hierarchical data” is much more descriptive, yet concise.
• Path: Required. Almost every bug system requires and area/path assignment. You should do your best to assign this to where you think appropriate. If you are crystal clear, select the specific path. If you aren’t, then don’t guess too deep. Putting something at Product\Feature-Team\Feature-Area\Functional-Area\Control-Name might be too “deep” of a guess and would get bounced around. When in doubt, get up to the feature level and let triage teams who likely know the right path put it deeper or re-assign.
• Keywords: Optional. This is helpful for you usually. If there is a place to set keywords, you should consider putting some that might be helpful for searchers or yourself. However don’t spackle keywords all over titles/descriptions.
• Assigned: This usually defaults in bug systems to something like “Active” or “Not Assigned” – unless you know the specific person it should be assigned to, don’t guess.
• Priority/Severity: Optional. This usually defaults in systems as well. If you think the default is not correct for your issue, then change it. But if you are changing it to a higher (read: must fix ASAP) then you owe it to someone to send an email out. Logging high-pri bugs without an email is usually not a good idea.
• Build/Branch/Version: Required. Systems call these things different, and sometimes they are individually separated as well. However, this should be seen as required information. When someone looks at a bug they should know on what version of software you are using. They may immediately know “oh that was version 4.12, cool I know we fixed that in 4.2” and add that information to you back. Absent that information the investigator is left guessing on which build/version you were using and they may be flailing for a while with various versions. Even if there are fields for this in the system I usually always put this at the top of the repro steps as well for greater visibility.
• Description: Required. Yes you put a title, but you still owe it to the investigator/triage to provide a bit more depth. The description should re-iterate the title, but also provide the scenario and other supporting detail. Things like “it worked in version XYZ” or “works on competitor platform Z and adds significant value to customer” are some additional data points that help understanding why you are logging it. Descriptions maintain history of the bug so pasting emails for context, etc. are entirely appropriate IMO.
  • NOTE: I used an example of “worked on version XYZ” – some systems have specific indicators whether this is a Regression or not. Even if they do I would indicate regression data in the description.
• Repro Steps: Required. Perhaps the absolute most critical portion of your bug log. IMO this should include:
  • Version/Build/Branch
  • Steps – verbatim – yes I usually write these out with numbered lists:
    • 1. Open VS
    • 2. Create new C# Application
    • 3. Open designer and write this code
    • …
  • Once you hit the issue at hand break into Expected/Actual statements. This is where you would note what you Expected (i.e., “Control would render on UI surface”) and what was the Actual (i.e., “UnhandledException of type foo shown and crashed VS”).  In the Actual section would be a good place to paste call stacks, exception messages, etc. in addition to any simple error seen.

Here’s a template I use for the “repro steps” area of a bug report to remind me:

Repro on Version XYZ

Repro Steps:

1. Step 1
2. Step 2

Expected:

What did you expect to happen

Actual:

What actually happened

I find this has been helpful to continue to remind me of what is required and helpful.  Most systems allow you to create bug templates.  I have a bug template that includes this primer data in the repro steps for me.

• Sample Repro: Required. In my opinion, this is required. There is no greater path to resolution or understanding of a given bug then by having your specific issue immediately available. Without it the repro steps are helpful but may not be detailed enough in the code that you are using as well. This sample repro should be isolated to the issue at hand and should be simple to get up and running. If you send me a zip of your ERP system and expect me to fix why text is right-aligned incorrectly, that’s a colossal waste of my time getting a huge app set up trying to find the one issue. Often this exercise of narrowing the repro also identifies some issues where it might be user code. Regardless, provide a repro and keep it simple.
  • Sometimes attaching screenshots of what you are seeing is appropriate as well to help with an issue that might be a visual one and leads to faster understanding than setting up the repro.
• Close Bugs: When bugs are resolved back to you as fixed do your diligence to verify your bug quickly and close it out. Historically up to 15% of resolved bugs are re-activated. If we re-activate too late, we run the risk of missing key quality points in that area where you logged the bug. If you cannot verify it (because of time, out on vacation, etc.) then reach out to a test lead and indicate you cannot verify soon enough and request assistance to re-assign the Resolved bug to close.

Following these principles helps drive quality into our products. It does this by leading to faster times to get to understood, agreed-upon bugs. Please make sure we help our own team and partner teams when we log bugs to ensure we’re providing good information to help them fix what we think are issues.

------

I hope this helps perhaps as some guiding principles when you log bugs against products that you use and/or within your own team.