Legacy:A Bug's Life
I originally posted this article in our internal Jailbreak testing forums and moved it on the Wiki at a second thought. It's generically phrased, so it should apply to almost every project that's in development.
For another and rather entertaining look on the same subject, have a look at Painless Bug Tracking by Joel Spolsky. One of the most important things Joel has to teach to us techies is that you have to make documents fun to read to get people to actually read them.
Instead of listing a set of rules how bug reports should be posted, I'm going to describe A Bug's Life here. After all, all rules I could possibly post boil down to a certain understanding of which information is necessary (and why) in order to squish a bug.
What's a Bug?
First off, whether something is a bug is a mere matter of definition. That fragged players get respawned in jail is a distinct feature of Jailbreak, but for common Deathmatch it'd be a bug. (Yes, that's a bit far-fetched, but you get the idea.) Generally speaking, if a program behaves in a way the developer intended it not to, that's a bug.
There are a number of things where you can safely assume that a developer did not intend them to happen:
- Crashes.
- Things that obviously disagree with existing documentation. – That could just as well be an error in the documentation, but that'd be a bug in its own right.
- Things that are needlessly inconvenient for the user. – On the other hand, they could also be necessarily inconvenient for the user, but then that should be made obvious to prevent frustration on the users' part. Have a look at the Modding Etiquette about that.
If you find such a thing, post a bug report. If you find something where you're not sure whether it's a bug or intentional behavior, ask the developer; chances are that it is a bug anyway, and if it's not it should at least be changed to make that clear.
A Bug's Life Phases
After this short intro to what a bug is, let's have a look at the bug's life.
- A bug is born somewhere on the way between the developer's brain and the fingers typing the code.
- Sometimes a bug's cause is as simple as a typo in the code (when the coder lacks coffee, or sleep, or both, or had too much coffee, or something like that).
- Sometimes a coder thought she'd understood a particular piece of code, and how she could get her own code to work with it effectively. Unfortunately she is subtly mistaken (which could even be due to another bug in that existing code which naturally isn't documented anywhere).
- Sometimes one part of her own code puts the program in a state some other part of her own code doesn't expect and handle; that's the most annoying and most avoidable kind of bug for a decent programmer.
- Now that the bug has crept into the code, it stays there until it is discovered by chance by somebody. More often than not that's in the middle of a complex situation that is impossible to reproduce in every detail, and the next time the program is executed and the situation is slightly different, the bug manages to hide again.
- But the bug's days are numbered since its existence has been given away already. Of course, just like with real-life bugs, the coder needs to see the bug in order to fix it. So the bug hunter tries to recall as much as possible of the situation that caused the bug to appear the first time and the events preceding it and then successively strips it of everything unnecessary. Ideally the bug hunter ends this process with a short and simple list of steps that reliably lead to the bug showing itself.
- Now it's the coder's turn to squish the bug. She traces the program's execution from the start until she finds the point where a piece of code doesn't change the program's state as she meant it to. Then she tries to trace the program's flow back to the cause of that behavior. She changes it, and the bug is fixed.
Bug Hunters
You, as bug hunters, are responsible for the second and third phase in a bug's hopefully short lifespan.
Just discovering the bug by chance is not enough; given that there are more bug hunters than developers working on a project, leaving the third step up to the developers is a big waste of project time.
Imagine a small project with one coder and three testers. For the sake of the argument, let's assume that the third phase (reproduction) takes one hour per bug, and the fourth phase (fixing) half an hour (which may or may not be way off in either direction depending on the bug). Let's also assume that every tester discovers a single distinct bug.
- If all testers stop working at the second phase (discovery) and let the developer handle the third and fourth phase (reproduction and fixing), we have: a single coder working one and a half hours on each of three bugs, resulting in four and a half hours of work.
- If all testers continue working until the third phase is completed (discovery and reproduction) and then let the developer fix the bug, we have: three bug hunters working in parallel on three bugs for one hour, then the developer fixing each of the three bugs for half an hour each, resulting in two and a half hours of work.
Do you notice how the second way to do it actually saves two hours of development time in this example?
Reporting Bugs
When you report a bug, include all information that's necessary for the developers to reliably reproduce it. And just as important: Leave unnecessary information away.
Things every bug report should contain in some way or another:
- A brief, but descriptive bug report subject (a thread subject if you post it in a forum, for instance) for your bug report that clearly summarizes the reported bug. As a rule of thumb, a descriptive subject always contains at a verb. The program does something wrong after all.
- A thorough, but to-the-point description of the buggy behavior, including a short description of how you expect the program to behave instead. Screenshots, logs and the like can be helpful there.
- A thorough, but to-the-point description of how to reproduce the buggy behavior which you worked out in the third phase of a bug's life as described above.
- The revision of the program you're reporting the bug for.
- An up-to-date description of your system (hardware, operating system version and patch level, drivers, and so on). You don't have to repeat the same information over and over again, of course, but the developers should always be able to look it up if they need to. (For example, the developers could ask all testers to submit their system specs at the start of the testing period.)
Comments
Mychaeel: I originally posted that in the internal "Jailbreak Testing" forum, but knowing the Wiki as a great sounding board for articles like this one I decided to move it here and link it from our testing forum instead. Comments are appreciated.
EntropicLqd: Nice article. Is it worth mentioning that the number of bugs in any software system generally decays exponentially? How about adding a suggested set of minimum information that should be submitted with a bug report (game version, mod version, log files, screen shots if applicable etc).
Mychaeel: I expressly did not want to give a minimum set of information to post with a bug report in terms of "map name, log, screen shots" and so on. It only leads people to ticking those points off one by one without thinking about whether they're actually giving the developers helpful information or not.
Tarquin: That's sort of ... zen :D
Piglet: Hmm. Good and intresting section, however you have seemed to have neglected the 'immediatly obvious' type of bug. Which are generally more common (at least in the programming that i do). Not a big complaint, just a minor point.
Mychaeel: Hmm. The really obvious things should actually never get into testing; and I would rather not tempt testers to skip bug research by simply claiming that it's "obvious." My stance is that, if a bug was obvious, it wouldn't be there for others to find in the first place. I assume that programmers do a minimum amount of testing themselves before releasing a piece of software to anyone.
