How to write a good testcase

What is a testcase?

A testcase is a single small source code listing demonstrating your problem. Give us a good testcase, and we'll see if we can help you solve/understand your problem.

What is a good testcase?

A good testcase meets the requirements from the checklist below. The quality of your testcase matters: a good testcase will get you helped quickly and effectively, a bad testcase will get you ignored or, at best, lectured. The points below are not absolutes, but the more rules you break, the more you discourage people from helping you.

Checklist

  1. Use a suitable pasting mechanism

    Pasting more than a single line of code in the channel is a mortal sin. Putting a raw source file on a webserver can make it hard to talk about line numbers. The preferred mechanism is to use a paste site. Recommended paste sites change from time to time as services come and go, so ask in the channel which site is currently favoured. If for some reason you prefer to use another paste site, please make sure that it at least meets the following requirements:

  2. Make sure your pasted code is indented properly and has no other pasting artifacts

    When using paste sites or similar mechanisms, sometimes you end up with a paste in which indentation is lost or empty lines are stripped (or even added!). These things make the paste hard to read. Check to see if this is the case for your paste, and fix if necessary.

  3. Describe the problem - code alone is not enough!

    A simple source listing accompanied by the phrase "it does not work" doesn't tell us what your problem actually is.

    If your problem is a compiler error (or warning), copy-paste it into the source as a comment either at the end of the code or, if the snippet is very small, at the location that the error/warning refers to.

    If the problem is a runtime error, ideally it should either be in the form of a program crash or a "logic error" (which means that the program produced output different from what you expected). In case of a program crash (segmentation fault, assertion failure, unexpected exception, etc), use a debugger to find out on which line the crash occurs and note it as above. In case of a logic error, explicitly mention (in a comment) what output you expected and what output the program actually produced.

  4. Make sure any line numbers you mention in the testcase are correct

    This should be an obvious requirement; incorrect line numbers are misleading.

  5. Use very high warning levels [1], a checked standard library implementation [2], and debuggers [3]

    People in ##iso-c++ help others because they enjoy sharing their C++ skills, knowledge, experience, and resourcefulness. There is no skill involved or experience required for diagnosing problems simply by compiling the code with higher warning levels, running it through a debugger, or using a checked standard library implementation. This is grunt work that you should do yourself.

    Of course, if you encounter a compiler warning or debugger message that you do not know how to interpret or fix, then by all means dedicate your testcase to this and we'll be happy to help (unless it's a very debugger-software-specific thing, in which case it's more appropriate to find help channels specific to that debugger software).

  6. Make sure your testcase is self-contained and actually reproduces the problem

    A testcase is said to reproduce the problem if, when we independently try to compile and run the testcase locally, we get the same compiler error (or our compiler's version of it) or erroneous program output specified in the testcase (see point 3). A testcase that does not reproduce the problem is useless: if we get different errors or output (or no output at all), then the code apparently does not correspond to the actual code you're having problems with, and consequently analyzing it is pointless.

    A testcase consisting of randomly copy&pasted bits of code that you thought were relevant can obviously not reproduce the problem. Worse, such testcases force your perception of the problem upon us, preventing us from taking a fresh look at it and making an unbiased analysis.

    For us to be able to compile and run the testcase locally, it must be self-contained, meaning that it must not have any external dependencies: it must not refer to classes/functions/variables defined elsewhere, nor must it #include headers other than those of the C++ standard library. If you are making a testcase to demonstrate a problem you encountered while developing a nontrivial program, this implies that you will have to isolate it (see the next point).

  7. Isolate the problem as much as possible

    The larger the testcase, the harder it is to analyze. This is especially true when the analyst is not familiar with the program that the testcase was extracted from. Therefore, it is very important that the testcase be as small as possible: every function/variable/class/template/etc that does not directly contribute to the problem should be cut. Isolation is generally by far the most time-consuming process in making a testcase.

    Fortunately, isolation is a skill that can be learned like any other, and that you will become proficient in over time. Furthermore, it is a skill that you will find tremendously useful; systematic isolation virtually guarantees that a problem can ultimately be analyzed. As you gradually become better at isolating problems, you'll find that increasingly often you will be able to isolate the problem to only a few lines in which the problem has become entirely obvious.

    See http://jcatki.no-ip.org/fncpp/TestCase for a practical demonstration of turning original code into an isolated testcase.

    Experience shows that the vast majority of problems people have can be shown in less than 15 lines of code. As a rule of thumb, many of us will flat out ignore testcases larger than, say, 50 lines of code.

  8. Remember the scope of the channel

    ##iso-c++ is a topical channel focusing on standard C++ — the language and its standard library. We are not interested in problems relating to the API's of specific operating systems or libraries. See the ##iso-c++ Policy.

  9. Use English comments and identifiers.

    For comments, this should be an obvious requirement.

    For identifiers, the choice of whether to use single-letter identifiers or more descriptive identifiers is entirely up to you. For many testcases, the former are actually quite okay. If however you decide to use descriptive identifiers, then we will assume that they were necessary to express the intent of the code, and will be put off if we cannot understand them because of the language in which they are written.


Footnotes

  1. For gcc, the following make a good set of warning/error related-flags:

    -std=c++98 -pedantic-errors -Wall -Werror -Wfatal-errors -Wwrite-strings -ftrapv -fno-merge-constants -fno-nonansi-builtins -fno-gnu-keywords -fstrict-aliasing
  2. The standard library implementation used by gcc, libstdc++, features an extremely useful debug mode.

  3. On *nix, GDB and Valgrind are invaluable tools.