The following is provided as a reference for those interested in understanding how the test suite works, and how to add and maintain tests. Overview of types of tests ========================== Darcs has tests in three formats. Unit tests that directly test Haskell functions are written in Haskell and live in src/unit.lhs. Functional tests that test the darcs binary are written in Shell and Perl, and live in tests/. Additionally, there are some functional tests which are expected to fail, kept in bugs/ and used for documenting bugs and todos. Haskell tests -------------------------- These are QuickCheck tests primarily testing the Darcs core. Shell tests --------------------------- Shell tests are useful because they are easy to create from a copy/paste of actual shell commands executed. They are considered successful if no bad exit codes are returned from the commands in the script. Perl tests --------------------------- The Perl tests are useful because of the featureful testing framework that Perl provides, and the potential they offer to create more portable tests. Right now both shell and perl tests are accepted by the project, but shell tests are preferred by the main developers, as they are generally more useful and easier to debug. See "The future of testing darcs" below. How to run tests ============================= "make unit" builds the unit tests, "./unit" runs them. They take a while. Output like "good" and "OK, passed 100 tests." is good. Output like "*** Gave up! Passed only n tests." is a shortage of quickcheck test cases, not a test failure. "make test" causes all the functional tests in "tests/" to be run against repositories in the old, hashed, and darcs2 formats. "make bugs" runs the scripts in "bugs/", also on all three types of repository, with the expectation that they currently fail. Because "make test" can take a long time to run, it's useful to run fewer tests at once. To help with that, the following make targets are also available: make test_shell # darcs1 old format make test_shell_hashed # darcs1 hashed format make test_shell_format2 # darcs2 hashed format make test_perl make test_perl_hashed make test_perl_format2 make bugs_shell make bugs_shell_hashed make bugs_shell_format2 make bugs_perl make bugs_perl_hashed make bugs_perl_format2 Running specific tests ------------------------------ Add test file names to tests_to_run to do only those tests: echo annotate.pl > tests/tests_to_run echo annotate.sh >> tests/tests_to_run make test_shell # or whichever you like! Running a single test ----------------------------- For consistent output, perl tests are usually run with the prove script: cd tests perl bin/prove annotate.pl and shell tests are run with the shell_harness script: perl shell_harness annotate.sh ("chmod +x bin/prove shell_harness" to use them directly.) Tips for writing (and reading) tests ==================================== - Avoid including a repo format type to "darcs init" This insures that all three repo formats will be tested. However, if you know that the test only passes under some repo formats, *do* explicity include a format option to "darcs init". Tips for writing Shell tests ------------------------------- - Copy an existing test, which will already have the following properties: - Simply call darcs using "darcs" as you would in the shell. It is the responsibility of the test harness to ensure that the darcs we are testing is first in the path. - Always use bash explicitly--this improves the portability of our tests. - Always add this near the top of the script: set -ev The "v" causes the contents of the script to be printed as part of the run, which is helpful for debugging. The "e" causes the script to exit as soon as there is an error. - Try to avoid defining functions where possible (see below) Tips for writing Perl tests ------------------------------- - Be aware of the testing modules that are available. See the docs for each of these, in order of importance. They are in the tests/lib/perl directory. (Use perldoc file.pm to see the docs). Test::More Test::Darcs Shell::Command File::Slurp - Avoid direct system calls. It will be tempting to use backticks to drop in direct shell commands. By writing as much of the tests as possible in pure Perl, they will remain more portable. The modules above cover the common cases. - ./bin/prove has many options for running the Perl tests. The future of testing darcs ================================= We would prefer to have all tests written in bash, as these tests are both easier to write (since you ordinarily run darcs from a shell in any case), and are more helpful when trying to track down bugs, for the same reason: it is easy to run bash tests by hand in a shell, examining each intermediate state. For this same reason, try to avoid defining functions in your test scripts. This makes them harder to run and generally harder to use. There are certainly cases where it is appropriate to define a function, but please do not do this just to avoid a little duplication.