Testing with b0 test

Specifying and listing tests

A test is a b0 unit tagged with B0_meta.test and B0_meta.run. Long tests can be tagged with B0_meta.long tag, this prevents them from running by default on b0 test.

Tests are labelled by a T in b0 list invocations, they can be filtered with:

b0 list --tests

Those that are in green are executed by default on b0 test.

Running tests

Tests are run with the b0 test command. The operating principle of b0 test is simple:

1. It makes a build exactly like a b0 invocation does.
2. It executes the action of any tests that must build in that build.

For example the following invocations build the default pack and execute the tests therein:

b0 test
b0 test -l      # Do not skip the long tests

If any test exits with a non-zero error code, the b0 test invocation eventually exits with 1.

Adding option --what to a b0 test invocation reports what is build and which tests are executed.

b0 test --what  # Show what builds and runs

For b0 the testing granularity is the executable unit. You can easily devise your own test sets via packs and the usual build selection options -u, -p, -x and -X. For example:

b0 test -p special-tests -x known_to_fail

Finally since tests are just executable units you can also run them individually with the usual b0 invocation:

b0 -- mytest …

This can be useful if the testing framework used by mytest provides command line options that cannot be specified by using the generic b0 test interface. For example to run tests with a finer granularity.

Random and snapshot testing

The b0 test command supports an environment variable interface for random and snapshot testing. By invoking b0 test with:

b0 test --seed 123    # Set SEED=123 while running tests
b0 test --correct     # Set CORRECT=true while running tests

The environment variables SEED and/or CORRECT are set for running the test executables. What the tests do with these environment variables is left to them but here is the expected semantics:

• If SEED is set to n, the executable should use n to seed the pseudorandom number generator (PRNG) used for randomized testing.
• If CORRECT is set to true, the executable should correct failed snapshot tests to the values computed during the run.

Testing OCaml code

The infrastructure described above allows to hook the testing framework of your wish by simply tagging your test executables with B0_meta.test and B0_meta.run and, if appropriate, looking up the SEED and CORRECT environment variables.

b0 itself provides a few testing tools to make such executables for OCaml:

• B0_testing in the b0.std library is a basic testing framework with support for unit, random and snapshot testing and integrates nicely with b0 test's rendering.
• FIXME get rid of it in favor of B0_testing. B0_expect in the b0.kit library provides snapshot testing. It is expected to change a little bit in order to integrate better with b0 test and B0_testing.