B0_testing cookbookA few recipes and starting blueprints for using B0_testing.
Note. Some of the code snippets here assume they are done after:
open B0_std
open B0_testingIf you are in a testing hurry or migrating tests you can always simply use OCaml's assert in the test functions: you just won't get any rendering on assertion failures and tests stop at the first assertion failure.
In general the following is suggested:
test directory at the root of your projectM call the main source file test_m.mlM call the main source file test_mytool.ml or test_mytool_tool.ml if it clashes with the name of a module.If you are testing the function M.f in a test call the test "M.f". If you are testing multiple function in a single test use glob-like syntax, for example for f and g use "M.{f,g}".
For external snapshot files it is suggested to:
test/snapshotsB0_testing.Snap.run to use .run for the file extension.Tests often entail comparing a computed or "found" value against an expected or reference one. The convention is to specify first the found value and then the expected one. For example if testing the sin function:
Test.float (sin 0.) 0.There is no choice if you snapshot values as the snapshot correction mecanism enforces the reference snapshot to be on the second position.
Test.snap (sin 0.) @> __POS_OF__ 0.Just devise a name for it and call B0_testing.Test.test.
let test_my_test =
Test.test "my test" @@ fun () -> ()The test name gets logged on execution and can be used to select or exclude them via the command line see How do I run only certain tests?
Tests are let bound functions of type unit -> unit. So you can call them directly to execute them. However usually this is done automatically via a call to B0_testing.Test.autorun in the test executable main function, see the minimal blueprint.
The B0_testing.Test.pass and B0_testing.Test.fail can be called to pass and fail tests. Note that B0_testing.Test.fail does not stop the test use B0_testing.Test.failstop if you want to stop the test abrubtly.
let test_pass =
Test.test "pass" @@ fun () ->
Test.pass ()
let test_fail =
Test.test "fail" @@ fun () ->
Test.fail "Oh no!";
Test.Log.msg "But let's continue";
()Basic assertions for testing equality of basic types are in the B0_testing.Test module. They can be used as follows:
let test_values =
Test.test "values" @@ fun () ->
Test.int (1 + 1) 2 ~__POS__;
Test.(list T.int) (List.map succ [1;2]) [2;3] ~__POS__;
()The function B0_testing.Test.eq allows to compare a value of an arbitrary type against an expected value of that type. You need to provide a tester B0_testing.Test.T.t for the value. Often you can using a type's module directly:
let test_fpath =
Test.test "fpath" @@ fun () ->
Test.eq (module Fpath) Fpath.null Fpath.null ~__POS__;
()Assuming you understood how testing a value works, snapshoting is the same except the value you test against can be automatically corrected.
let snap_values =
Test.test "snap values" @@ fun () ->
Snap.int (1 + 1) @> __POS_OF__ 2;
Snap.(list T.int) (List.map succ [1;2]) @> __POS_OF__ [2;3];
()If you don't want to specify the first snapshot just use a simple element for the type and let the machinery correct it for you. For example:
let snap_list =
Test.test "snap list" @@ fun () ->
Snap.(list T.int) (List.map succ [1;2]) @> __POS_OF__ [];
()Now running the test executable with option -c will correct the empty list to the result of the computation.
Use the B0_testing.(!@) operator with the file path for the snapshot. Relative files are relative to the test directory.
let test_snap =
Test.test "snap" @@ fun () ->
Snap.string "hey" !@ (Fpath.v "snapshots/hey.string");
()For now only snapshots of type string can be stored in files.
The B0_testing.Snap.run combinator does that. It snapshots the exit status, stdin and stderr. If you snapshot to a relative file it is relative to the test directory.
let test_echo =
Test.test "echo" @@ fun () ->
Snap.run Cmd.(tool "echo" % "Hey") !@ (Fpath.v "snapshots/echo-hey.run");
Snap.run Cmd.(tool "echo" % "Hey!") @> __POS_OF__
{|exited:0
┌─ stdout:5
Hey!
┌─ stderr:0
|}
If you don't care about the exit status and stderr, B0_testing.Snap.stdout only snapshots the standard output.
TODO need to streamline this, see jsont, cmarkit and idea would be to augment the PATH with a directory which has the executables in the build.
Use the B0_testing.Test.Log module.
let test_log =
Test.test "log" @@ fun () ->
Test.Log.msg "All is good";
Test.pass ()Anything can be used in a test function. However to ensure file paths are looked up correctly in every contexts in which the test executable may be invoked, read and write persistent data relative to the absolute path given by B0_testing.Test.dir. See also What is the test directory?
If some tests are defined in a file you can read the file and pass them to the tests as an argument and execute the tests in a B0_testing.Test.block. This is a better strategy than defining tests dynamically (especially if there are thousands of them). Assuming the tests are loaded in the tests arguments the test below does that. To see how you can add a command line option to specify the external file and load it see this blueprint.
let tests = Test.Arg.make ()
let file_tests =
Test.test' tests "file tests" @@ fun tests ->
Test.block ~kind:"file test" (fun () -> List.iter (run_test cmd) tests)If you are using Result values and you are get trapped in the error just use Test.error_to_fail and Test.error_to_failstop.
let test =
Test.test "munge file" @@ fun () ->
Test.error_to_failstop @@
let* text = Os.File.read Fpath.(Test.dir () / "data.txt") in
Test.string text "Hey!" ~__POS__;
Ok ()It depends but if the setup is going to be shared among tests then it's better to do it in the main function and communicate the result to tests via test arguments.
Note that the main function can be made to Test.failstop like mentioned in How do I deal with test setup errors? to deal with setup errors. See for example this blueprint.
A test can depend on arguments. In order to do so you need to declare the argument with B0_testing.Test.Arg.make, then declare your test with B0_testing.Test.test', and the argument value must be supplied in the in the main procedure. This leads to the following structure:
let limit = Test.Arg.make ()
let test =
Test.test' limit "Try to break it" @@ fun limit ->
for i = 1 to limit do () done
let main () =
Test.main @@ fun () ->
Test.autorun () ~args:Test.Arg.[value limit 10]
let () = if !Sys.interactive then () else exit (main ())To add more command line arguments to the test executable use the the B0_testing.Test.main'. See this blueprint for an example.
A long test is a test that you don't want to execute on every test run because it's too costly.
Use the corresponding option in Test.test.
Long tests are not executed by default. Invoke the test executable with -l to run them.
The support is minimal at the moment. There is just infrastructure for seeding and reproducing B0_testing.Test.Rand.state.
Tests can be selected by the name you specify in B0_testing.Test.test calls.
The -x and -i options of the test executable can respectively be used to exclude or include tests by name. Invoke the test executable with --list to list the names. Completion should also work if you have it setup. Invoke the test executable with --help for more information.
That depens on you how run the test. In general it's better to avoid relying on the current working directory of the test executable. Use the B0_testing.Test.dir path to make the file paths you want to read or write absolute. See also What is the test directory?
For the test executable it is better to rely on B0_testing.Test.dir to resolve relative file paths. This ensure everything will work fine regardless of the executable's current working directory.
However it may be useful to set the cwd on processes spawned by the test executable for example by B0_testing.Snap.run.
On example is snapshoting outputs which contains text file location. In this case you will want to set the cwd of the invocation and pass it relative file paths so that the snapshot of text locations does not mention absolute paths.
The test directory is an absolute path according to which the relative paths in your test executable should be resolved. It is notably used by snapshot testing for storing and updating the reference snapshots in sources and external files.
The test directory should be set by the test runner with the TEST_DIR environment variable or the --test-dir option of your test executable. For snapshot tests it is important for the test directory to be setup correctly (usually to the parent directory of the executable source file). Read Test directory to understand how b0 sets this variable on unit action executions and b0 test.
The test directory can be used for example to load test data and is available by calling B0_testing.Test.dir.
Since the tests are let bound you can easily invoke them in the toplevel individually. At least those that do not have arguments.
A minimal example.
open B0_testing
let test =
Test.test "nothing" @@ fun () ->
Test.pass ()
let main () = Test.main @@ fun () -> Test.autorun ()
let () = if !Sys.interactive then () else exit (main ())This adds an additional --data-file argument to the test executable. The data file is loaded and its data made available in a test via a test argument.
open B0_std
open Result.Syntax
open B0_testing
let data = Test.Arg.make ()
let test =
Test.test' data "nothing" @@ fun data ->
Test.pass ()
(* Command line interface *)
open Cmdliner
open Cmdliner.Term.Syntax
let main () =
Test.main' @@
let+ data_file =
let doc = "$(docv) is the data file to use." in
let default = Fpath.v "data.json" in
Arg.(value & opt B0_std_cli.filepath default & info ["data-file"] ~doc)
in
fun () ->
let file = Fpath.(Test.dir () // data_file) in
let* data = Os.File.read data |> Test.error_to_failstop in
Test.autorun ~args:Test.Arg.[value data_file tests] ()
let () = if !Sys.interactive then () else exit (main ())