b0 manual

Quick start

Conceptual overview

Build model

Effectively a b0 build system is an OCaml program that executes arbitrary external commands in parallel and whose effects on the file system are memoized across program runs with an on-disk cache.

There is no notion of build rule in b0: you simply generate and execute program commands using arbitrary OCaml functions. This allows to define modular and rich data structures for describing builds that are "compiled" down, on each build, to parallel invocations of memoized commands.

Configuration

Next to this simple build model b0 adds a configuration mecanism under the form of a typed, persisted, key-value store which builds can consult and depend on.

Since outputs from previous builds are kept in the cache, build configurations can be switched over and back almost instantaneously without loosing the earlier CPU cycles.

The configuration layer is also cross-compilation ready: any configuration key value can differ for the build and host operating system and the build system of b0 keeps tracks of build tools that are build and used by the build system to make sure they are built with the build OS toolchain. For programmers of the build system, cross compilation is oblivious in b0, it happens without the user having to perform anything special. More on configuration.

Build variants

The basic build library and model allows build operations to act anywhere and can be used as such. However to structure the builds, the notion of build variant is added on top of that. Build variant allow builds with different configurations to live next to each other or be performed in containers or on remote machines. They define a basic directory layout in which the build occurs and setup the build environment in which the configuration occurs and build tools are looked up. More on variants.

Deployments

Deployments abstract the general process of extracting part of the sources and/or build artefacts of your software to a new location. Examples of deployments are: installing build artefacts in a system (FIXME unclear), pushing build artefacts to a remote server or device, making source or binary distribution tarballs and pushing them to a location, interacting with package manager repositories.

More on deployments.

The b0 and d0 tools

The b0 and d0 tool allow to build projects that are described by writing one or more B0.ml OCaml files in a source tree or a composition thereof.

More on description files.

A tour of the _b0 directory

Generally the layout of the build directory is as follows:

• _b0/cache, holds the build cache.
• _b0/defaults, holds description defaults.
• _b0/i, holds the root description and compiled driver instances.
• _b0/v, path to build variants.
• _b0/d, path to deployments.

The structure of a build variant n is as follows:

• _b0/v/n/scheme, holds the variant's scheme name.
• _b0/v/n/outcome, holds the variant's build outcome (if any).
• _b0/v/n/conf, holds the variant's configuration (if any).
• _b0/v/n/trash, holds the build variant unit trash (if any).
• _b0/v/n/index, holds the variant's cache index (if any).
• _b0/v/n/proxy, path to data related to a proxy build (if any).
• _b0/v/n/b, path to the variant's build directory
• _b0/v/n/b/u1, holds the build of unit u1 of the build variant.
• _b0/v/n/b/u2, holds the build of unit u2 of the build variant.

The structure of a deployment n is as follows:

• _b0/d/n/scheme, holds the deployment scheme name.
• _b0/d/n/conf, holds the deployment configuration (if any).
• _b0/d/n/s, holds the deployment's staging directory.

Configuration

A configuration is a set of typed key-value bindings consulted by descriptions and build procedures to adjust their outcomes to the build environment and desires of the end user.

A configuration key the user did not explicitely set has a default value, specified in the description at key creation time. This value is either constant or discovered via a function.

A key can belong to at most one group which is simply a named set of related keys. Groups are used to easily select a subset of keys for user interaction. For example on b0 key get command invocations, using the -g ocaml option will report the value of all configuration keys that declared themselves to be part of the ocaml group.

Configuration presets are named sets of key-value bindings defined in descriptions. They are a convenience to set key subsets in bulk in configurations.

Last and stored configuration

The configuration used by the last build is persisted in the build outcome and called the last configuration. It is immutable and contains only the key-value pairs of the configuration that were accessed by the last build. It can be accessed via the b0 key get --last command.

The mutable stored configuration is the configuration to be used by the next build. It can be acted upon via the b0 key get and b0 key set commands.

Key value terminology

A configuration key has different values depending where and in which context it is looked up:

• Last value. The value used by the last build and stored in the last build outcome, if any.
• Stored value. The value from the stored configuration, if any.
• Default value. The value given at key definition time. This is either a constant or a value discovered by a function. The default value is used by a build if the key has no stored or environment value (see below). If the default value is discovered, the discovered value gets saved in the stored which is persisted at the end of the build to avoid repeated discovery; this behaviour may however be prevented by the default value definition.
• Environment value. The value found, if any, for a key named key in a B0_C_$KEY environment variable where $KEY is key uppercased and with '.' replaced by '_'. In a build this value overrides both the stored and default value of a key. It ends up defining the last value of a key but it doesn't get saved in the stored configuration.
• Effective value (for a build). The value used during a build: the environment value or if undefined, the stored value or if undefined, the default value.
• Preset value. The value of the key in a configuration preset. This is either a constant or a discovered value. It is used to set the stored value of a key to a user preference defined in the description.

During a build the effective value of keys is looked up using the stored configuration. As a side effect new key-value pairs may be added to the stored configuration for keys whose default value is used and discovered during the build. This modified stored configuration is persisted at the end of the build.

Build variants and variant schemes

A build variant is a build performed in a particular environment with a particular configuration.

The build environment is defined by a variant scheme which is responsible for setting up the environment for the variant. For example this can be: configuring and setting up the environment for an opam switch, spin a container or ssh to a remote machine. Build variants are identified by a name n which is used to operate on the variant. The build directory of a variant n is isolated from the others in _b0/v/n. Variants are created via:

b0 variant create [SCHEME] [-n NAME]

or implicitely on the first b0 build if there's no existing variant (see The initial variant). If you don't specify a variant name on creation a unique one will be automatically derived from the scheme name. If you don't specify a scheme, the default scheme (likely the nop scheme) will be used.

b0 allows variants to exist and be acted upon side by side, use b0 variant list to list them. Most b0 commands act on the variant specified explicitely via the -w or --variant argument or on the default variant as reported by b0 variant get. If there is no default variant or if it doesn't exist commands might error.

The nop variant scheme

The variant scheme Variant.Scheme.nop available under the name nop is the simplest variant scheme. It does nothing, it runs builds in the environment where the build tool b0 itself is run.

The default variant and variant schemes

The default variant can be consulted, set or cleared via:

b0 variant get [--effective | --stored]
b0 variant set [--clear | VARIANT]

If the B0_VARIANT environment variable is defined, it's value will define the default. The default variant is automatically set to a newly created variant this can be prevented with the -c option:

b0 variant create -c SCHEME  # Do not set the new variant as the default

The initial variant

If no variant exists and there is no default variant when b0 build (or equivalently b0) is run, a variant is created using the default variant scheme. So on a fresh checkout of your project:

b0

automatically creates a variant, set it as the default one and builds your project.

Description values

b0 descriptions are made of a grab bag of OCaml values, configuration keys, build units, packages, variants, variant schemes, deployments, etc. In order to operate on these values from end-user interfaces (e.g. the b0 and d0 tools), the following must be guaranteed:

1. Values and their names need to be defined during the toplevel initialization phase of the program without being conditioned by external factors b0 may not be aware of (FIXME implement Def locking).
2. Values names need to be unique to ensure all the values are accessible and can be operated on.

As far as 1. is concerned, b0 relies on the discpline of B0.ml file writers. They should define all their description values through toplevel let definitions and never conditionalize their existence or the definition of their components. For examples this should NOT be done:

let myprogram =
  (* NEVER DO THIS *)
  let default = Conf.const (if Sys.win32 then "bla.exe" else "blu") in
  Conf.(key "myprogram" ~default)

As far as 2. is concerned. b0 handles this automatically. In two manners:

• If a name used in a description clashes with a name defined by a library b0 logs a warning and automatically rename the new definition.
• When multiple B0.ml are composed toghether. The names defined by subdescriptions get automatically namespaced by their position in the file hierarchy. TODO Give example.

Deployments

Deployements are handled via the d0 tool. They do not necessarily need a build to exist but can request for builds of specific packages to exist. They occur through a sequence of steps, all of which are configurable and made available through deployment schemes.

1. Pre-stage check and build requirements.
2. Stage function, prepare deploy artefacts in the deployment staging directory.
3. Post-stage check.
4. Pre-push check.
5. Deployment push, push the staged artefacts.
6. Post-push check.

Descriptions files

A description file is either:

1. A B0.b0 file that describes how to compile a description.
2. A B0.ml OCaml file in a directory without a B0.b0 file.

If your description is simple or uses only the default b0 library then a simple B0.ml description will do. If not, a B0.b0 file is an s-expression based configuration file that describes how to compile a self-contained and isolated description. It can specify additional (and conditional) sources and libraries to use, compilation flags and control how subdescriptions (see below) are looked up.

Root description and directory

b0 supports file hierarchies that contain more than one description file. In general, to ease build setup understanding, it is better to keep a single description per project. However multiple descriptions allow to merge the description of multiple parallel and interdependent projects into a root description that is built in a root directory. We first explain formally how an invocation of b0 finds the root directory, examples follow. Given the root directory we can proceed to describe which descriptions belong to the root description. When started in a directory dir, b0, unless invoked with --root option, finds a root directory for the build as follows:

1. Starting with dir (included) and moving up in the hierarchy, find the first start directory with a description file (a B0.b0 or B0.ml file). If there is no such directory there is no root directory and no build description.

2. From start move to the parent directory up and:

   • If up has a description file and does not exclude start via the subs key of an up/B0.b0 file, let start be up and go to 2.
   • If there is no description in up or if it excludes start then start is the root directory.

Here's an example of a file hierarchy with multiple descriptions:

d
└── root
    ├── B0.b0
    ├── B0.ml
    ├── p1
    │   ├── B0.b0
    │   └── B0.ml
    ├── p2
    │   ├── B0.ml
    │   ├── hop
    │   │   └── B0.ml
    │   └── sub
    │       ├── a
    │       │   └── B0.ml
    │       └── b
    └── src
        ├── bin
        └── lib

In the example above starting a driver in d/root, d/root/src/bin, d/root/p1, d/root/p2/sub/b will all find the root directory d/root. However starting a driver in d/root/p2/sub/a will find the root directory d/root/p2/sub/a as there is no description in root/p2/sub. Adding an empty file d/root/p2/sub/B0.b0 would allow to find d/root.

Given a root directory with (a possibly empty) description, b0 gathers and merge the descriptions files of all direct subdirectories and recursively into the root description. The subs key of B0.b0 files can be used to control which direct subdirectories are looked up. The OCaml sources of different sub descriptions cannot refer to each other directly; they are properly isolated and linked in any (but deterministic) order.

Assuming no B0.b0 file makes use of the subs key in the above example, the root description in root takes into account all descriptions files except d/root/p2/sub/a/B0.ml. Here again adding an empty file d/root/p2/sub/B0.b0 would allow to take it into account.

B0.b0 description files

A B0.b0 description file is a possibly empty sequence of s-expressions of the form (key value). Here's an annoted example:

(b0-version 0)       ; Mandatory, except if the file is empty
(libs (b0_cmdliner)) ; Always compile with the external b0_cmdliner library
; Describe the sources that make up the description in dependency order.
; As a convention if you split your build in many build files put them
; in a B0.d/ directory. If the [srcs] key is absent and a B0.ml file
; exists next to the B0.b0 file it is always automatically added as if
; ("B0.ml" () "B0.ml file") was appended at the end of srcs.
(srcs
  ; If the source path has no suffix looks up both for an .ml and mli file
  ((B0.d/util () "Utility module")
   ; The following source needs the b0_jsoo library and is only added to
   ; the description if the library is found to be installed.
   (B0.d/with_jsoo.ml (b0_jsoo) "Description with jsoo support")))
(compile (-w -23)) ; Disable warning 23 for compiling the description

Key parsing and semantics

An B0.b0 file without keys and without a B0.ml file sitting next to it is an empty and valid description. If a key is defined more than once, the last one takes over; other than that the key order is irrelevant. Except for keys that start with x-, unknown keys trigger parse warnings.

Relative file paths. Relative file paths are relative to the description file directory.

Library lookup. FIXME. Library lookup is currently quite restricted and done according to the following name mapping:

• libname, $LIBDIR/libname/libname.cm[x]a
• libname.sub, $LIBDIR/libname/libname_sub.cm[x]a

With $LIBDIR being defined (first match) by:

1. The value of the environment variable B0_DRIVER_LIBDIR
2. The value of the environment variable OPAM_SWITCH_PREFIX post
3. The value of $(ocamlc -where)/..

Dependency resolution on the libraries is not performed and cmi files have to be in the corresponding libname directory.

Key reference

• (b0-version 0). File format version, mandatory, except if the description is empty.
• (libs (<libname>...)). Libraries unconditionally used to compile and link.
• (drop-libs (<libname>...)). Libraries dropped from compile and link (e.g. to drop b0 or B0_care).
• (srcs ((<path> (<libname>...) <docstring>)...)) OCaml source files to use for the description. Each source is described by a path to an ml file, libraries whose existence condition their usage and a documentation string describing their purpose. If <path> doesn't end with .ml assumes both <path>.ml and <path>.mli exist and are used. A B0.ml file sitting next to the B0.b0 is always automatically added at the end of the list.

• (subs (<op> (<dirname>...))). Subdescription lookup.

  • By default, if unspecified, all the subdirectories of the directory in which the B0.b0 resides that do not start with . or _ are looked up for descriptions.
  • If <op> is include only the specified list of directory names are looked up.
  • If <op> is exclude excludes the given list of directory names from the default lookup.
• (compile-kind <kind>) with <kind> one of byte, native, auto. The kind of binary to compile to. Allows to force the use of ocamlc or ocamlopt. Defaults to auto which selects native code if available and bytecode otherwise. Subdescriptions propagate their constraint to the root. Inconsistent compilation kind in subdescripitions lead to failure. Can be overriden with the `--d-compile-kind` option or by the B0_D_COMPILE_KIND environment variable.
• (b0-dir <path>). The b0 directoy to use. Can be overriden on the command line with the `--b0-dir` option or by the B0_DIR environment variable. Defaults to _b0.
• (driver-dir <path>). The driver directory to use. Can be overriden on the command line or by the B0_DRIVER_DIR environment variable.
• (compile (<arg>...)). Arguments added to byte and native compilation. More can be added on the command line via the `--d-compile` option.
• (compile-byte (<arg>...)). Arguments added to byte compilation.
• (compile-native (<arg>...)). Arguments added to native compilation.
• (link (<arg>...)). Arguments added to byte and native linking. More can be added on the command line via theh `--d-link` option.
• (link-byte (<arg>...)). Arguments added to byte linking.
• (link-native (<arg>...)). Arguments added to native linking.
• (ocamlfind <bin>). ocamlfind binary to use. Ignored in subdescriptions.
• (ocamlc <bin>). ocamlc binary to use. Ignored in subdescriptions. Can be overriden with the `--d-ocamlc` option or by the B0_D_OCAMLC environment variable.
• (ocamlopt <bin>). ocamlopt binary to use. Ignored in subdescriptions. Can be overriden with the `--d-ocamlopt` option or by the B0_D_OCAMLOPT environment variable.
• (x-<key> value). Arbitrary user defined key.

B0.b0 key merges

When multiple B0.b0 file are used, their specification is merged with the root description. During this process the key values of subdescriptions are either:

• Ignored (e.g. ocamlc, ocamlopt, etc.).
• Merged according to key specific strategies which can fail; one example of failure is when one subdescription mandates (compile-kind native) and another one (compile-kind byte).

More build concepts

Output command digests

Commands are assumed to be pure functions of their inputs and declared process environment.

Concatenate and digest: The digest of the executable, the command line arguments, the spawn process environment, the digest of the contents of inputs, the output path. This becomes the name of the file in the cache.

Cleaning build

In a cleaning run, outputs that are not rebuild and were present in the previous run are deleted at the end of the run.

Build correctness

If you spend some time thinking about building software incrementally and correctly you quickly realize that our current file system and tool based approach is entirely hopless. The fact that you can't guarantee noone fiddles with the outputs of your build steps across build system runs. b0 is not different and you can entirely trip it by fiddling with the contents of its _b0 dir.

Recipes and menagerie

Writing conf discovery

Error only if really needed. Otherwise log with warning and default to a reasonable deafult value. Build units can still abort if they can't use the value.

Old TODO.md

• Allow tool lookup to be an option Review what happens in build outcomes on failure.
• Build part cleaning
• Redo build finish reporting and cycle analysis, start from the parts, not from the ops !
• Tool version example.
• OCaml version example/scheme.
• Doc vs synopsis, e.g. for variants. Maybe not. Lookup in the API.
• `b0 env`, `b0 exec-env`
• Consider reording tool & conf. Also `Conf` became a mess make it more `Build`-like.
• Build need a way to know if we are a root unit or not.
• Serialization move away from Marshal, so that proxy build data can be read back.
• Find something forwards looking w.r.t relative paths specified to units. We don't want to be in the build fun I think. src_root which defaults to sub description location ?
• `Build.await_file` ? Basically `Build.read` but without the read.
• Add `unit path -b`
• Page `cmd` info, like we page `log`.
• copy_file handle permissions
• Add a flag to allow multiple writes.
• Reproducible builds. Should we force by default BUILD_PATH_PREFIX_MAP in the environment ? https://reproducible-builds.org/specs/build-path-prefix-map/
• https://hub.docker.com/r/ocaml/opam2-staging/tags/
• B0_ocaml, stub changes do not retrigger some compilation. Make library archive depend on stubs. May also be the reason for spurious failures in examples.
• b0 log -f -n -i --failure --reads --writes --file --pkg
• Distingish paths in `Cmd.t` ?

B0.Def

• Provide a way to lock definitions so that any def afterwards raises. This ensures that description files do not generate definitions dynamically (e.g. conf keys, build units, variants schemes).
• For b0 as a library. Provide a way to disable the retaining of definitions (Unit.list, Variant.Scheme.list). Related to locking.

B0.Conf

Discover should be `Build.t` based. This will give us caching/reacting to the environment for free and will allow to hide the build aim. Add a special build unit for it: this can give a directory to write things in aswell.

B0.Pkg

Should have a unit associated or maybe not it should be easy to sync on all the units of a package once we have the notion of package in units. This should be sufficient for `META` file generation and `opam` file sync.

`b0 build` improvements

• Implement `--c-$VAR val ` to set configuration values.
• Improve `-u` unit requests, in particular at the API level we need `Build.request : Build.t -> Unit.t -> unit`. Also add `--no-deps`, `--no-prev`.

• Incremental performance. Already pretty good but could be better:

  • Consider not re-hashing the cache elements, keep their hash in the build outcome. I.e. trust that noone fiddles with _b0/cache
  • Implement a fast check for the whole build on the roots. This is cheating but is useful for e.g. `b0 run` to quickly check and report the user is running outdated programs. It's also useful for projects like `odig` to see if the doc are up-to-date before showing them.
  • Implement per unit fast checks. This is less cheating, the parts that pass the check don't need to be trashed and relinked and we avoid computing all the commands.
  • Dolan. Make the cache read only.
  • Michel. Hash server using modern fs notification APIs. b0 simply consults the server.

`b0` driver

• Something is wrong with the error strategy lots of boiler plate. Also the variant access API is messy.
• `B0.b0` make `b0-version` really mandatory except on empty files.

B0.Tool

Provide support for checking minimal version. Or not.

Proxy variants and _b0

In `B0_docker` The current scheme means that we copy rather than link(2). This is due to the way we mount stuff and the business of excluding parts of `_b0`. The whole way of proxying needs revisiting so that we can simply mount the root dir without excluding `_b0`. This should not be a big deal, somehow we only need to deal with `_b0/i` so that the proxy instance can live there (e.g. setup `B0_DRIVER_DIR` and/or forward with `--d-dir`).

However we also want to keep the current style for e.g. `ssh` proxies where we don't want to rsync all the local variants.

Variants

• Do we want a configuration API ? Also should we really namespace the variants ? With the current scheme we end up with a lot of redundancy on project composition.
• Logging for setup operations.
• Add a suggested name to the Variant.Scheme API, naming a variant after the variant scheme may not work that well (but that may be due to 1.).
• Document/think about how a new scheme can be added e.g. to a whole opam switch. (Add a new package with the scheme, set environment variable `B0_D_LIBS`).
• Proxy results, they should be stored seperatly so that we don't try build them.
• Proxy we might want to access the namespaced name. E.g. for the docker image.
• Metadata on variants ? Could be useful for cross-variant deployements.
• Review direct functions signature, unify with proxy_conf ?
• need to distinguish b0 runs from build product runs.
• It is unclear whether proxy scheme should hard-code the proxied variant. Maybe this could show up and specified in the cli interface e.g. via `--proxy`.

Build context

A topkg-like build context is needed. It's a bit unclear where this should fit. Ponder the following alternative.

1. A configuration key in B0_care.
2. A value held by `Env` and thus defined by the variant scheme.

In any case this should be exposed in `Build.t` values.

Also maybe another name than build context should be found. Source context, run runner, goal.

In general the behaviour of b0 tries to be as independent from the operating context as possible. However some configuration keys and/or actions are sensitive to the what is called the build context. This makes the invocations to specify on the command line and in package descriptions (e.g. the build: field of opam files) terser and thus generally improves the DRYness of the tool.

More specifically b0 distinguishes between the following contexts:

• `Distrib iff not (vcs s). No VCS is present this is a build from a distribution. If there are configuration bits they should be setup according to the build environment.
• `Dev iff (vcs c) && not (dev_pkg c). This is a development build invoked manually in a source repository. The repository checkout should likely not be touched and configuration bits not be setup. This is happening for example if the developer is building the software in her working source repository by invoking b0.
• `Dev_pkg iff (vcs c) && (dev_pkg c). This is a package manager development build. In this case the repository checkout may need to be massaged into a pseudo-distribution for the software to be installed. This meas that distribution watermaking and massaging should be performed. See distribution description and the prepare_on_pin argument. Besides existing configuration bits should be setup according to the build environment.