b0 manual

The old manual, kept for reference.

Quick start

Conceptual overview

Build model

Effectively a b0 build system is an OCaml program that executes arbitrary external commands in parallel and whose effects on the file system are memoized across program runs with an on-disk cache.

There is no notion of build rule in b0: you simply generate and execute program commands using arbitrary OCaml functions. This allows to define modular and rich data structures for describing builds that are "compiled" down, on each build, to parallel invocations of memoized commands.

Configuration

Next to this simple build model b0 adds a configuration mecanism under the form of a typed, persisted, key-value store which builds can consult and depend on.

Since outputs from previous builds are kept in the cache, build configurations can be switched over and back almost instantaneously without loosing the earlier CPU cycles.

The configuration layer is also cross-compilation ready: any configuration key value can differ for the build and host operating system and the build system of b0 keeps tracks of build tools that are build and used by the build system to make sure they are built with the build OS toolchain. For programmers of the build system, cross compilation is oblivious in b0, it happens without the user having to perform anything special. More on configuration.

Build variants

The basic build library and model allows build operations to act anywhere and can be used as such. However to structure the builds, the notion of build variant is added on top of that. Build variant allow builds with different configurations to live next to each other or be performed in containers or on remote machines. They define a basic directory layout in which the build occurs and setup the build environment in which the configuration occurs and build tools are looked up. More on variants.

Deployments

Deployments abstract the general process of extracting part of the sources and/or build artefacts of your software to a new location. Examples of deployments are: installing build artefacts in a system (FIXME unclear), pushing build artefacts to a remote server or device, making source or binary distribution tarballs and pushing them to a location, interacting with package manager repositories.

More on deployments.

The b0 and d0 tools

The b0 and d0 tool allow to build projects that are described by writing one or more B0.ml OCaml files in a source tree or a composition thereof.

More on description files.

A tour of the _b0 directory

Generally the layout of the build directory is as follows:

The structure of a build variant n is as follows:

The structure of a deployment n is as follows:

Configuration

A configuration is a set of typed key-value bindings consulted by descriptions and build procedures to adjust their outcomes to the build environment and desires of the end user.

A configuration key the user did not explicitely set has a default value, specified in the description at key creation time. This value is either constant or discovered via a function.

A key can belong to at most one group which is simply a named set of related keys. Groups are used to easily select a subset of keys for user interaction. For example on b0 key get command invocations, using the -g ocaml option will report the value of all configuration keys that declared themselves to be part of the ocaml group.

Configuration presets are named sets of key-value bindings defined in descriptions. They are a convenience to set key subsets in bulk in configurations.

Last and stored configuration

The configuration used by the last build is persisted in the build outcome and called the last configuration. It is immutable and contains only the key-value pairs of the configuration that were accessed by the last build. It can be accessed via the b0 key get --last command.

The mutable stored configuration is the configuration to be used by the next build. It can be acted upon via the b0 key get and b0 key set commands.

Key value terminology

A configuration key has different values depending where and in which context it is looked up:

During a build the effective value of keys is looked up using the stored configuration. As a side effect new key-value pairs may be added to the stored configuration for keys whose default value is used and discovered during the build. This modified stored configuration is persisted at the end of the build.

Build variants and variant schemes

A build variant is a build performed in a particular environment with a particular configuration.

The build environment is defined by a variant scheme which is responsible for setting up the environment for the variant. For example this can be: configuring and setting up the environment for an opam switch, spin a container or ssh to a remote machine. Build variants are identified by a name n which is used to operate on the variant. The build directory of a variant n is isolated from the others in _b0/v/n. Variants are created via:

b0 variant create [SCHEME] [-n NAME]

or implicitely on the first b0 build if there's no existing variant (see The initial variant). If you don't specify a variant name on creation a unique one will be automatically derived from the scheme name. If you don't specify a scheme, the default scheme (likely the nop scheme) will be used.

b0 allows variants to exist and be acted upon side by side, use b0 variant list to list them. Most b0 commands act on the variant specified explicitely via the -w or --variant argument or on the default variant as reported by b0 variant get. If there is no default variant or if it doesn't exist commands might error.

The nop variant scheme

The variant scheme Variant.Scheme.nop available under the name nop is the simplest variant scheme. It does nothing, it runs builds in the environment where the build tool b0 itself is run.

The default variant and variant schemes

The default variant can be consulted, set or cleared via:

b0 variant get [--effective | --stored]
b0 variant set [--clear | VARIANT]

If the B0_VARIANT environment variable is defined, it's value will define the default. The default variant is automatically set to a newly created variant this can be prevented with the -c option:

b0 variant create -c SCHEME  # Do not set the new variant as the default

The initial variant

If no variant exists and there is no default variant when b0 build (or equivalently b0) is run, a variant is created using the default variant scheme. So on a fresh checkout of your project:

b0

automatically creates a variant, set it as the default one and builds your project.

Description values

b0 descriptions are made of a grab bag of OCaml values, configuration keys, build units, packages, variants, variant schemes, deployments, etc. In order to operate on these values from end-user interfaces (e.g. the b0 and d0 tools), the following must be guaranteed:

  1. Values and their names need to be defined during the toplevel initialization phase of the program without being conditioned by external factors b0 may not be aware of (FIXME implement Def locking).
  2. Values names need to be unique to ensure all the values are accessible and can be operated on.

As far as 1. is concerned, b0 relies on the discpline of B0.ml file writers. They should define all their description values through toplevel let definitions and never conditionalize their existence or the definition of their components. For examples this should NOT be done:

let myprogram =
  (* NEVER DO THIS *)
  let default = Conf.const (if Sys.win32 then "bla.exe" else "blu") in
  Conf.(key "myprogram" ~default)

As far as 2. is concerned. b0 handles this automatically. In two manners:

Deployments

Deployements are handled via the d0 tool. They do not necessarily need a build to exist but can request for builds of specific packages to exist. They occur through a sequence of steps, all of which are configurable and made available through deployment schemes.

  1. Pre-stage check and build requirements.
  2. Stage function, prepare deploy artefacts in the deployment staging directory.
  3. Post-stage check.
  4. Pre-push check.
  5. Deployment push, push the staged artefacts.
  6. Post-push check.

Descriptions files

A description file is either:

  1. A B0.b0 file that describes how to compile a description.
  2. A B0.ml OCaml file in a directory without a B0.b0 file.

If your description is simple or uses only the default b0 library then a simple B0.ml description will do. If not, a B0.b0 file is an s-expression based configuration file that describes how to compile a self-contained and isolated description. It can specify additional (and conditional) sources and libraries to use, compilation flags and control how subdescriptions (see below) are looked up.

Root description and directory

b0 supports file hierarchies that contain more than one description file. In general, to ease build setup understanding, it is better to keep a single description per project. However multiple descriptions allow to merge the description of multiple parallel and interdependent projects into a root description that is built in a root directory. We first explain formally how an invocation of b0 finds the root directory, examples follow. Given the root directory we can proceed to describe which descriptions belong to the root description. When started in a directory dir, b0, unless invoked with --root option, finds a root directory for the build as follows:

  1. Starting with dir (included) and moving up in the hierarchy, find the first start directory with a description file (a B0.b0 or B0.ml file). If there is no such directory there is no root directory and no build description.
  2. From start move to the parent directory up and:

    • If up has a description file and does not exclude start via the subs key of an up/B0.b0 file, let start be up and go to 2.
    • If there is no description in up or if it excludes start then start is the root directory.

Here's an example of a file hierarchy with multiple descriptions:

d
└── root
    ├── B0.b0
    ├── B0.ml
    ├── p1
    │   ├── B0.b0
    │   └── B0.ml
    ├── p2
    │   ├── B0.ml
    │   ├── hop
    │   │   └── B0.ml
    │   └── sub
    │       ├── a
    │       │   └── B0.ml
    │       └── b
    └── src
        ├── bin
        └── lib

In the example above starting a driver in d/root, d/root/src/bin, d/root/p1, d/root/p2/sub/b will all find the root directory d/root. However starting a driver in d/root/p2/sub/a will find the root directory d/root/p2/sub/a as there is no description in root/p2/sub. Adding an empty file d/root/p2/sub/B0.b0 would allow to find d/root.

Given a root directory with (a possibly empty) description, b0 gathers and merge the descriptions files of all direct subdirectories and recursively into the root description. The subs key of B0.b0 files can be used to control which direct subdirectories are looked up. The OCaml sources of different sub descriptions cannot refer to each other directly; they are properly isolated and linked in any (but deterministic) order.

Assuming no B0.b0 file makes use of the subs key in the above example, the root description in root takes into account all descriptions files except d/root/p2/sub/a/B0.ml. Here again adding an empty file d/root/p2/sub/B0.b0 would allow to take it into account.

B0.b0 description files

A B0.b0 description file is a possibly empty sequence of s-expressions of the form (key value). Here's an annoted example:

(b0-version 0)       ; Mandatory, except if the file is empty
(libs (b0_cmdliner)) ; Always compile with the external b0_cmdliner library
; Describe the sources that make up the description in dependency order.
; As a convention if you split your build in many build files put them
; in a B0.d/ directory. If the [srcs] key is absent and a B0.ml file
; exists next to the B0.b0 file it is always automatically added as if
; ("B0.ml" () "B0.ml file") was appended at the end of srcs.
(srcs
  ; If the source path has no suffix looks up both for an .ml and mli file
  ((B0.d/util () "Utility module")
   ; The following source needs the b0_jsoo library and is only added to
   ; the description if the library is found to be installed.
   (B0.d/with_jsoo.ml (b0_jsoo) "Description with jsoo support")))
(compile (-w -23)) ; Disable warning 23 for compiling the description

Key parsing and semantics

An B0.b0 file without keys and without a B0.ml file sitting next to it is an empty and valid description. If a key is defined more than once, the last one takes over; other than that the key order is irrelevant. Except for keys that start with x-, unknown keys trigger parse warnings.

Relative file paths. Relative file paths are relative to the description file directory.

Library lookup. FIXME. Library lookup is currently quite restricted and done according to the following name mapping:

With $LIBDIR being defined (first match) by:

  1. The value of the environment variable B0_DRIVER_LIBDIR
  2. The value of the environment variable OPAM_SWITCH_PREFIX post
  3. The value of $(ocamlc -where)/..

Dependency resolution on the libraries is not performed and cmi files have to be in the corresponding libname directory.

Key reference

B0.b0 key merges

When multiple B0.b0 file are used, their specification is merged with the root description. During this process the key values of subdescriptions are either:

More build concepts

Output command digests

Commands are assumed to be pure functions of their inputs and declared process environment.

Concatenate and digest: The digest of the executable, the command line arguments, the spawn process environment, the digest of the contents of inputs, the output path. This becomes the name of the file in the cache.

Cleaning build

In a cleaning run, outputs that are not rebuild and were present in the previous run are deleted at the end of the run.

Build correctness

If you spend some time thinking about building software incrementally and correctly you quickly realize that our current file system and tool based approach is entirely hopless. The fact that you can't guarantee noone fiddles with the outputs of your build steps across build system runs. b0 is not different and you can entirely trip it by fiddling with the contents of its _b0 dir.

Recipes and menagerie

Writing conf discovery

Error only if really needed. Otherwise log with warning and default to a reasonable deafult value. Build units can still abort if they can't use the value.

Old TODO.md

B0.Def

B0.Conf

Discover should be `Build.t` based. This will give us caching/reacting to the environment for free and will allow to hide the build aim. Add a special build unit for it: this can give a directory to write things in aswell.

B0.Pkg

Should have a unit associated or maybe not it should be easy to sync on all the units of a package once we have the notion of package in units. This should be sufficient for `META` file generation and `opam` file sync.

`b0 build` improvements

`b0` driver

B0.Tool

Provide support for checking minimal version. Or not.

Proxy variants and _b0

In `B0_docker` The current scheme means that we copy rather than link(2). This is due to the way we mount stuff and the business of excluding parts of `_b0`. The whole way of proxying needs revisiting so that we can simply mount the root dir without excluding `_b0`. This should not be a big deal, somehow we only need to deal with `_b0/i` so that the proxy instance can live there (e.g. setup `B0_DRIVER_DIR` and/or forward with `--d-dir`).

However we also want to keep the current style for e.g. `ssh` proxies where we don't want to rsync all the local variants.

Variants

Build context

A topkg-like build context is needed. It's a bit unclear where this should fit. Ponder the following alternative.

  1. A configuration key in B0_care.
  2. A value held by `Env` and thus defined by the variant scheme.

In any case this should be exposed in `Build.t` values.

Also maybe another name than build context should be found. Source context, run runner, goal.

In general the behaviour of b0 tries to be as independent from the operating context as possible. However some configuration keys and/or actions are sensitive to the what is called the build context. This makes the invocations to specify on the command line and in package descriptions (e.g. the build: field of opam files) terser and thus generally improves the DRYness of the tool.

More specifically b0 distinguishes between the following contexts: