Brzo manual

Quick-setting builds – Builds ideas, not scaffolds.

Introduction

Philosophy

brzo lets you think about your ideas, not the bureaucracy needed to execute them. It is a simple build tool made for exploratory design and learning. It is unsuitable for building software distributions.

With this goal in mind brzo curtails configuration options and favors heuristics and advices over formal configuration.

If you find yourself fighting with the tool or seeking more control it likely means your ideas have grown into a project that needs a proper build infrastructure. brzo should be gleefully ditched at that point.

Getting started

In a simple directory structure, a bare brzo invocation should succeed in building and executing your source files. For example, assuming you have brzo and an OCaml compiler in your executable path:

> touch BRZO
> cat > echo.ml <<EOCAML
let echo oc ss = output_string oc (String.concat " " ss ^ "\n")
let () = echo stdout (List.tl (Array.to_list Sys.argv))
EOCAML
> brzo -- 'Quick!'
'Quick!'

Not very impressive, but illustrates that a bare brzo invocation goes through the following ordered steps:

  1. Find the root: starting from the current directory and upwards, stop at the first directory with a BRZO file.
  2. Collect source files in the file hierarchy pinned at the root and determine from their extensions a domain. A domain determines a build logic. It usually corresponds to a particular language.
  3. Using the domain's build logic, compile the source files to an executable outcome artefact.
  4. Run the outcome artefact with the provided command line arguments.

To understand which root, source files and domain brzo determines, the following commands can be used:

brzo root      # show root directory
brzo sources   # show source files considered
brzo domain    # show selected domain

If some directories or files get in the way during source file collection you can include and exclude them with the -i and -x options and/or via a BRZO file:

brzo -x attic   # excludes paths prefixed by 'attic' (relative to the root)

Play a bit with the brzo source command and these flags until you get the selection you desire. Relative paths given to these options are relative to the root.

The relative order between -i and -x options is irrelevant, the semantics is that all source file paths whose prefix match one of the -i options are gathered into a set and then all elements whose prefix match one of the -x option are removed from it. For example below the -i option is ineffective in both cases:

brzo -i . -i attic -x attic   # excludes 'attic' prefixed paths
brzo -i . -x attic -i attic   # also excludes 'attic' prefixed paths

In general brzo tries to be quiet on success so that it doesn't interfere with the standard outputs of the outcome action. However if you want to have a bit more information on stderr about what brzo is doing, use the -v option. Alternatively consult the build log after the build with the log command.

brzo -v           # show build steps on stderr
brzo log          # show build operations of the last build
brzo log --help   # see more options to explore the build

brzo's exit code will be that of your outcome program unless brzo fails in some way before it gets to execute it. In the latter case the exit code should be attributed and understood as brzo's one. Exit codes of brzo and its subcommands are documented in brzo --help.

To only build the outcome, only perform the outcome action, locate the outcome artefact in the file system, delete the outcome build or see the outcome configuration, use the following invocations:

brzo -b               # only build the outcome artefact
brzo -a -- 'Quick!'   # only perform the outcome action
brzo --path           # path to the outcome artefact (built or not)
brzo -d               # delete the outcome build (but not the cache)
brzo --conf           # show determined configuration

Design documentation

Domains provide a documentation outcome for designing your programs. This outcome builds and shows documentation about the source files; most likely using the domain's language documentation tool. It can be built and run by invoking:

brzo --doc   # build and show documentation about source files

Selecting domains and outcomes

You may not always be happy about brzo's automatic domain selection. Or you may want to switch between different domains in the same root or use domain specific outcomes and options.

In these cases you can specify the domain as the first argument to brzo or use a BRZO file.

brzo ocaml -- 'Grunt!'   # force the use of the OCaml domain

A domain can provide more options for the default outcomes. For example, the OCaml domain has options to compile the (default) executable outcome to different targets:

brzo ocaml --byte -- 'Quick!'    # compile and execute a bytecode program
brzo ocaml --html -- 'Quick!'    # compile and execute an HTML program
brzo ocaml --native -- 'Quick!'  # compile and execute a native code program
brzo ocaml --node -- 'Quick!'    # compile and execute a JavaScript program

Automatic domain selection provides only two build outcomes: --exec (default) and --doc. A specific domain can provide more outcomes. For example the OCaml domain has outcomes to run sources in toplevels:

brzo ocaml --top          # compile and load in the OCaml toplevel
brzo ocaml --top --html   # compile and load in the HTML OCaml toplevel
brzo ocaml --utop         # compile and load in utop

To list available domains, their outcomes and options use:

brzo --help          # see the DOMAIN COMMANDS section
brzo DOMAIN --help   # see the OUTCOMES and OUTCOME OPTIONS section

Cleaning up

At that point you certainly noticed brzo creates a _b0 directory at the brzo root. This directory has the outcome artefacts in their build directories usually of the form _b0/brzo/DOMAIN-OUTCOME, a build cache _b0/.cache used for incremental builds and the (binary) build log _b0/brzo/.log.

Outcome builds and artefacts can be deleted using the following invocations. Note that this does not delete cached build artefacts.

brzo -d                # delete the build of default executable outcome
brzo ocaml --html -d   # delete the build of the OCaml HTML program outcome
brzo delete            # delete all outcomes builds (but not the cache)

In general unless you hit a build logic bug (please report) it's better to refrain from deleting the _b0 directory – get rid of "make clean" habits. If you think the cache is taking too much space use the brzo cache command to manage it:

brzo cache size     # show information about the cache
brzo cache gc       # delete unused cache keys (fs with hardlink support)
brzo cache trim     # trim to 50% of the current size
brzo cache delete   # delete all cache keys (but not the build outcomes)

But if you really want to get rid of the _b0 directory issue:

brzo delete -c   # delete (clean) the _b0 directory

Where to go next ?

Back to your ideas.

brzo strives to provide a hassle and scaffolding free build experience. If it succeeds you should be able to stop here and simply use the tool.

But if you get tired of specifying arguments on the command line you may want to read the section about BRZO files. You may also want to give a look at configuring your editor.

If a brzo invocation doesn't succeed or you are puzzled by an outcome result, it may be useful to go through the manual section of the domain you are using. You will get a better understanding on how it works and which domain specific outcomes, options and BRZO file keys it provides. A short list of links to these sections can be found here.

The conceptual overview of the next chapter expands slightly more formally on the material seen in this introduction.

Conceptual overview

Whenever brzo is executed to perform an outcome it proceeds through the following steps.

  1. Find the root directory.
  2. Collect supported source files in the file hierarchy pinned at the root.
  3. Select the build domain.
  4. Using domain and outcome specific logic, build an outcome artefact.
  5. Perform an action on the outcome artefact.

Root

When brzo is invoked in a directory cwd, the root is the first directory, starting with cwd and moving upwards, that has a BRZO file. If none can be found the tool errors.

Alternatively the root directory can be forced without a BRZO file via the --root option or the BRZO_ROOT environment variable. Invoke brzo root to find out which root brzo determines.

Source files

The set of source files collected by brzo are the files which are deemed useful by any of the domains and whose paths are prefixed by the include prefixes specified by the -i PATH option and the srcs-i BRZO file key. They must also not be prefixed by any of the exclude prefixes specified by the -x PATH option and the srcs-x BRZO file key. If no -i or srcs-x is specified the root path is implied.

Relative file paths specified in these options are relative to the brzo root, not relative to the current working directory or BRZO file. Note that these are prefixes, globs are not supported.

The relative order between -i and -x options is irrelevant. The semantics is that all source files from the file system whose prefix match one of the -i options are collected into a set and then any element of that set whose prefix match any of the -x option is removed from it. Paths with dotfile, _b0, or _build segments are automatically excluded, this can be overriden by specifying them explictely via -i.

If a -i is specified on the command line and there's a corresponding path in the srcs-x key of the BRZO file, then the latter is removed. Dually if a -x is present on the command line and there's a corresponding path in the srcs-i key of the BRZO file, then the latter is removed (but that's by definition of the semantics anyways).

Invoke brzo source to find out the final source set and brzo --conf to find out which includes and excludes are determined from the BRZO file and the command line.

Domains and selection

Brzo partitions build logics into domains. Domains broadly correspond to build logics for a given language, even though those can overlap, for example when we build mixed OCaml and C programs.

Unless the domain is explicitely specified on the command line by invoking a subcommand like brzo ocaml or at the root via a BRZO file, brzo selects a domain according to the following rules, applied in order:

  1. ocaml domain, if a source file ends with .ml, .mli, .mld, .mll or .mly.
  2. c domain, if a source file ends with .h or .c.
  3. latex domain, if a source file ends with .tex, .sty, .bib or .bibdoi.
  4. cmark domain, if a source file ends with .md.

Invoke brzo domain to find out which domain brzo determines.

Outcomes

Each domain has a number of supported build outcomes. An outcome determines a build artefact and an action performed on it.

Every domain defines at least two standard outcomes:

Domains may provide additional outcomes. Use brzo DOMAIN --help and the domain specific sections of this manual listed here to learn about them.

Outcome actions

Most outcome actions fall in one of the following categories.

Program execution action

In a program execution action, the action simply runs a built program and gives it the arguments you specify on the brzo invocation. Use -- to separate execution arguments from those given to brzo, this allows optional arguments to be interpreted by your program and not by brzo:

brzo --help      # this is brzo's help
brzo -- --help   # this is your program's help (if programmed to do so)

Browser action

In a browser action, the action usually opens a local file: URI in your browser. The browser to use can be specified in the BROWSER environment variable or via the --browser option.

When the outcome is a web of HTML files, the opened URI is the directory in which these are rooted. brzo then tries to reload a tab whose address is rooted in the directory's hierarchy rather than the directory itself. The application used to show the URI can be kept in the background with the --background option.

Sadly, both tab prefix reload and --background are subject to severe platform limitations, it seems the whole idea of refreshing a tab with a given address from the cli is a total blind spot of browser makers – get in touch if you know lightweight workarounds.

PDF viewer action

In a PDF viewer action, the action opens a PDF file in a viewer. The viewer to use can be specified in the PDFVIEWER environment variable or via the --pdf-viewer option.

Editor support

Emacs

If you want to have brzo as a default compile command add one of the following to your .emacs:

(require 'compile)
(setq compile-command "brzo -b") ; Globally instead of the default make -k

; Only in a particular mode. Here caml-mode.
(add-hook 'caml-mode-hook
  (lambda ()
    (set (make-local-variable 'compile-command) "brzo -b")))

BRZO files

A BRZO file determines a root, allows to guide domain specific logic and provides default values. It can be empty just for root marking. BRZO files can be modified programmatically, see brzo file --help for more information.

To disable a BRZO file at the root use the option --brzo-file with /dev/null; doing this does not affect root finding.

A BRZO file is a sequence of s-expressions. Here's a sample file:

(domain c)             ; force the default domain to C
(srcs-i src)           ; limit source file collection to the src directory
(srcs-x src/attic)     ; exclude source files from src/attic
(ocaml                 ; OCaml domain specific options
  (outcome top))       ; force the default outcome to --top

Keys

Relative file paths in the file are always relative to the root, not relative the BRZO file location – these may not coincide if the --brzo-file option is used to specify the BRZO file.

The following global keys are defined:

C domain

The C domain is todo.

Cmark domain

Provides support for handling CommonMark files via the cmark tool.

Exec outcome

Build a web of HTML programs (HTML, JavaScript, CSS) and media files in a directory and open it in the user's browser.

Let ROOT be the root, BUILD the outcome build directory and P match file paths. The following steps are taken:

  1. Any ROOT/P.ext source file with ext in the set B0_fexts.www extension is copied over verbatim to the corresponding BUILD/html/P.ext path.
  2. Any ROOT/P.md file is compiled to a BUILD/html/P.html file using the cmark tool. Any .js and .css file in BUILD is linked in the file. The page title is either the basename of the file without the extension or, if that results in "index", the basename of the .md file's parent directory.
  3. Open directory BUILD in a browser.

Doc outcome

Aliased to exec outcome.

BRZO file keys

The cmark domain has no specific keys.

LATEX domain

Provides support for handling the LATEX document preparation system. The domain uses the XƎTEX toolchain as it handles Unicode and UTF-8 out of the box; it is distributed with TEX Live.

Exec outcome

Create a PDF document from the LATEX sources and open it in a PDF viewer.

Let ROOT be the root, R be the basename of ROOT, BUILD the outcome build directory and P match file paths. The following steps are taken:

  1. Any ROOT/P.bibdoi file is compiled to a ROOT/P.brzo.bib BibTEX file as described here. This BibTEX file can be included in your .tex files. Do not edit it manually it will be overwritten by brzo.

  2. Determine in ROOT a main M.tex file. This is either, in order:

    1. The file mentioned via the --main option
    2. The file mentioned in the BRZO file key latex.main, see here.
    3. If there's a single .tex file, that file
    4. A file whose basename is main.tex
    5. A file whose basename is R.tex
  3. Compile M.tex to BUILD/M.pdf using xelatex until the fix point is reached.
  4. Open file BUILD/M.pdf in a PDF viewer.

Doc outcome

Aliased to exec outcome.

Listing outcome

Creates a PDF document with some of the source files listed by brzo sources. The list of supported source extensions is documented in brzo latex --help under the --listing option. Sources are assumed to be UTF-8 encoded.

Let ROOT be the root, R be the basename of ROOT, BUILD the outcome build directory and P match file paths. The following steps are taken:

  1. Generate a document BUILD/R-listing.tex which has one section for each source file ROOT/P.ext with ext in the supported source extensions. Source files are included using the listings package.
  2. Compile BUILD/R-listing.tex to BUILD/R-listing.pdf using xelatex until the fix point is reached.
  3. Open file BUILD/R-listing.pdf in a PDF viewer.

BRZO file keys

LATEX specific keys are found in the latex dictionary. Here's an example that indicates the main .tex file is doc/manual.tex

(latex (main doc/manual.tex))

The available keys are:

.bibdoi files

A .bibdoi file is a list of DOIs written as a sequence of s-expression atoms. Here's a sample .bibdoi file:

; Some useful references
10.1145/365230.365257
https://doi.org/10.2307/1968337

As shown above one can specify DOI either by the DOI itself or by an URI in which case the DOI number is taken to be the URI path without the leading /.

Each DOI D in the file is resolved to a BibTEX entry using the following HTTP request:

curl -LH "Accept: application/x-bibtex; charset=utf-8" https://doi.org/D

The URI used for the resolving can be changed in the BRZO file.

A P.bibdoi file defines a corresponding P.brzo.bib BibTEX file by concatenating the result of resolving each of its DOIs.

OCaml domain

Provides support for handling a set of OCaml, C and JavaScript files.

Exec outcome

Builds source files to an executable and runs it. There are four different outcome targets which define different kind of executables and actions:

Unless the target is explicitely specified --native is used if ocamlopt is available and --byte otherwise.

Byte and native code targets

Let BUILD be the outcome build directory.

  1. Using best-effort compilation compile sources to objects and link them to a BUILD/a.out executable.
  2. Execute BUILD/a.out.

HTML target

Let ROOT be the root, BUILD the outcome build directory and P match file paths. The following steps are taken:

  1. Using best-effort compilation compile sources to objects and link them to a BUILD/a.out bytecode executable.
  2. Using js_of_ocaml compile BUILD/a.out to BUILD/a.js.
  3. Compile a shim for the action's environment and arguments to BUILD/brzo_tty_glue.js. Action arguments which correspond to files that exist on the file system are crunched to that shim in js_of_ocaml's virtual file system so that they can be read by the HTML program.
  4. Any ROOT/P.ext source file with ext in the set B0_fexts.www extension is copied over verbatim to the corresponding BUILD/P.ext.
  5. If there is no BUILD/index.html file one is generated with any BUILD/P.js and BUILD/P.css files linked in the file, with brzo_tty_glue.js and a.js comming last.
  6. Open file BUILD/index.html in a browser.

Node target

Let ROOT be the root, BUILD the outcome build directory and P match file paths. The following steps are taken:

  1. Using best-effort compilation compile sources to objects and link them to a BUILD/a.out bytecode executable.
  2. Using js_of_ocaml compile BUILD/a.out to BUILD/a.out.js
  3. Any ROOT/P.js source file is copied over verbatim to the corresponding BUILD/P.js file.
  4. Using jsoo_link link the BUILD/P.js files and BUILD/a.out.js to BUILD/a.js.
  5. Execute BUILD/a.js via node.

Doc outcome

Build and show documentation about source files using odoc.

Let ROOT be the root, R be its basename, BUILD the outcome build directory, P match file paths and FILE be the basename of P.

  1. Using best-effort compilation produce a BUILD/cmti/FILE.cmi files for each ROOT/P.mli file. This produces BUILD/cmti/FILE.cmti as a side effect which is resolved for local .odoc dependencies and compiled to BUILD/odoc/R/FILE.odoc.
  2. Compile any ROOT/P.mld file to BUILD/odoc/R/page-FILE.odoc.
  3. Compile ROOT/P.md files to BUILD/html/_docs/P.html using the cmark tool (if available).
  4. If there is no index.mld file produce one which links to the .mli module pages, .mld pages and the .md HTML documents and compile it to BUILD/odoc/R/page-index.odoc
  5. Generate from .odoc files the HTML for a package named R in BUILD/html using the user's odig odoc theme if set.
  6. Open directory BUILD/html/R/ in a browser.

Top outcome

Build source files to object files and load them in the OCaml toplevel. There are four different outcome targets which define different actions:

Unless the target is explicitely specified --native is used if ocamlnat is available and --byte otherwise.

Byte and native code targets

Let ROOT be the root, BUILD the outcome build directory and P match file paths and FILE be the base names of P.

  1. Using best-effort compilation compile sources to objects and gather them in a BUILD/brzo_top.ext library archive with ext depending on the target.
  2. Execute ocaml or ocamlnat with BUILD/brzo_top.ext, its link dependencies and BUILD as an include directory. If arguments are specified they are given to the toplevel program invocation.

HTML target

Let ROOT be the root, BUILD the outcome build directory and P match file paths. The following steps are taken:

  1. Using best-effort compilation compile sources to objects and link them to a BUILD/a.out bytecode executable.
  2. Using js_of_ocaml compile BUILD/a.out to BUILD/a.js.
  3. TODO

Utop outcome

Except for the --native, --html and --node targets which are not supported this behaves like the top outcome but loads using the utop tool rather than ocaml.

BRZO file keys

OCaml specific keys are found in the ocaml dictionary. Here's an example that forces the executable target to html and constrains library resolution:

(ocaml
  (target html)      ; Default to produce an HTML program
  (libs ptime/jsoo)) ; Force the use the js_of_ocaml support of ptime

The available keys are:

Best-effort compilation

Let ROOT be the root, BUILD the outcome build directory and P match file paths and FILE be the base names of P. The following steps are taken:

  1. Module dependencies of ROOT/P.{mli,ml} files are resolved and sources are compiled to objects BUILD/FILE.ext in the correct order using compilation dependency resolution.
  2. ROOT/P.c C files are compiled to BUILD/FILE.o
  3. If ROOT/.merlin does not exist or exists and starts with "# Generated by brzo", write an ocamlmerlin file there according to the dependency resolutions that were found.
  4. According to the outcome proceed to the link phase with all the objects using link dependency resolution.

Note that the build scheme is flat so you cannot have two same FILE names in different directories of the source tree. If this happens a warning is output and one of the files is arbitrarily dropped.

Best-effort dependency resolution

Compilation dependencies

Given an .ml or .mli file SRC, the set CMIS of cmi files that need to be visible to compile SRC is determined as follows.

  1. ocamldep -modules is run on SRC to produce a set of module names DS to resolve to cmi files. Stdlib is added to this set. Modules names L that match OCaml sources in the brzo ROOT are removed from DS.
  2. For each of the local resolutions L found at the preceeding step add L.cmi to CMIS and wait for its compilation. Get all the module names that are defined in L.cmi and remove these from DS.

  3. For each remaining unresolved dependency E in DS perform an external dependency resolution for a file named E.cmi.

    • If there a single match add this file to CMIS. Get the all the module names that are defined in E.cmi and remove these from DS.
    • If there's no match or more than one match put the resolution of E on hold and proceed with subsequent dependencies to see if it might resolve by later resolutions

    Repeat until there is no dependency left or it is not possible to find a E.cmi for the remaining ones. In that case if:

    • E.cmi cannot be found. Then keep the dependency unresolved and proceed.
    • E.cmi resolves to more than one file. Fail the build and propose a dependency restriction to resolve the ambiguity.
  4. For all files E.cmi that were externally resolved and L.cmi files we then lookup their imported interfaces and perform a resolution for these module references and add them to CMIS and recursively; otherwise the build may fail because we may be missing type equations.

At the end of this process if there are still remaining unresolved dependencies in DS we nevertheless proceed to compile with what we have found. If that fails we try to spellcheck these unresolved names with existing cmis in library directories and/or propose opam packages that could help (if available).

The set of cmi files used to compile an object may have been over approximated by compilation dependency resolution. However the OCaml compiler records in the object files exactly which ones were used. Therefore we read back the real compilation dependencies from the compiled objects.

We then hunt for implementations for these dependencies. We do so by performing cmi file lookups for them. If the file does not resolve locally an external dependency resolution is performed to find the cmi file. Once found this directory is scanned for a library archive that holds a corresponding implementation. If none is found the direct subdirectories are also looked up and if that fails an "mli-only" module is assumed.

Once an archive is selected this lookup procedure needs to be applied on each of the dependencies of the object files it contains and recursively so that linking does not fail.

Eventually this results in correctly sorted local objects and external archives that can be used for the final link step.

External dependencies and restrictions

brzo assumes external OCaml library archives and objects are installed in root directories specified in the OCAMLPATH environment variable.

If the variable is undefined it defaults to:

OCAMLPATH=$(opam var lib):$(ocamlc -where)
OCAMLPATH=$(opam var lib) # if opam is dirs coincide
OCAMLPATH=$(ocamlc -where) # if opam is available

and in this case when ocamlc -where is used the brzo maps ocaml specially to it.

Filenames (e.g. module.cmi) are looked up on a best-effort basis in all root directories of OCAMLPATH and their subdirectories without order. In case such a file is found in multiple directories the build fails and brzo suggests library search restrictions.

Library search restrictions are specified with ordered --lib LIB options or in the BRZO file. A library LIB is a relative file path used to order or restrict the search performed in the libraries of OCAMLPATH.

Let OCAMLPATH=R1:R2 and LIB1 and LIB2 be library restrictions in that order. A library restriction LIB1 has two effects:

  1. First before starting the build brzo checks that at least one of R1/LIB1 or R2/LIB1 exists. If neither does, the build fails and LIB1 is spellchecked against existing directories in R1 and R2 and/or matched against uninstalled opam packages to propose an opam install (if available).

  2. Whenever a filename resolution is ambiguous but has a candidate in an existing library restriction then resolution is forced to that candidate. More precisely the candidate that is prefixed by the first of the following paths is taken:

    1. R1/LIB1
    2. R2/LIB1
    3. R1/LIB2
    4. R2/LIB2

Unless you have restrictions for all discovered dependencies, new installs in root directories of OCAMLPATH may introduce new ambiguities, resulting in builds ambiguously failing while they were previously succeeding. If you want to restrict the search to those directories defined as dependency restrictions use the --lock-libs option (see corresponding key in the BRZO file).

For example the ptime package provides multiple implementation for the Ptime_clock module: one for your operating system and another one for your browser. This results in an ambiguous resolution when Ptime_clock is used in a source file. These libraries are respectively installed in these directories:

$(opam var lib)/ptime/jsoo
$(opam var lib)/ptime/os

In order to force the usage of the operating system clock library use:

brzo ocaml --lib ptime/os

or add the following to the BRZO file:

(ocaml (libs ptime/os))

If you'd rather compile for the browser use:

brzo ocaml --html --lib ptime/jsoo

The BRZO file can be edited programmatically and a failing build may suggest you to do either of:

brzo file set ocaml.libs.v[0] ptime/os    # For the OS
brzo file set ocaml.libs.v[0] ptime/jsoo  # For the browser

which will insert the library restriction at the beginning of the ocaml.libs dictionary key.

Comments and limitations

Here are a few points that should be taken into account or that could be improved.