Brzo manual

Quick-setting builds – Builds ideas, not scaffolds.

Introduction

Philosophy

brzo lets you think about your ideas, not the bureaucracy needed to execute them. It is a simple build tool made for exploratory design and learning. It is unsuitable for building software distributions.

With this goal in mind brzo curtails configuration options and favors heuristics and advices over formal configuration.

If you find yourself fighting with the tool or seeking more control it likely means your ideas have grown into a project that needs a proper build infrastructure. brzo should be gleefully ditched at that point.

Getting started

In a simple directory structure, a bare brzo invocation should succeed in building and executing your source files. For example, assuming you have brzo and an OCaml compiler in your executable path:

cat > echo.ml <<EOCAML
print_endline (String.concat " " (List.tl (Array.to_list Sys.argv)))
EOCAML
> brzo -- 'Quick!'
'Quick!'

Not very impressive, but illustrates that a bare brzo invocation goes through the following ordered steps:

  1. Find the root: starting from the current directory and upwards, stop at the first directory with a BRZO file or, if none, use the current working directory.
  2. Collect source files in the file hierarchy pinned at the root and determine from their extensions a domain. A domain determines a build logic. It usually corresponds to a particular language.
  3. Using the domain's build logic, compile the source files to an executable outcome artefact.
  4. Run the outcome artefact with the provided command line arguments.

To understand which root, source files and domain brzo determines, the following commands can be used:

brzo root      # show root directory
brzo sources   # show source files considered
brzo domain    # show selected domain

If some directories get in the way during source file collection you can include and exclude them with the -i and -x options or via a BRZO file:

brzo -x attic    # ignore the 'attic' directory (relative to the root)
brzo -x +attic   # same, but doesn't override a BRZO file's srcs-x field

Play a bit with the brzo source command and these flags until you get the selection you desire. Note that relative paths given to these options are relative to the root.

In general brzo tries to be quiet on success so that it doesn't interfere with the standard outputs of the outcome action. However if you would like to have a bit more information on stderr about what brzo is doing, use the -v option. Alternatively consult the build log after the build with the log command.

brzo -v           # output build steps on stderr
brzo log          # output build operations of the last build
brzo log --help   # See more options to explore the build

brzo's exit code will be that of your outcome program unless brzo fails in some way before it gets to execute it. In the latter case the exit code should be attributed and understood as brzo's one. Exit codes of brzo and its subcommands are documented in brzo --help.

To only build the outcome, only perform the outcome action, locate the outcome artefact in the file system or delete the outcome build, use the following invocations:

brzo -b               # only build the outcome artefact
brzo -a -- 'Quick!'   # only perform the outcome action
brzo --where          # path to the outcome artefact (built or not)
brzo -d               # delete the outcome build (but not the cache)

Design documentation

Domains provide a documentation outcome for designing your programs. This outcome builds and shows documentation about the source files; most likely using the domain's language documentation tool. It can be built and run by invoking:

brzo --doc   # build and show documentation about source files

Selecting domains and outcomes

You may not always be happy about brzo's automatic domain selection. Or you may want to switch between different domains in the same root or use domain specific outcomes and options.

In these cases you can specify the domain as the first argument to brzo or use a BRZO file.

brzo ocaml -- 'Grunt!'   # force the use of the OCaml domain

A domain can provide more options for the default outcomes. For example, the OCaml domain has options to compile the (default) executable outcome to different targets:

brzo ocaml --byte -- 'Quick!'    # compile and execute a bytecode program
brzo ocaml --html -- 'Quick!'    # compile and execute an HTML program
brzo ocaml --native -- 'Quick!'  # compile and execute a native code program
brzo ocaml --node -- 'Quick!'    # compile and execute a JavaScript program

Automatic domain selection provides only two build outcomes: --exec (default) and --doc. A specific domain can provide more outcomes. For example the OCaml domain has outcomes to run sources in toplevels:

brzo ocaml --top          # compile and load in the OCaml toplevel
brzo ocaml --top --html   # compile and load in the HTML OCaml toplevel
brzo ocaml --utop         # compile and load in utop

To list available domains, their outcomes and options use:

brzo --help          # see the DOMAIN COMMANDS section
brzo DOMAIN --help   # see the OUTCOMES and OUTCOME OPTIONS section

Cleaning up

At that point you certainly noticed brzo creates a _b0 directory at the brzo root. This directory has the outcome artefacts in their build directories usually of the form _b0/brzo-DOMAIN-OUTCOME, a build cache _b0/.cache used for incremental builds and the (binary) build log _b0/.log.

Outcome builds and artefacts can be deleted using the following invocations. Note that this does not delete cached build artefacts.

brzo -d                # delete the build of default executable outcome
brzo ocaml --html -d   # delete the build of the OCaml HTML program outcome
brzo delete            # delete all outcomes builds (but not the cache)

In general unless you hit a build logic bug (please report) it's better to refrain from deleting the _b0 directory – get rid of "make clean" habits. If you think the cache is taking too much space use the brzo cache command to manage it:

brzo cache size     # show information about the cache
brzo cache gc       # delete unused cache keys (fs with hardlink support)
brzo cache trim     # trim to 50% of the current size
brzo cache delete   # delete all cache keys (but not the build outcomes)

But if you really want to get rid of the _b0 directory issue:

brzo delete -c   # delete (clean) the _b0 directory

Where to go next ?

Back to your ideas.

brzo strives to provide a hassle and scaffolding free build experience. If it succeeds you should be able to stop here and simply use the tool.

But if you get tired of specifying arguments on the command line you may want to read the section about BRZO files.

If a brzo invocation doesn't succeed or you are puzzled by an outcome result, it may be useful to go through the manual section of the domain you are using. You will get a better understanding on how it works and which domain specific outcomes, options and BRZO file keys it provides. A short list of links to these sections can be found here.

The conceptual overview of the next chapter expands slightly more formally on the material seen in this introduction.

Conceptual overview

Whenever brzo is executed to perform an outcome it proceeds through the following steps.

  1. Find the root directory.
  2. Collect supported source files in the file hierarchy pinned at the root.
  3. Select the build domain.
  4. Using domain and outcome specific logic, build an outcome artefact.
  5. Perform an action on the outcome artefact.

Root

When brzo is invoked in a directory cwd, the root is the first directory, starting with cwd and moving upwards, that has a BRZO file or, if none can be found, cwd itself.

Alternatively the root directory can be forced via the --root option or the BRZO_ROOT environment variable. Invoke brzo root to find out which root brzo determines.

Source files

Source files collected by brzo are those that are in the file hierarchy pinned at the root and which are deemed useful by any of the domains.

Source file selection can be controlled by the -x and -i command line options or by the BRZO file. Relative file paths are relative to the brzo root, not relative to the current working directory.

Paths with dotfile, _b0, or _build segments are automatically excluded, this can be overriden by specifying them explictely via -i.

Domains and selection

Brzo partitions build logics into domains. Domains broadly correspond to build logics for a given language, even though those can overlap, for example when we build mixed OCaml and C programs.

Unless the domain is explicitely specified on the command line by invoking a subcommand like brzo ocaml or at the root via a BRZO file, brzo selects a domain according to the following rules, applied in order:

  1. ocaml domain, if a source file ends with .ml, .mli .mld, .mll or .mly.
  2. c domain, if a source file ends with .h or .c.
  3. latex domain, if a source file ends with .tex, .sty, .bib or .bibdoi.
  4. cmark domain, if a source file ends with .md.

Outcomes

Each domain has a number of supported build outcomes. An outcome determines a build artefact and an action performed on it.

Every domain defines at least two standard outcomes:

Domains may provide additional outcomes. Use brzo DOMAIN --help and the domain specific sections of this manual listed here to learn about them.

Outcome actions

Most outcome actions fall in one of the following categories.

Program execution action

In a program execution action, the action simply runs a built program and gives it the arguments you specify on the brzo invocation. Use -- to separate execution arguments from those given to brzo, this allows optional arguments to be interpreted by your program and not by brzo:

brzo --help      # this is brzo's help
brzo -- --help   # this is your program's help (if programmed to do so)

Browser action

In a browser action, the action usually opens a local file: URI in your browser. The browser to use can be specified in the BROWSER environment variable or via the --browser option.

When the outcome is a web of HTML files, the opened URI is the directory in which these are rooted. brzo then tries to reload a tab whose address is rooted in the directory's hierarchy rather than the directory itself. The application used to show the URI can be kept in the background kept with the --background option.

Sadly, both tab prefix reload and --background are subject to severe platform limitations, it seems the whole idea of refreshing a tab with a given address from the cli is a total blind spot of browser makers – get in touch if you know lightweight workarounds.

PDF viewer action

In a PDF viewer action, the action opens a PDF file in a viewer. The viewer to use can be specified in the PDFVIEWER environment variable or via the --pdf-viewer option.

BRZO files

A BRZO file determines a root, allows to guide domain specific logic and provides default values. It can be empty just for root marking. BRZO files can be modified programmatically, see brzo file --help for more information.

A BRZO file is a sequence of s-expressions. Here's a sample file:

(domain c)             ; force the default domain to C
(srcs-i src)           ; limit source file collection to the src directory
(srcs-x src/attic)     ; exclude source files from src/attic
(ocaml                 ; OCaml domain specific options
  (outcome doc))       ; force the default outcome to --doc

Relative file paths in the file are relative to the root, not relative the BRZO file location – these may not coincide if the --brzo-file option is used to specify the BRZO file.

The following general keys are defined:

C domain

The C domain is todo.

Cmark domain

Provides support for handling CommonMark files via the cmark tool.

Exec outcome

Build a web of HTML programs (HTML, JavaScript, CSS) and media files in a directory and open it in the user's browser.

Let ROOT be the root, BUILD the outcome build directory and P match file paths. The following steps are taken:

  1. Any ROOT/P.ext source file with ext in the set Brzo_b0.Fexts.www extension is copied over verbatim to the corresponding BUILD/html/P.ext path.
  2. Any ROOT/P.md file is compiled to a BUILD/html/P.html file using the cmark tool. Any .js and .css file in BUILD is linked in the file. The page title is either the basename of the file without the extension or, if that results in "index", the basename of the .md file's parent directory.
  3. Open directory BUILD in a browser.

Doc outcome

Aliased to exec outcome.

BRZO file keys

The cmark domain has no specific keys.

LATEX domain

Provides support for handling the LATEX document preparation system. The domain uses the XƎTEX toolchain as it handles Unicode and UTF-8 out of the box; it is distributed with TEX Live.

Exec outcome

Create a PDF document from the LATEX sources and open it in a PDF viewer.

Let ROOT be the root, R be the basename of ROOT, BUILD the outcome build directory and P match file paths. The following steps are taken:

  1. Any ROOT/P.bibdoi file is compiled to a ROOT/P.brzo.bib BibTEX file as described here. This BibTEX file can be included in your .tex files. Do not edit it manually it will be overwritten by brzo.
  2. Determine in ROOT a main M.tex file. This is either, in order:

    1. The file mentioned via the --main option
    2. The file mentioned in the BRZO file key latex.main, see here.
    3. If there's a single .tex file, that file
    4. A file whose basename is main.tex
    5. A file whose basename is R.tex
  3. Compile M.tex to BUILD/M.pdf using xelatex until the fix point is reached.
  4. Open file BUILD/M.pdf in a PDF viewer.

Doc outcome

Aliased to exec outcome.

Listing outcome

Creates a PDF document with some of the source files listed by brzo sources. The list of supported source extensions is documented in brzo latex --help under the --listing option. Sources are assumed to be UTF-8 encoded.

Let ROOT be the root, R be the basename of ROOT, BUILD the outcome build directory and P match file paths. The following steps are taken:

  1. Generate a document BUILD/R-listing.tex which has one section for each source file ROOT/P.ext with ext in the supported source extensions. Source files are included using the listings package.
  2. Compile BUILD/R-listing.tex to BUILD/R-listing.pdf using xelatex until the fix point is reached.
  3. Open file BUILD/R-listing.pdf in a PDF viewer.

BRZO file keys

LATEX specific keys are found in the latex dictionary. Here's an example that indicates the main .tex file is doc/manual.tex

(latex (main doc/manual.tex))

The available keys are:

.bibdoi files

A .bibdoi file is a list of DOIs written as a sequence of s-expression atoms. Here's a sample .bibdoi file:

; Some useful references
10.1145/365230.365257
https://doi.org/10.2307/1968337

As shown above one can specify DOI either by the DOI itself or by an URI in which case the DOI number is taken to be the URI path without the leading /.

Each DOI D in the file is resolved to a BibTEX entry using the following HTTP request:

curl -LH "Accept: application/x-bibtex; charset=utf-8" https://doi.org/D

The URI used for the resolving can be changed in the BRZO file.

A P.bibdoi file defines a corresponding P.brzo.bib BibTEX file by concatenating the result of resolving each of its DOIs.

OCaml domain

Provides support for handling a set of OCaml, C and JavaScript files.

Exec outcome

Builds source files to an executable and runs it. There are four different outcome targets which define different kind of executables and actions:

Unless the target is explicitely specified --native is used if ocamlopt is available and --byte otherwise.

Byte and native code targets

Let BUILD be the outcome build directory.

  1. Using best-effort compilation compile sources to objects and link them to a BUILD/a.out executable.
  2. Execute BUILD/a.out.

HTML target

Let ROOT be the root, BUILD the outcome build directory and P match file paths. The following steps are taken:

  1. Using best-effort compilation compile sources to objects and link them to a BUILD/a.out bytecode executable.
  2. Using js_of_ocaml compile BUILD/a.out to BUILD/a.js.
  3. Compile a shim for the action's environment and arguments to BUILD/brzo_tty_glue.js. Action arguments which correspond to files that exist on the file system are crunched to that shim in js_of_ocaml's virtual file system so that they can be read by the HTML program.
  4. Any ROOT/P.ext source file with ext in the set Brzo_b0.Fexts.www extension is copied over verbatim to the corresponding BUILD/P.ext.
  5. If there is no BUILD/index.html file one is generated with any BUILD/P.js and BUILD/P.css files linked in the file, with brzo_tty_glue.js and a.js comming last.
  6. Open file BUILD/index.html in a browser.

Node target

Let ROOT be the root, BUILD the outcome build directory and P match file paths. The following steps are taken:

  1. Using best-effort compilation compile sources to objects and link them to a BUILD/a.out bytecode executable.
  2. Using js_of_ocaml compile BUILD/a.out to BUILD/a.out.js
  3. Any ROOT/P.js source file is copied over verbatim to the corresponding BUILD/P.js file.
  4. Using jsoo_link link the BUILD/P.js files and BUILD/a.out.js to BUILD/a.js.
  5. Execute BUILD/a.js via node.

Doc outcome

Build and show documentation about source files using odoc.

Let ROOT be the root, R be its basename, BUILD the outcome build directory, P match file paths and FILE be the basename of P.

  1. Using best-effort compilation produce a BUILD/cmti/FILE.cmi files for each ROOT/P.mli file. This produces BUILD/cmti/FILE.cmti as a side effect which is resolved for local .odoc dependencies and compiled to BUILD/odoc/R/FILE.odoc.
  2. Compile any ROOT/P.mld file to BUILD/odoc/R/page-FILE.odoc.
  3. Compile ROOT/P.md files to BUILD/html/_docs/P.html using the cmark tool (if available).
  4. If there is no index.mld file produce one which links to the .mli module pages, .mld pages and the .md HTML documents and compile it to BUILD/odoc/R/page-index.odoc
  5. Generate from .odoc files the HTML for a package named R in BUILD/html using the user's odig odoc theme if set.
  6. Open directory BUILD/html/R/ in a browser.

Top outcome

Build source files to object files and load them in the OCaml toplevel. There are four different outcome targets which define different actions:

Unless the target is explicitely specified --native is used if ocamlnat is available and --byte otherwise.

Byte and native code targets

Let ROOT be the root, BUILD the outcome build directory and P match file paths and FILE be the base names of P.

  1. Using best-effort compilation compile sources to objects and gather them in a BUILD/brzo_top.ext library archive with ext depending on the target.
  2. Execute ocaml or ocamlnat with BUILD/brzo_top.ext, its link dependencies and BUILD as an include directory. If arguments are specified they are given to the toplevel program invocation.

HTML target

Let ROOT be the root, BUILD the outcome build directory and P match file paths. The following steps are taken:

  1. Using best-effort compilation compile sources to objects and link them to a BUILD/a.out bytecode executable.
  2. Using js_of_ocaml compile BUILD/a.out to BUILD/a.js.
  3. TODO

Utop outcome

Except for the --native, --html and --node targets which are not supported this behaves like the top outcome but loads using the utop tool rather than ocaml.

BRZO file keys

OCaml specific keys are found in the ocaml dictionary. Here's an example that forces the executable target to html and constrains dependency resolution:

(ocaml
  (target html)      ; Default to produce an HTML program
  (deps ptime/jsoo)) ; Use the js_of_ocaml support of ptime

The available keys are:

Best-effort compilation

Let ROOT be the root, BUILD the outcome build directory and P match file paths and FILE be the base names of P. The following steps are taken:

  1. Module dependencies of ROOT/P.{mli,ml} files are resolved and sources are compiled to objects BUILD/FILE.ext in the correct order using compilation dependency resolution.
  2. ROOT/P.c C files are compiled to BUILD/FILE.o
  3. If ROOT/.merlin does not exist or exists and starts with "# Generated by brzo", write an ocamlmerlin file there according to the dependency resolutions that were found.
  4. According to the outcome proceed to the link phase with all the objects using link dependency resolution.

Note that the build scheme is flat so you cannot have two same FILE names in different directories of the source tree. If this happens a warning is output and one of the files is arbitrarily dropped.

Best-effort dependency resolution

Compilation dependencies

Given an .ml or .mli file SRC, the set CMIS of cmi files that need to be visible to compile SRC is determined as follows.

  1. ocamldep -modules is run on SRC to produce a set of module names DS to resolve to cmi files. Stdlib is added to this set. Modules names L that match OCaml sources in the brzo ROOT are removed from DS.
  2. For each of the local resolutions L found at the preceeding step add L.cmi to CMIS and wait for its compilation. Get all the module names that are defined in L.cmi and remove these from DS.
  3. For each remaining unresolved dependency E in DS perform an external dependency resolution for a file named E.cmi.

    • If there a single match add this file to CMIS. Get the all the module names that are defined in E.cmi and remove these from DS.
    • If there's no match or more than one match put the resolution of E on hold and proceed with subsequent dependencies to see if it might resolve by later resolutions

    Repeat until there is no dependency left or it is not possible to find a E.cmi for the remaining ones. In that case if:

    • E.cmi cannot be found. Then keep the dependency unresolved and proceed.
    • E.cmi resolves to more than one file. Fail the build and propose a dependency restriction to resolve the ambiguity.
  4. For all files E.cmi that were externally resolved we then lookup its imported interfaces and perform a resolution for these module references and add them to CMIS; otherwise the build may fail (XXX a bit unclear exactly why but if the E.cmi found has types from other modules but our sources don't mention these it seems we still need these cmis aswell (equations ?). For now a single step is done unclear whether this should be done recursively, ask the experts at some point).

At the end of this process if there are still remaining unresolved dependencies in DS we nevertheless proceed to compile with what we have found. If that fails we try to spellcheck these unresolved names with existing cmis in dependency directories and/or propose opam packages that could help (if available).

The set of cmi files used to compile an object may have been over approximated by compilation dependency resolution. However the OCaml compiler records in the object files exactly which ones were used. Therefore we read back the real compilation dependencies from the compiled objects.

We then hunt for implementations for these dependencies. We do so by performing cmi file lookups for them. If the file does not resolve locally an external dependency resolution is performed to find the cmi file. Once found this directory is scanned for a library archive that holds a corresponding implementation. If none is found the direct subdirectories are also looked up and if that fails an "mli-only" module is assumed.

Once an archive is selected this lookup procedure needs to be applied on each of the dependencies of the object files it contains and recursively so that linking does not fail.

Eventually this results in correctly sorted local objects and external archives that can be used for the final link step.

External dependencies and restrictions

brzo assumes external OCaml libraries and objects are installed in dependency directories specified via the repeatable and ordered --dep-dir option. If no --dep-dir option is specified the following dependency directories are implied (if available):

--dep-dir $(ocamlc -where) --dep-dir $(opam var lib)

Filenames (e.g. module.cmi) are looked up on a best-effort basis in all dependency directories and their subdirectories without order. In case such a file is found in multiple directories the build fails and brzo suggests dependency restrictions.

Dependency restrictions are specified with the --dep DEP option or in the BRZO file. A dependency DEP is a relative file path used to order (or restrict) the search performed in dependency directories.

Unless you have restrictions for all discovered dependencies, new installs in dependency directories may introduce new ambiguities, resulting in builds failing while they were previously succeeding. If you want to restrict the search to those directories defined as dependency restrictions use the --lock-deps option.

Let D1 and D2 be dependency directories in that order. And DEP1 and DEP2 be dependency restrictions in that order. A dependency restriction DEP1 has two effects:

  1. First brzo checks that at least one of D1/DEP1 or D2/DEP1 exists. If neither does, the build fails and DEP1 is spellchecked against existing dependencies and/or matched against uninstalled opam packages to propose an opam install (if available).
  2. Whenever a filename resolution is ambiguous but has a candidate in an existing restriction path then resolution is forced to that candidate. More precisely the candidate that is prefixed by the first of the following paths is taken:

    1. D1/DEP1
    2. D2/DEP1
    3. D1/DEP2
    4. D2/DEP2

For example the ptime package provides multiple implementation for the Ptime_clock module: one for your operating system and another one for your browser. This results in an ambiguous resolution when Ptime_clock is used in a source file. These implementations are respectively installed in these directories:

$(opam var lib)/ptime/jsoo
$(opam var lib)/ptime/os

In order to force the usage of the operating system clock use:

brzo ocaml --dep ptime/os

or add the following to the BRZO file:

(ocaml (deps ptime/os))

The BRZO file can be edited programmatically and a failing build may suggest you to do either of:

brzo file set ocaml.deps.v[0] ptime/os    # For the OS
brzo file set ocaml.deps.v[0] ptime/jsoo  # For the browser

which will insert the dependency at the beginning of the ocaml.deps dictionary key.

Comments and limitations

Here are a few points that should be taken into account or that could be improved.