Design considerations and todo

Next

Bugs

• The ocamlfind library resolver is doing strange stuff on post-execution. Using B0_zero.Op.disable_reviving has now no effect on a revived operation, otherwise we'd get no hashes in the log. If we really wanted to have the real effect we'd need to prune the cache. In any case this would give us logs without hashes for these operations which would needlessly delete stuff on b0 cache gc. Note it's a bit unclear whether there's not a problem in reviving `Tee which is used here as a hack as we would end up in the disabling cases. It should be investigated.

b0.std library

• Review the tool lookup stuff. First we may want to generalize the lookup path to a function (rather than list of paths). Then maybe we should simply have a DLS global (really ?).
• B0_std.Os.Fd.copy use sendfile on Unix. On macOS we could use clonefile (> 10.12) or copyfile (> 10.5, uses clonefile if supported) – note though that this work on paths. See also copy_file_range.
• Review the error reporting strategy in file system operations and properly document it.
• B0_std.Log review usage of header. I think that we could drop it in favor of some kind of label in Fmt.

b0.memo library

• The signature of B0_memo.write looks wrong you want a fut. E.g. we don't want to the Fut.sync here:

  let crunch_file m file ~in_dir =
    let open Fut.Syntax in
    let mn = B0_ocaml.Mod.Name.of_mangled_filename (Fpath.basename file) in
    let out = Fpath.(in_dir / String.Ascii.uncapitalize mn + ".ml") in
    B0_memo.Memo.write m out ~reads:[file] begin fun () ->
    (* FIXME b0 the sig of write is wrong *)
    let data = Fut.sync (B0_memo.Memo.read m file) in
    Ok (B0_memo_ocaml.Crunch.string_to_string ~id:"file" ~data)
    end;
    Fut.return out

• Bzero.Op.post_exec we may want to error. See for example B0_ocaml.script.

• To sequence spawns k is a bit annoying to use. Synchronizing through writes could also be nice. We'd need an easy way to add a written file that is not written by the spawn but by the memo machinery.

• Shouldn't we warn against multiple writes/out of order reads as soon as they happen ?

b0.file library

• B0.ml file finding. Should we stop at the last (vs the first now) B0.ml file upwards ? This would be nice if you operate in a vendored dir. But this also avoid problems like I do a checkout e.g. in a scratch directory in _b0 to build it. On the other hand we have --b0-file for these cases.

• Should a @@@B0.version be added ? Think harder about it an especially with respect to B0_kit.V000 versioning. and coherence with @@@B0.includes. Also we could require to specify say #require "b0.kit.v000". Write down a few properties we would like to have w.r.t. opam and/or inclusion and end-user support. Also if we use @@@B0.version rather than open we might get in trouble with merlin. If would rather say no.

• Add a repeatable b0-include=[scope,]path option that simply prepends a corresponding directive to the B0.ml file. This allows for quick transient vendors.

Build fragments

At the b0 level we need to expose build fragments. It seems the build procedures of units is a good candidate but for now it's a bit unconvient to do that. We need to clarify the configuration store and dynamic metadata see B0_meta. For now as a temporary hack we add the wrap parameter to units.

B0_scope

* Is still quite dirty. Review. * When we B0.include a scope `B0_unit.list ()` will have them is that expected ?

B0_env

• Consider merging B0_build into B0_env. Rethink that when get back to redo build variants.

• Maybe we should start exposing a mini Os in B0_env for actions. This might be needed for remote builds. In fact at some point we should indicate on which Os the action happens `Build, `Host or `Driver (b0 invocation). Maybe it's then sufficient to a do proxy dance by executing the same b0 driver executable on the remote platform, at which point the Os invocation run on the right platform. Do we ever want mixed `Build and `Host ? Not undoable but a bit cold feet on replicating all the B0_std.Os module in B0_env to allow that.

• Also see we what to do with B0_build.B0_dir.

B0_unit

• Should we perhaps move build_proc to a meta key ? This would allow to name the build procedures like we do with actions.

B0_unit.Action

• Maybe we could specify an option that allows an action to always requires a build and defaults to the default build if none is explicitely specified. This could be useful or not for .ocaml ocaml. Though you likely often only want to test an excutable or library so it's just a b0 -u myexec -- .ocaml ocaml.

• Store keys on the build of an action, that works but it means we can't do that on specific subcommands when the action is a cmdliner group command. E.g. it would be nice to have everything for ocaml under .ocaml but `.ocaml ocaml` will want a byte code build and `.ocaml ocamlnat` will want a native code build. So for now we have `.ocaml.repl`.

• Access control. Maybe actions accessing the full builds opens a can of worms (or not) we could try to enforce more access control via the definition scope of the action (lib vs. file).

• Execution logic. If a tool conflicts with action warn but use tool, b0 unit action can be used for the action. If a tool conflicts with a unit warn if it's not the same unit. We should have all of b0 unit action, b0 tool exec to resolve conflicts.

• The current scheme is not yet what we want. One reason is that it's not a good idea to first build and then parse the cli args, we should error asap. The second one is that we really want some actions to be able to define what to build dynamically; but it's likely a good thing to forbid new builds then the action runs. This is unconvenient to do right now (there is a hack).

B0_build

• More design is needed around B0_unit.tool_is_user_accessible in builds. The B0_unit.in_root_scope logic entails that build procedures may work when a build procedure may work when defined in a B0.ml file and used as a root but no longer when it is included. Not good.

• Differences between actions and build procedures should be lessened. Case in point, you may start with an action and decide to memoize it. This entails needless churn, for example we access the same directories but through a different API, should B0_build disapear in favour of B0_env (with some API entry points failing in b0_action ?).

B0_meta

• Move B0_meta.Key to B0_key ?
• They used to be serializable, see if we don't want that again. Especially in the optic of a merge with B0_store. For dynamic keys it would also be nice to see what a build determined.

• Dynamic values. Comment dump.

In the context of a merge with B0_store it should be possible to determine keys in various ways without needing to be explicit about it like for example the datatype of B0_unit.Action.env is. But If we build dynamics directly into keys, we should keep the option to keep keys static only.

It remains we want to built-in dynamic keys. Maybe explicit better as it allows to specify the dynamism scope on a key per key basis via the arguments of the function. We should however agree on a return type.

Dynamics. For now we used 'a Fut B0_meta.key or functional keys (see B0_unit.Action. See how it goes fares for sync. Push/pull. In the optic of direct style I think we rather want ivars.

B0_store

It's a bit unclear whether we don't want to merge B0_meta and B0_store perhaps at the expense of more complex B0_meta.Key. Another take is to merge B0_store into memo.

Also for now only actions can mandate store bindings (e.g. to request a bytecode build). This should of course be user definable, likely at the build variant level.

• The B0_store is also useful for storing stuff that multiple

• The B0_build.self hack is a bit dangerous some of the function on the build value depend on current unit. But the build value has (rightly) no current unit.
• It seem we often wish to depend on the build. But we need to be careful the store is global so e.g. if we want to store the VCS describe of a scope we cannot store it as a single key. Should store keys be scoped ? (Think local thread storage).
• Another thing is to sort out the relation ship to B0_std.Bval. For consumers the keys in B0_store act like a B0_std.Bval, it's just that access triggers its computation.

B0_ocaml

• We should let constraints be defined on libraries. Then we can auto-generate opam constraints.
• In the library resolver implement a mecanism that allows to specify public/private (scope), deprecation and block policies.
• .ocaml findmod MOD, some stuff of `omod` could be integrated.
• Try to have a better solution for the htmlact-examples. By default on Auto B0_ocaml.wanted we get Native but this also dynamically needs Byte. For now we simply play on Auto's behaviour to get byte aswell but it's unsatisfactory. Static requires and/or build scheme level restrictions should be added. *)

B0_srcs

• Review.
• Integrate watermarks, needs ini.

b0.kit library

B0_c

Is missing.

B0_dune

On hold until we get the B0_srcs thing right.

Testing and B0_testing and B0_expect

• Implicit source location improvements. We could get rid of __POS__ and __POS_OF__. We could add a --test-locs option, which outputs test function locations (we did not add a __POS__ argument to Test.test

• Now that we added Test.Snapshot we should try to meld B0_expect into B0_testing.
• b0 test, run as soon as build, run in parallel.
• b0 test, more exit code semantics for skips and long tests.
• b0 test, document the full test executable protocol (env var + exit codes) concisely.
• b0 test now that we have a protocol with test executables should we drop the skips at the unit level ?
• Timeouts (see pathological.ml in cmarkit)
• Progress.
• Once we run snapshot tests on Windows we will likely need a newline normalization business.
• Test.Arg add lazy arguments.
• We want tests to be let bound so that it's easy to invoke them induvidually, e.g. in the toplevel.
• Have test for a full program execution effect (file system, exit code, standard outputs) via a dedicated machine readable file that is updated by the run.
• Get a story for binary/visual regression. First we should have a good `git difftool` and `pixeldrift` setup regardless of testing. Then we likely want combinators that automatically checkout the rewritten path if the perception threshold is not trigerred. Basically in case we detect a modified `path`, invoke a perceptual tool with `path` and `git show HEAD:path`, if the tool exits with 0 do a `git checkout path` to make it as if nothing happened.
• Add a notion of group in Outcome.test.
• Add filtering to selectively test expecations. Based on group names and file globs.

• Logging maybe move to:

  • normal, paths but no diff
  • long, paths and diffs
  • short, only final summary

B0_opam

• Worfklow to easily check opam file in sync. Part of more general workflow ? In fact it would be better to have https://github.com/ocaml/opam/issues/3797

B0_release

• Finish archive creation. Need to solve watermarks. Need to allow custom exclusion and possibly a cusom massage hook.
• Need to implement archive linting/building/testing.
• Implement .release publish.
• Review and finish the docs.
• The topkg delegate had a way to list issues on the cli. Maybe integrate something like (along with open PRs) in `.release status`.

B0_show_url

• For now only tool names defined in units (as defined by b0 tool list) can be used for the URL/tool execution form.

  We likely want to be able to use it with actions aswell. But this needs a bit of twisting around since currently the action bit gets invoked at the end of the build. We need to be able to influence the build itself (especially since actions can influence the store) and for example the pack and units requires need to be able to be defined dynamically.

  Basically the larger problems seems to be that currently it's difficult for actions to invoke actions.

B0_jsoo

Got a bit stuck on trying to clean up the main units and getting rid of the show-uri bespoke action because .show-url.url has not future support and B0_jsoo.html_page relies on that, but then maybe it shouldn't. Unit ideas:

1. Unit exe (js only) 2. Unit html_page or bundle (more or less what current B0_jsoo.html_page. 3. Unit packed_html_page (packs all resource for single file distribution).

Tool

Deploy

For now let's see where the new actions bring us. But the idea is that a deploy is a way to selectively "save" the build.

Units should have a function that list what they want to save given a deploy env. This should just be a predicate, with keys possibly depending on build logics.

  b0 deploy web
  b0 deploy install
  b0 deploy -u unit install

Init

• Move away from the hardcoded stuff. Take the best of `carcass` and integrate it. Focus on files and fragments that output on stdout rather than file hierarchies.
• The command operation is confusing. I think the fragments should move to a b0 snippet command.

Documentation overhaul

• How to on action/unit execution

• B0.ml file style guide. Toplevel defs, constructor at the end except for actions. Doc strings either inline or first thing defining let (lisp style).

• Metadata convention. While it may entail user errors if certain meta is needed by build unit. It's better that the optional `meta` always provides a final override over any key.

Dump

• Profile stuff

  xcrun xctrace record --template='Time Profiler' --launch -- \
  $(b0 --path -- zipc unzip -t tmp/ebook.epub)

  Also checkout the CPU profiler.

• Add b0 cache, for trim support --fit-build as an alias to gc or mention gc in the docs. In b0-cache we need multiple log files otherwise we might nuke the driver build. Also make the log files explicit current discovery doesn't work.