B00 manual

Design considerations

Kernel extraction from B0

Concepts kept:

Concepts dropped and to be seen how we can recover them at a higher level:

TODO and resolve

Handling removals

Knowing files before the build starts doesn't fit the build model, besides removals is under the responsability of `Memo` clients (e.g. for now `odig` doesn't deal with it). The approach currently taken is the first one below. The second one could be considered but hampers correctness.

  1. Clean builds. Usually builds are performed in a dedicated directory, always delete the previous directory at the beginning of the build and restart from scratch. Deleting is slow and should not impede the build, hence the B00.Memo.trash that simply renames to a trash directory and the asynchronous deletion at the end of the build.
  2. Diffing. If we know which files were generated in the previous run (easy to get from the build build log), we can remove stale artefacts after the new run has been completed (and do a clean build if you lost the info). This may however lead to incorrect builds (`-I` problems) where tool may pickup stale artefacts

A middle ground between the two could be the following. Currently we trash paths in a dedicated directory and delete it asynchronously at the end of the build as this may be slow. However we could track the renames to the path directory. When we revive from the cache to a given path one can check if the path exists in the trash with the same inode number, if it does we can rename it from there to the build rather than create a new link.

The nice thing of this approach is that at the end of the build we get in the trash exactly the things that really need to be deleted, rather than delete all the build all of the time (note that this is still more than say a `make` based build system because those overwrites paths).

Design answers

A few explanation about design choices we may not remember in the future.

Why arent't file permission of a read file not part of the build operation stamp ?

It is clearly something that can influence tool outputs and failures. However so can any bit of file metadata (e.g. groups) in general and a boundary needs to be drawn. For possible wide area cache sharing scenarios it also feels better not to consider, say changes in other file permissions.

Related to the answer about caching failures.

Why are failed operations not cached ?

First let's define what caching build operations would likely mean. It would mean caching it's metadata bit, i.e. for tool spawns the user interface output and the exit code. Caching the actual (likely partial) file writes would be difficult to do since the actual file system state on tool error may be difficult to characterize.

Caching failed operations may report build errors more quicky for example if a build operation is slow in erroring and nothing changed as far as its operation stamp indicates.

One problem though is is that the failure may be due to a reason that is not captured by the operation stamp. In that case even though the problem may be resolved by the user, the operation stamp would not change and the operation would be revived as a failure rather than re-executed.

To give a concrete example, suppose an operation fails because of the file permissions of a file it reads. This file permission is not part of the build operation stamp (see this answer). So changing the file permission to make the build operation succeed would not only revive the old failure, but likely also output a puzzling error message unrelated to the current state of the system. The way out for the user is long head scratching ended by a clear of the cache for that operation. That doesn't feel like a great user experience.

It seems not caching failures leads to a better user experience for b0 based build systems, at least as long as this remains true. Also build failure information should be, most of the time, available in the build logs so it may not be necessary to reinvoke a long failing operation to get its error back.