Bytesrw 5259ab3

Manuals

The following manuals are available:

• The quick start should get you started
• The tutorial shows you how to use and devise your own stream processors

Library bytesrw

This library has the base definition of bytes reader and writers as an extension of the Stdlib.Bytes module.

• Bytesrw Extended Stdlib.Bytes module. Open to use it.

• Bytesrw.Bytes.Slice Byte slices.
• Bytesrw.Bytes.Stream Byte streams.
• Bytesrw.Bytes.Reader Byte stream readers.
• Bytesrw.Bytes.Writer Byte stream writers.

Library bytesrw.kit

The following modules rely only the Stdlib:

• Bytesrw_utf UTF streams.

Libraries bytesrw.{blake3,md,unix,xxhash,zlib,zstd}

Each of these modules lives in its corresponding library. Compression libraries depend on their canonical C library.

• Bytesrw_blake3 BLAKE3 stream hashes.
• Bytesrw_md SHA-1 and SHA-2 stream hashes.
• Bytesrw_unix Unix file descriptor bytes readers and writers.
• Bytesrw_xxhash XXH3-64 and XXH3-128 stream hashes.
• Bytesrw_zlib deflate, zlib and gzip streams.
• Bytesrw_zstd zstd compressed streams.

Quick start

The following example compresses standard input to standard output with zstd using either a compressing bytes reader (pull) or writer (push).

open Bytesrw

let stdio_compress_reads () =
  try
    let stdin = Bytes.Reader.of_in_channel In_channel.stdin in
    let stdout = Bytes.Writer.of_out_channel Out_channel.stdout in
    let params = Bytesrw_zstd.Cctx_params.make ~checksum:true () in
    let zstdr = Bytesrw_zstd.compress_reads ~params () stdin in
    Bytes.Writer.write_reader ~eod:true stdout zstdr;
    Ok ()
  with
  | Bytes.Stream.Error e -> Bytes.Stream.error_to_result e
  | Sys_error e -> Error e

let stdio_compress_writes () =
  try
    let stdin = Bytes.Reader.of_in_channel In_channel.stdin in
    let stdout = Bytes.Writer.of_out_channel Out_channel.stdout in
    let params = Bytesrw_zstd.Cctx_params.make ~checksum:true () in
    let zstdw = Bytesrw_zstd.compress_writes ~params () ~eod:true stdout in
    Bytes.Writer.write_reader ~eod:true zstdw stdin;
    Ok ()
  with
  | Bytes.Stream.Error e -> Bytes.Stream.error_to_result e
  | Sys_error e -> Error e

Read filters can easily be applied to your strings:

let id s =
  let filters = Bytesrw_zstd.[compress_reads (); decompress_reads ()] in
  Bytes.Reader.filter_string filters s

More in the tutorial.

TODO

• Slice, add bigbytes converters.
• Push backs. Should we disallow pushing back to a negative position (in which case Bytesrw.Bytes.Reader.empty can become a value again) ? We might want to push a header before a stream but then we have Bytesrw.Bytes.Reader.append for that.
• What should we do with channel/unix reader/writer exceptions ? Convert their exceptions to Bytes.Stream.Error or not ?
• Think again about giving access to the reader/writer in read and write functions ? Is it useful ? Some combinators in Bytesrw internally mutate the read field afterwards because they want to access the definition in read, this is not possible with the API. In filters for read errors those are generally reported in the reader one filters which is available. It's a bit less clear for writers.
• Do we really need taps ? They are just Fun.id filters with a side effect.

See also the design notes.