Module Serialk_text.Tdec

UTF-8 text decoder.

A decoder inputs valid UTF-8 text, maintains text locations according to advances in the input and has a lexeme buffer for lexing.

Decoder

type t

The type for UTF-8 text decoders.

val from_string : ?⁠file:Tloc.fpath -> string -> t

from_string ~file s decodes s using file (defaults to Tloc.no_file) for text location.

Locations

val file : t -> Tloc.fpath

file d is the input file.

val pos : t -> Tloc.pos

pos d is the current decoding byte position.

val line : t -> Tloc.line_pos

line d is the current decoding line position. Lines increment as described here.

val loc : t -> first_byte:Tloc.pos -> last_byte:Tloc.pos -> first_line:Tloc.line_pos -> last_line:Tloc.line_pos -> Tloc.t

loc d ~first_byte ~last_bytex ~first_line ~last_line is Tloc.v using file d for the file.

val loc_to_here : t -> first_byte:Tloc.pos -> first_line:Tloc.line_pos -> Tloc.t

loc_to_here d ~first_byte ~first_line is a location that starts at ~first_byte and ~first_line and ends at the current decoding position as determined by pos and line.

val loc_here : t -> Tloc.t

loc_here d is like loc_to_here with the start position at the current decoding position as determined by pos and line.

Errors

exception Err of Tloc.t * string

The exception for errors. A location and an english error message

val err : Tloc.t -> string -> 'b

err loc msg raises Err (loc, msg) with no trace.

val err_to_here : t -> first_byte:Tloc.pos -> first_line:Tloc.line_pos -> ('a, Stdlib.Format.formatter, unit, 'b) Stdlib.format4 -> 'a

err_to_here d ~first_byte ~first_line fmt ... is err d (loc_to_here d ~first_byte ~first_line) fmt ...

val err_here : t -> ('a, Stdlib.Format.formatter, unit, 'b) Stdlib.format4 -> 'a

err_here d is err d (loc_here d) fmt ....

Error message helpers

val err_suggest : ?⁠dist:int -> string list -> string -> string list

err_suggest ~dist candidates s are the elements of candidates whose edit distance is the smallest to s and at most at a distance of dist of s (defaults to 2). If multiple results are returned the order of candidates is preserved.

type 'a fmt = Stdlib.Format.formatter -> 'a -> unit

The type for formatters.

val pp_and_enum : ?⁠empty:unit fmt -> 'a fmt -> 'a list fmt

and_enum ~empty pp_v ppf l formats l according to its length.

  • 0, formats empty (defaults to nop).
  • 1, formats the element with pp_v.
  • 2, formats "%a and %a" with the list elements
  • n, formats "%a, ... and %a" with the list elements
val pp_or_enum : ?⁠empty:unit fmt -> 'a fmt -> 'a list fmt

or_enum is like and_enum but uses "or" instead of "and".

val pp_did_you_mean : 'a fmt -> 'a list fmt

did_you_mean pp_v formats "Did you mean %a ?" with or_enum if the list is non-empty and nop otherwise.

val pp_must_be : 'a fmt -> 'a list fmt

must_be pp_v formats "Must be %a." with or_enum if the list is non-empty and nop otherwise.

val pp_unknown : kind:unit fmt -> 'a fmt -> 'a fmt

unknown ~kind pp_v formats "Unknown %a %a." kind () pp_v.

val pp_unknown' : kind:unit fmt -> 'a fmt -> hint:('a fmt -> 'a list fmt) -> ('a * 'a list) fmt

unknown ~kind pp_v ~hint (v, hints) formats unknown followed by a space and hint pp_v hints if hints is non-empty.

Decoding

val eoi : t -> bool

eoi d is true iff the decoder is at the end of input.

val byte : t -> int

byte d is the byte at current position or 0xFFFF if eoi d is true.

val accept_uchar : t -> unit

accept_uchar d accepts an UTF-8 encoded character starting at the current position and moves to the byte location after it. Raises Err in case of UTF-8 decoding error.

val accept_byte : t -> unit

accept_byte d accepts the byte at the current position and moves to the byte location after it. Warning. Faster than accept_uchar but the client needs to make sure it's not accepting invalid UTF-8 data, i.e. that byte d is an US-ASCII encoded character (i.e. <= 0x7F).

Lexeme buffer

val lex_clear : t -> unit

lex_clear d sets the lexeme to the empty string.

val lex_pop : t -> string

lex_pop d is the lexeme and lex_clears it.

val lex_add_byte : t -> int -> unit

lex_add_byte d b adds byte b to the lexen.

val lex_add_bytes : t -> string -> unit

lex_add_byte d s adds bytes s to the lexen.

val lex_add_char : t -> char -> unit

lex_add_char d c adds character c to the lexen.

val lex_add_uchar : t -> Stdlib.Uchar.t -> unit

lex_add_uchar t u adds the UTF-8 encoding of character u to the lexen.

val lex_accept_uchar : t -> unit

lex_accept_uchar d is like accept_uchar but also adds the UTF-8 byte sequence to the lexeme.

val lex_accept_byte : t -> unit

lex_accept_byte d is like accept_byte but also adds the byte to the lexeme. Warning. accept_byte's warning applies.