Module mem

Functions for converting between different in-RAM representations of text and for quickly checking if the Unicode Bidirectional Algorithm can be avoided.

By using slices for output, the functions here seek to enable by-register (ALU register or SIMD register as available) operations in order to outperform iterator-based conversions available in the Rust standard library.

Note: “Latin1” in this module refers to the Unicode range from U+0000 to U+00FF, inclusive, and does not refer to the windows-1252 range. This in-memory encoding is sometimes used as a storage optimization of text when UTF-16 indexing and length semantics are exposed.

The FFI binding for this module are in the encoding_c_mem crate.

Enums

Latin1Bidi

Classification of text as Latin1 (all code points are below U+0100), left-to-right with some non-Latin1 characters or as containing at least some right-to-left characters.

Functions

check_str_for_latin1_and_bidi

Checks whether a valid UTF-8 buffer contains code points that trigger right-to-left processing or is all-Latin1.

check_utf8_for_latin1_and_bidi

Checks whether a potentially invalid UTF-8 buffer contains code points that trigger right-to-left processing or is all-Latin1.

check_utf16_for_latin1_and_bidi

Checks whether a potentially invalid UTF-16 buffer contains code points that trigger right-to-left processing or is all-Latin1.

convert_latin1_to_str

Converts bytes whose unsigned value is interpreted as Unicode code point (i.e. U+0000 to U+00FF, inclusive) to UTF-8 such that the validity of the output is signaled using the Rust type system.

convert_latin1_to_str_partial

Converts bytes whose unsigned value is interpreted as Unicode code point (i.e. U+0000 to U+00FF, inclusive) to UTF-8 such that the validity of the output is signaled using the Rust type system with potentially insufficient output space.

convert_latin1_to_utf8

Converts bytes whose unsigned value is interpreted as Unicode code point (i.e. U+0000 to U+00FF, inclusive) to UTF-8.

convert_latin1_to_utf8_partial

Converts bytes whose unsigned value is interpreted as Unicode code point (i.e. U+0000 to U+00FF, inclusive) to UTF-8 with potentially insufficient output space.

convert_latin1_to_utf16

Converts bytes whose unsigned value is interpreted as Unicode code point (i.e. U+0000 to U+00FF, inclusive) to UTF-16.

convert_str_to_utf16

Converts valid UTF-8 to valid UTF-16.

convert_utf8_to_latin1_lossy

If the input is valid UTF-8 representing only Unicode code points from U+0000 to U+00FF, inclusive, converts the input into output that represents the value of each code point as the unsigned byte value of each output byte.

convert_utf8_to_utf16

Converts potentially-invalid UTF-8 to valid UTF-16 with errors replaced with the REPLACEMENT CHARACTER.

convert_utf8_to_utf16_without_replacement

Converts potentially-invalid UTF-8 to valid UTF-16 signaling on error.

convert_utf16_to_latin1_lossy

If the input is valid UTF-16 representing only Unicode code points from U+0000 to U+00FF, inclusive, converts the input into output that represents the value of each code point as the unsigned byte value of each output byte.

convert_utf16_to_str

Converts potentially-invalid UTF-16 to valid UTF-8 with errors replaced with the REPLACEMENT CHARACTER such that the validity of the output is signaled using the Rust type system.

convert_utf16_to_str_partial

Converts potentially-invalid UTF-16 to valid UTF-8 with errors replaced with the REPLACEMENT CHARACTER such that the validity of the output is signaled using the Rust type system with potentially insufficient output space.

convert_utf16_to_utf8

Converts potentially-invalid UTF-16 to valid UTF-8 with errors replaced with the REPLACEMENT CHARACTER.

convert_utf16_to_utf8_partial

Converts potentially-invalid UTF-16 to valid UTF-8 with errors replaced with the REPLACEMENT CHARACTER with potentially insufficient output space.

copy_ascii_to_ascii

Copies ASCII from source to destination up to the first non-ASCII byte (or the end of the input if it is ASCII in its entirety).

copy_ascii_to_basic_latin

Copies ASCII from source to destination zero-extending it to UTF-16 up to the first non-ASCII byte (or the end of the input if it is ASCII in its entirety).

copy_basic_latin_to_ascii

Copies Basic Latin from source to destination narrowing it to ASCII up to the first non-Basic Latin code unit (or the end of the input if it is Basic Latin in its entirety).

decode_latin1

Converts bytes whose unsigned value is interpreted as Unicode code point (i.e. U+0000 to U+00FF, inclusive) to UTF-8.

encode_latin1_lossy

If the input is valid UTF-8 representing only Unicode code points from U+0000 to U+00FF, inclusive, converts the input into output that represents the value of each code point as the unsigned byte value of each output byte.

ensure_utf16_validity

Replaces unpaired surrogates in the input with the REPLACEMENT CHARACTER.

is_ascii

Checks whether the buffer is all-ASCII.

is_basic_latin

Checks whether the buffer is all-Basic Latin (i.e. UTF-16 representing only ASCII characters).

is_char_bidi

Checks whether a scalar value triggers right-to-left processing.

is_str_bidi

Checks whether a valid UTF-8 buffer contains code points that trigger right-to-left processing.

is_str_latin1

Checks whether the buffer represents only code points less than or equal to U+00FF.

is_utf8_bidi

Checks whether a potentially-invalid UTF-8 buffer contains code points that trigger right-to-left processing.

is_utf8_latin1

Checks whether the buffer is valid UTF-8 representing only code points less than or equal to U+00FF.

is_utf16_bidi

Checks whether a UTF-16 buffer contains code points that trigger right-to-left processing.

is_utf16_code_unit_bidi

Checks whether a UTF-16 code unit triggers right-to-left processing.

is_utf16_latin1

Checks whether the buffer represents only code point less than or equal to U+00FF.

str_latin1_up_to

Returns the index of first byte that starts a non-Latin1 byte sequence, or the length of the string if there are none.

utf8_latin1_up_to

Returns the index of first byte that starts an invalid byte sequence or a non-Latin1 byte sequence, or the length of the string if there are neither.

utf16_valid_up_to

Returns the index of the first unpaired surrogate or, if the input is valid UTF-16 in its entirety, the length of the input.