unicode

unicode

The unicode module provides Unicode normalization, codepoint category lookup, and character-class queries.

Import

use std::core::unicode

Functions

unicode::normalize(text: string, form: string) -> string

Normalize a string to a Unicode normalization form. form is one of "NFC", "NFD", "NFKC", or "NFKD". An unknown form is a runtime error.

// Compose any decomposed sequences into precomposed form.
let composed = unicode::normalize("café", "NFC")
print(composed)

unicode::category(codepoint: int) -> string

Get a Unicode general-category abbreviation for a codepoint. The result is one of "Lu" (uppercase letter), "Ll" (lowercase letter), "Lo" (other letter), "Nd" (decimal digit), "No" (other number), "Zs" (space separator), "Cc" (control), "Po" (punctuation), or "Cn" (unassigned / uncategorized).

print(unicode::category(65))    // "Lu"  (codepoint for 'A')
print(unicode::category(97))    // "Ll"  (codepoint for 'a')
print(unicode::category(48))    // "Nd"  (codepoint for '0')

Note

unicode::category classifies codepoints with the standard-library character predicates rather than the full Unicode Character Database, so it returns a representative subset of the general-category abbreviations rather than the complete UCD set.

unicode::is_letter(char: string) -> bool

Return true if the first character of char is alphabetic in Unicode.

print(unicode::is_letter("a"))      // true
print(unicode::is_letter("é"))      // true
print(unicode::is_letter("1"))      // false

unicode::is_digit(char: string) -> bool

Return true if the first character of char is a Unicode digit.

print(unicode::is_digit("0"))       // true
print(unicode::is_digit("9"))       // true
print(unicode::is_digit("a"))       // false

Function Reference

Function	Signature	Description
unicode::normalize(text, form)	(string, string) -> string	NFC / NFD / NFKC / NFKD normalization
unicode::category(codepoint)	(int) -> string	General-category abbreviation (Lu, Ll, Nd, …)
unicode::is_letter(char)	(string) -> bool	First character is alphabetic
unicode::is_digit(char)	(string) -> bool	First character is a digit

Grapheme segmentation

The unicode stdlib source also declares unicode::graphemes(text: string) -> Array<string>, which splits a string into Unicode grapheme clusters — the user-visible characters, which may correspond to multiple codepoints.

Caution

unicode::graphemes does not run at HEAD. Its native body builds an Array<string> return value, and the typed-marshal layer has no projection for that return shape yet — a call fails with a project_typed_return not-implemented error (tracked as the W17-marshal-return-arms follow-up). Until it ships, iterate over a string’s characters with the built-in string methods instead.

See Also

• Strings — the built-in string type and its methods