unicode_grapheme

Unicode text segmentation for Crystal. Splits byte sequences into extended grapheme clusters per UAX #29 and reports the terminal column width of each cluster.

Built for TUI work: no allocations, no intermediate Strings, and cluster slices that point directly into the buffer you passed in.

• Conformant against the full official GraphemeBreakTest.txt suite (Unicode 17.0.0)
• Handles CRLF, combining marks, Hangul syllables, regional indicator flags, emoji ZWJ sequences, skin-tone modifiers, and Indic conjuncts (GB9c)
• Compile-time property tables, no runtime initialization and no heap allocation
• Zero dependencies

Installation

Add the dependency to your shard.yml:

dependencies:
  unicode_grapheme:
    github: shpeckman/unicode_grapheme

Then run shards install.

Usage

require "unicode_grapheme"

UW.each("héllo 🇧🇪") do |cluster, width|
  puts "#{String.new(cluster)} (#{width})"
end

Three operations, each with a String and a Bytes overload:

UW.each(input) { |cluster : Bytes, width : Int32| }
UW.width(input) : Int32
UW.count(input) : Int32

UW.count("e\u0301a\u0301")             # => 2
UW.count("\u{1F468}\u200D\u{1F469}")   # => 1

UW.width("hello")                      # => 5
UW.width("\u4E00")                     # => 2
UW.width("e\u0301")                    # => 1
UW.width("\u{1F1FA}\u{1F1F8}")         # => 2
UW.width("\t")                         # => 0

Cluster slices

each yields Bytes views into the original buffer. Nothing is copied, so a cluster is only valid for as long as the input is. Materialize with String.new(cluster) if you need to keep it:

clusters = [] of String
UW.each(line) { |cluster, _| clusters << String.new(cluster) }

Width policy

Width is computed per cluster, not per codepoint, which is why a flag or a family emoji is two columns rather than the sum of its parts.

Combining marks, joiners and variation selectors add nothing on their own, so a base character with any number of marks attached stays at the width of its base.

Invalid input

Bytes input is not assumed to be well-formed UTF-8. Overlong encodings, surrogates, out-of-range values and truncated sequences are each treated as a single-byte cluster, so iteration always advances and never loops:

UW.each(Bytes[0xFF, 0x41]) { |cluster, _| p cluster.to_a }
# => [255]
# => [65]

Version constants

UW::VERSION          # shard version, read at compile time
UW::UNICODE_VERSION  # "17.0.0"

Development

make spec          # run the spec suite
make bench         # run the benchmarks
make bench-pinned  # run them pinned to a performance core at realtime priority
make gen           # regenerate the property tables
make gen-check     # fail if the committed tables are stale
make clean         # remove caches (tool/.ucd, lib, .shards)

The spec suite downloads GraphemeBreakTest.txt for the declared UNICODE_VERSION and caches it under spec/fixtures/.

Property tables

src/unicode_grapheme/data/ holds the generated tables: a 128-byte ASCII table, a run-length encoded lo/hi/v range set, and a page index that narrows lookups to a short scan. Each entry packs the grapheme break class, Indic conjunct break class, extended pictographic flag and wide flag into one byte. Hangul syllables are computed arithmetically instead of being tabulated.

tool/gen_tables.py regenerates all of it from the UCD:

python3 tool/gen_tables.py           # write src/unicode_grapheme/data
python3 tool/gen_tables.py --check   # exit 1 if the tables are stale

It reads the target version from UNICODE_VERSION in src/unicode_grapheme.cr, pulls GraphemeBreakProperty.txt, DerivedCoreProperties.txt, emoji-data.txt and EastAsianWidth.txt, and caches them gzipped under tool/.ucd/<version>/. Python 3 standard library only.

Bumping Unicode versions is a one-line change to UNICODE_VERSION followed by make gen && make spec — the conformance fixture is fetched for the same version.

License

MIT. See LICENSE.