xxhash_cr

[!NOTE]
Lots of this code written by AI

Pure-Crystal implementation of the xxHash family:

• XXH32 and XXH64 — classic 32/64-bit hashes
• XXH3-64 and XXH3-128 — the modern, SIMD-friendly hashes

No runtime dependencies. Zero allocations for one-shot hashing. SIMD acceleration (AVX2 on x86_64, NEON on AArch64) with a portable scalar path on every other CPU.

Installation

Add to your shard.yml:

dependencies:
  xxhash_cr:
    github: zuf/xxhash_cr

Then run shards install.

Usage

require "xxhash_cr"

XXH32.hash("hello")        # => UInt32
XXH64.hash("hello")        # => UInt64
XXH3.hash64("hello")       # => UInt64
XXH3.hash128("hello")      # => UInt128
XXH3.hash("hello")         # alias for hash64

XXH64.hex("hello")         # => "26c7827d889f6da3"
XXH3.hex128("hello")       # => "b5e9c1ad071b3e7fc779cfaa5e523818"
XXH64.bytes("hello")       # canonical big-endian Bytes

XXH64.hash(io)             # streams any IO
XXH3.hash64(io)

Every method accepts a seed:

XXH64.hash("hello", seed: 42_u64)
XXH3.hash128("hello", seed: 42_u64)

Low-level, zero-allocation API

XXH64.hash(ptr, size, seed)          # UInt8*, Int, UInt64
XXH3.hash64(ptr, size, seed)
XXH3.hash128(ptr, size, seed)
XXH32.hash(ptr, size, seed)

These take an pointer and a byte count. They allocate nothing and do not copy the input. The input may be unaligned.

Streaming

state = XXH3::State.new
state.update("hello")
state.update(" world")
state.digest64   # => UInt64
state.digest128  # => UInt128

state.digest64   # digest does not mutate the state; keep feeding it
state.reset      # start over

XXH32::State, XXH64::State and XXH3::State are structs — they live on the stack.

Files

XXH64.file("archive.tar")           # UInt64
XXH64.file_hex("archive.tar")       # String
XXH3.file128("archive.tar")         # UInt128
XXH64.files(["a.bin", "b.bin"])     # Hash(String, String)

Files are mapped with mmap in windows (default 16 MiB). Use arg window: size to change the window. Set arg mmap: false to use buffered reads instead. The buffered-read path uses read_buffer_size: (default 1 MiB).

XXH3.file("huge.iso", window: 64 * 1024 * 1024)
XXH64.file("pipe", mmap: false, read_buffer_size: 256 * 1024)

Empty files, pipes, devices and 32-bit targets fall back to buffered reads.

Digest interface

require "xxhash_cr"

XXH64::Digest.hexdigest("hello")
XXH3::Digest128.hexdigest("hello")

digest = XXH3::Digest.new
digest.update("hello")
digest.final

Portability

Build with -Dxxhash_portable to force the generic scalar path on any target:

crystal build -Dxxhash_portable your_app.cr

The generic path is always compiled, so the library never depends on a vector ISA being present.

Development

crystal spec                 # run the suite
bash scripts/coverage.sh     # kcov line coverage into ./coverage
bash scripts/check_simd.sh   # assert the kernels still vectorize
bash scripts/gen_vectors.sh  # regenerate spec/vectors (needs libxxhash)
crystal run --release bench/bench.cr
crystal run --release bench/size_sweep.cr      # ours only, 0 B..16 MiB
crystal run --release bench/in_memory.cr           # ours vs libxxhash vs Crystal::Hasher (16 B..5 GiB)
crystal run --release bench/large_compare.cr   # 100 MiB..5 GiB vs libxxhash
crystal run --release bench/large_files.cr     # sparse 3/5 GiB files vs libxxhash
crystal run --release scripts/hash_dir_compare.cr -- /path/to/dir   # walk files vs libxxhash

License

MIT