Flint

Pure-Crystal LZ4 block codec.

flint implements LZ4 blocks with optional raw-content dictionaries. Public BlockCodec#compress prepends a 4-byte little-endian decoded-size header before the raw LZ4 block. #compress_raw and #decompress_raw expose raw LZ4 blocks for interoperability. The shard does not use FFI, liblz4, or a Rust extension.

Dictionary#id is metadata for callers that need to tag dictionary bytes in their own envelope. Raw blocks do not serialize or check the dictionary ID.

API

require "flint"

codec = Flint::BlockCodec.new
plain = "hello hello hello".to_slice

compressed = codec.compress(plain)
decoded = codec.decompress(compressed, max_size: plain.size)
decoded = codec.decompress(compressed)

Dictionary use:

dict = Flint::Dictionary.new("common prefix: ".to_slice)
codec = Flint::BlockCodec.new(dict)

msg = "common prefix: payload".to_slice
compressed = codec.compress(msg)
decoded = codec.decompress(compressed, max_size: msg.size)

Benchmarks

Self benchmark:

crystal run --release bench/flint_block.cr

Silesia corpus benchmark:

sh bench/silesia_compare.sh

The Silesia files download lazily into ignored corpus/silesia/. The benchmark reports compression and decompression MB/s per file plus geomean.

Optional comparison against naqvis/lz4.cr lives in bench/compare_lz4_cr. It uses that shard's frame API, because that shard is a liblz4 frame binding rather than a raw block codec.

cd bench/compare_lz4_cr
shards install
crystal run --release src/compare.cr

To compare the 84codes/lz4.cr fork, change the lz4 dependency in bench/compare_lz4_cr/shard.yml from naqvis/lz4.cr to 84codes/lz4.cr.