fathom v0.1.0
fathom
Charset / encoding detection for Crystal. Pure Crystal, zero runtime dependencies, and zero-allocation hot paths.
Installation
dependencies:
fathom:
github: x-ror/fathom
version: ~> 0.1.0
Usage
Basic detection
require "fathom"
encoding = Fathom.detect(bytes) # => "WINDOWS-1251", or nil
text = String.new(bytes, encoding) if encoding
# Streaming IO (reads at most 64 KiB by default)
File.open("data.txt") do |file|
encoding = Fathom.detect(file)
end
Confidence scores & candidate lists
if match = Fathom.match(bytes)
puts "#{match.encoding} (#{match.confidence.round(2)}, bom: #{match.bom?})"
end
Fathom.detect_all(bytes).each do |candidate|
puts "#{candidate.encoding}: #{candidate.confidence.round(2)}"
end
Custom detector & extensible probers
struct MyProber
include Fathom::Prober
def probe(bytes : Bytes, stats : Fathom::ByteStats) : Fathom::Match?
# Return Fathom::Match.new("...", confidence) or nil
end
end
detector = Fathom::Detector.new(
sample_size: 32_768,
min_confidence: 0.25,
probers: Fathom::SingleByte::PROBERS + ([MyProber.new] of Fathom::Prober),
)
detector.match(bytes)
What it detects
- UTF-8, UTF-16LE/BE, and UTF-32LE/BE by BOM
- UTF-8 and ASCII without a BOM (reported as
UTF-8, including truncated trailing multi-byte sequences) - BOM-less
UTF-16LEandUTF-16BE(for ASCII/mixed-null payloads) - Cyrillic:
WINDOWS-1251,KOI8-R,KOI8-U,ISO-8859-5,IBM866 - Western European:
WINDOWS-1252
Not yet built in: Central European, Greek, Turkish, CJK (can be plugged in via Fathom::Prober).
Development
crystal spec
crystal run tools/gen_tables.cr > src/fathom/tables.cr # regenerate precomputed byte tables
crystal run --release bench/benchmark.cr # run throughput benchmark
License
MIT
Repository
fathom
Owner
Statistic
- 0
- 0
- 0
- 0
- 0
- 9 minutes ago
- November 21, 2026
License
MIT License
Links
Synced at
Sun, 22 Nov 2026 00:03:18 GMT
Languages