VirtualFs

A virtual filesystem for Crystal. Mount directories, .vpak archives, .zip, and .tar into a single merged tree. Read files through a stdlib-style API — File, Dir, IO — without caring where they physically live.

Why

Game engines, desktop apps, and tools need to ship assets alongside a binary. You want to:

• Patch without rebuilding — mount a patch archive at higher priority, it shadows the originals
• Ship fewer files — pack thousands of small files into one .vpak with deduplication and compression
• Encrypt assets — per-entry AES-256-GCM or ChaCha20-Poly1305 encryption
• Read efficiently — content-defined chunking (FastCDC) enables range reads, mmap, and partial decompression
• Mount once, read everywhere — same code reads from disk, .vpak, .zip, .tar, or in-memory data

Install

# shard.yml
dependencies:
  virtualfs:
    github: naqvis/virtualfs

shards install

Quick start

require "virtualfs"

vfs = VirtualFs::Vfs.new

# Mount by path — container type inferred from extension
vfs.mount("assets/",    "/", priority: 0)   # directory
vfs.mount("patch.vpak", "/", priority: 10)  # shadows assets/ on conflict

# Read through the merged tree
vfs.exists?("config.json")          # => true
vfs.read("config.json")             # => String
vfs.open("data.bin") do |file|
  buf = Bytes.new(256)
  file.read(buf)
  file.read_range(64, 16)           # direct range read
end
vfs.ls("maps/")                     # => ["forest.map", "cave.map"]
vfs.walk("maps")                    # => recursive listing
vfs.glob("**/*.ogg")                # => ["audio/music.ogg", ...]

Build a .vpak

Via API

VirtualFs::PakFormat.build("assets.vpak", package_id: "game:1.0") do |b|
  b.add_file("src/logo.png", "logo.png", mime: "image/png")
  b.add_dir("src/audio/", mount_point: "audio")
  b.compress(:deflate)
  b.encrypt(key_provider)
end

Via CLI

# Build
vpack build assets/ --out game.vpak --compress deflate
vpack build assets/ --out secure.vpak --encrypt key.bin

# Inspect
vpack list game.vpak

# Verify
vpack verify game.vpak

Supported codecs: none, deflate, gzip, snappy, zlib. Supported ciphers: aes-256-gcm, chacha20-poly1305.

Features

Note on random access: .vpak entries, uncompressed OsDir files, and uncompressed .tar/.zip entries support true random access. Compressed .tar.gz archives are fully decompressed at mount time, and compressed .zip entries must be decompressed from the start for each seek, so these formats are best for full-file reads.

Mounting

# Priority order: highest first, first match wins
vfs.mount("base.vpak",  "/",    priority: 0)
vfs.mount("patch.vpak", "/",    priority: 10)  # shadows base on conflict
vfs.mount("dlc.vpak",   "/dlc", priority: 0)   # scoped to /dlc/ prefix

# Case-insensitive matching
vfs.mount("assets/", "/", priority: 0, case_insensitive: true)

# Direct container instances for explicit control
vfs.mount(VirtualFs::OsDir.new("assets/"), "/", priority: 0)
vfs.mount(VirtualFs::InMemory.new, "/", priority: 10)

Encryption

key = Bytes.new(32) { |i| (i + 1).to_u8 }
vfs.mount("secure.vpak", "/", priority: 0,
  key_provider: VirtualFs::Crypto::StaticKey.new(key))

Nonces are derived from (package_id, entry_hash, chunk_index) — no nonce storage, no reuse. AEAD tags verified on read; mismatch raises VirtualFs::CorruptEntry.

Compression

Codecs are stored per-entry in the .vpak format. BlockStore auto-detects on read — no need to specify codec: at mount time.

Register custom codecs:

VirtualFs::Codec.register("lz4", MyLz4Codec.new)

.vpak format

Binary, little-endian, designed for random access. See vpak-format.md for the full byte-level specification.

┌─────────────────────┐ offset 0
│ Header (128 bytes)  │  magic "VPK1\0\0\0\0", version, flags,
│                     │  index offset/length, file count, chunk size,
│                     │  dedup offset/length, tool version
├─────────────────────┤
│ Blob data           │  original file contents chunked via FastCDC;
│                     │  each chunk is compressed and/or encrypted independently
├─────────────────────┤
│ Dedup table         │  per chunk: SHA-256, offset, original length,
│                     │  compressed/encrypted length, CRC32
├─────────────────────┤
│ Index               │  per entry: path, flags, size, chunk refs,
│                     │    chunk original lengths, chunk CRC32s, mime,
│                     │    tags, attrs, timestamps, codec, cipher, mode
├─────────────────────┤
│ SHA-256 (32 bytes)  │  integrity hash of the index
├─────────────────────┤
│ Magic (8 bytes)     │  truncation check
└─────────────────────┘

Integers in the index use LEB128 varint encoding — flags, size, chunk count, tag/attr counts, timestamps, and mode all pack into 1–9 bytes. Only CRC32 values and the entry length prefix remain fixed-width.

Every unique transformed chunk is stored once. Multiple entries sharing identical compressed/encrypted content reference the same chunk data via SHA-256 hash. This is automatic — zero configuration.

Encrypted chunks use a random 12-byte nonce per chunk, stored as nonce || ciphertext || AEAD tag. The nonce is not derived from the path or chunk index, so identical plaintext does not produce identical ciphertext.

Three CRCs are stored: a whole-entry blob CRC (on-disk data), a whole-entry original data CRC, and a per-chunk CRC. The whole-entry CRCs are verified on full reads; the per-chunk CRCs are verified on random-access reads.

HTTP handler

vfs = VirtualFs::Vfs.new
vfs.mount("app.vpak", "/", priority: 10)
vfs.mount("dist/",    "/", priority: 20)

server = HTTP::Server.new([VirtualFs::StaticHandler.new(vfs)])
server.listen("127.0.0.1", 3000)

GET/HEAD, index.html for directories, Range requests for streaming, MIME detection from stdlib registry.

Architecture

Six layers, bottom-up. Cross-cutting features (compression, encryption, mmap, cache) live only in BlockStore — container drivers never decompress or decrypt.

┌─────────────────────────────────────────────────────────────────┐
│ 6. Vfs façade        mirrors Crystal stdlib: File / Dir / IO    │
├─────────────────────────────────────────────────────────────────┤
│ 5. Async + Events     open_async / prefetch;                    │
│                       on_progress / on_error callbacks on Handle│
├─────────────────────────────────────────────────────────────────┤
│ 4. MountStack         ordered read mounts + single write mount; │
│                       path to (Mount, Entry); per-file override  │
├─────────────────────────────────────────────────────────────────┤
│ 3. Index              per-mount file table: path, size, blob    │
│                       location, flags, checksums, metadata      │
├─────────────────────────────────────────────────────────────────┤
│ 2. BlockStore         the cross-cutting I/O engine:             │
│                       mmap, decompress, decrypt, dedup,          │
│                       prefetch, range/stream reads                │
├─────────────────────────────────────────────────────────────────┤
│ 1. Container drivers  Pak | Zip | Tar | OsDir | InMemory        │
│                       each yields an Index + raw byte ranges    │
└─────────────────────────────────────────────────────────────────┘

Adding a new container is cheap: produce an Index and serve raw byte ranges. The hard parts live in one testable place (BlockStore).

Error handling

All errors inherit from VirtualFs::Error < Exception and carry path and mount_id for diagnostics.

License

MIT

Contributing

1. Fork it (https://github.com/naqvis/virtualfs/fork)
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request

Contributors

• Ali Naqvi - creator and maintainer