VirtualFs

A virtual filesystem for Crystal. Mount directories, .vpak archives, .zip, and .tar into a single merged tree. Read files through a stdlib-style API — File, Dir, IO — without caring where they physically live.

Why

Game engines, desktop apps, and tools need to ship assets alongside a binary. You want to:

• Patch without rebuilding — mount a patch archive at higher priority, it shadows the originals
• Ship fewer files — pack thousands of small files into one .vpak with deduplication and compression
• Encrypt assets — per-entry AES-256-GCM or ChaCha20-Poly1305 encryption
• Read efficiently — content-defined chunking (FastCDC) enables range reads, mmap, and partial decompression
• Mount once, read everywhere — same code reads from disk, .vpak, .zip, .tar, or in-memory data

Install

# shard.yml
dependencies:
  virtualfs:
    github: naqvis/virtualfs

shards install

Quick start

require "virtualfs"

vfs = VirtualFs::Vfs.new

# Mount by path — container type inferred from extension
vfs.mount("assets/",    "/", priority: 0)   # directory
vfs.mount("patch.vpak", "/", priority: 10)  # shadows assets/ on conflict

# Read through the merged tree
vfs.exists?("config.json")          # => true
vfs.read("config.json")             # => String
vfs.open("data.bin") do |file|
  buf = Bytes.new(256)
  file.read(buf)
  file.read_range(64, 16)           # direct range read
end
vfs.ls("maps/")                     # => ["forest.map", "cave.map"]
vfs.walk("maps")                    # => recursive listing
vfs.glob("**/*.ogg")                # => ["audio/music.ogg", ...]

Build a .vpak

Via API

VirtualFs::PakFormat.build("assets.vpak", package_id: "game:1.0") do |b|
  b.add_file("src/logo.png", "logo.png", mime: "image/png")
  b.add_dir("src/audio/", mount_point: "audio")
  b.compress(:deflate)
  b.encrypt(key_provider)
end

Via CLI

# Build (--parallel uses the fork_join execution context by default)
vpack build assets/ --out game.vpak --compress deflate --parallel 4
vpack build assets/ --out secure.vpak --encrypt key.bin

# Inspect
vpack list game.vpak
vpack list game.vpak --json            # machine-readable manifest

# Extract
vpack extract game.vpak --out dist/ --key key.bin

# Update (add/replace/remove entries atomically)
vpack update game.vpak --add new_asset.png=assets/new_asset.png --remove obsolete.txt

# Delta patches
vpack diff game.vpak game2.vpak --out patch.vpch
vpack apply game.vpak patch.vpch --out game3.vpak

# Signing
vpack keygen --private key.bin --public pub.bin
vpack sign game.vpak --private-key key.bin
vpack verify game.vpak --public-key pub.bin --key key.bin

# Deep verification
vpack fsck game.vpak --public-key pub.bin --key key.bin

Supported codecs: none, deflate, gzip, snappy, zlib. Supported ciphers: aes-256-gcm, chacha20-poly1305.

Features

Note on random access: .vpak entries, uncompressed OsDir files, and uncompressed .tar/.zip entries support true random access. Compressed .tar.gz archives are fully decompressed at mount time, and compressed .zip entries must be decompressed from the start for each seek, so these formats are best for full-file reads.

Mounting

# Priority order: highest first, first match wins
vfs.mount("base.vpak",  "/",    priority: 0)
vfs.mount("patch.vpak", "/",    priority: 10)  # shadows base on conflict
vfs.mount("dlc.vpak",   "/dlc", priority: 0)   # scoped to /dlc/ prefix

# Case-insensitive matching
vfs.mount("assets/", "/", priority: 0, case_insensitive: true)

# Direct container instances for explicit control
vfs.mount(VirtualFs::OsDir.new("assets/"), "/", priority: 0)
vfs.mount(VirtualFs::InMemory.new, "/", priority: 10)

# Mount with signature verification (Ed25519 public key)
vfs.mount("release.vpak", "/", priority: 0, verify_signature: public_key)

Live reload

mount_watch mounts a directory and watches it for changes; the index is refreshed automatically and callbacks fire per change.

vfs.mount_watch("assets/", "/", interval: 300.milliseconds) do |path, kind|
  puts "#{kind} #{path}"
end

# Or register a global callback for all watched mounts
vfs.on_change { |path, kind| reload_scene(path) if kind.created? }

The watcher uses ReadDirectoryChangesW (Windows), inotify (Linux), or kqueue (macOS/BSD) with a portable polling backend everywhere else — same API on every platform. You can force a backend explicitly:

VirtualFs::Watcher.start(["assets/"], 300.milliseconds, backend: :poll) { |c| ... }
vfs.mount_watch("assets/", "/", backend: :poll)

backend: :auto (default) selects the platform-native implementation; :poll forces the portable polling engine.

Streaming reads

vfs.read("data.bin") { |file| file.read_at(0, 16) }  # auto-close, range read
vfs.each_line("log.txt") { |line| process(line) }
file = vfs.open("config.json")
file.size # => Int64, file.path, file.stat

Vfs#prefetch runs on a bounded fiber pool and reports completion:

ch = vfs.prefetch(["a.bin", "b.bin", "c.bin"])
result = ch.receive  # true, or the first Exception

Encryption

key = Bytes.new(32) { |i| (i + 1).to_u8 }
vfs.mount("secure.vpak", "/", priority: 0,
  key_provider: VirtualFs::Crypto::StaticKey.new(key))

Nonces are derived from (package_id, entry_hash, chunk_index) — no nonce storage, no reuse. AEAD tags verified on read; mismatch raises VirtualFs::CorruptEntry.

Compression

Codecs are stored per-entry in the .vpak format. BlockStore auto-detects on read — no need to specify codec: at mount time.

Register custom codecs:

VirtualFs::Codec.register("lz4", MyLz4Codec.new)

.vpak format

Binary, little-endian, designed for random access. See vpak-format.md for the full byte-level specification.

┌─────────────────────┐ offset 0
│ Header (128 bytes)  │  magic "VPK1\0\0\0\0", version, flags,
│                     │  index offset/length, file count, chunk size,
│                     │  dedup offset/length, tool version
├─────────────────────┤
│ Blob data           │  original file contents chunked via FastCDC;
│                     │  each chunk is compressed and/or encrypted independently
├─────────────────────┤
│ Dedup table         │  per chunk: SHA-256, offset, original length,
│                     │  compressed/encrypted length, CRC32
├─────────────────────┤
│ Index               │  per entry: path, flags, size, chunk refs,
│                     │    chunk original lengths, chunk CRC32s, mime,
│                     │    tags, attrs, timestamps, codec, cipher, mode
├─────────────────────┤
│ SHA-256 (32 bytes)  │  integrity hash of the index
├─────────────────────┤
│ Magic (8 bytes)     │  truncation check
└─────────────────────┘

Integers in the index use LEB128 varint encoding — flags, size, chunk count, tag/attr counts, timestamps, and mode all pack into 1–9 bytes. Only CRC32 values and the entry length prefix remain fixed-width.

Every unique transformed chunk is stored once. Multiple entries sharing identical compressed/encrypted content reference the same chunk data via SHA-256 hash. This is automatic — zero configuration.

Encrypted chunks use a random 12-byte nonce per chunk, stored as nonce || ciphertext || AEAD tag. The nonce is not derived from the path or chunk index, so identical plaintext does not produce identical ciphertext.

Three CRCs are stored: a whole-entry blob CRC (on-disk data), a whole-entry original data CRC, and a per-chunk CRC. The whole-entry CRCs are verified on full reads; the per-chunk CRCs are verified on random-access reads.

Updating packages

PakFormat.update edits an existing .vpak atomically (temp file + rename). Surviving entries keep their stored chunk bytes — no re-compression, no re-encryption, no re-reading of source files.

VirtualFs::PakFormat.update("game.vpak") do |editor|
  editor.add_file("new_asset.png", "assets/new_asset.png")
  editor.remove("obsolete.txt")
end

Delta patching

Because chunks are content-addressed, patches carry only the chunks the target has that the base lacks — plus the target index.

VirtualFs::Delta.diff("game.vpak", "game2.vpak", "patch.vpch")
VirtualFs::Delta.apply("game.vpak", "patch.vpch", "game3.vpak")

See vpch-format.md for the byte-level patch format.

Signing

Packages can be signed with Ed25519; the signature covers the index hash and package metadata and is verified at mount time when a public key is given.

seed, public = VirtualFs::Crypto::Ed25519.generate
VirtualFs::PakFormat.sign("game.vpak", seed)
vfs.mount("game.vpak", "/", verify_signature: public)

VirtualFs::PakFormat.verify("game.vpak", key_provider: kp, public_key: public)

Every package also stores a per-entry whole-file SHA-256, so verify/fsck can check encrypted entries (when a key is supplied) with a cryptographic hash instead of CRC32.

Remote containers

A .vpak served over plain HTTP Range requests can be mounted directly — only the header/index is fetched up front; file data streams chunk-by-chunk on demand. The server side is just StaticHandler (or any Range-capable CDN/object store).

remote = VirtualFs::RemoteContainer.new("https://cdn.example.com/game.vpak")
vfs.mount(remote, "/", key_provider: keyring)
vfs.read("assets/scene.bin") # fetched on demand

Archive live-reload

mount_watch_file watches an archive file and transparently re-mounts it when replaced — iterate on packed assets without restarting:

vfs.mount_watch_file("game.vpak", "/", interval: 300.milliseconds) do |path, kind|
  puts "re-mounted #{path} after #{kind}"
end

Custom containers

VirtualFs::Vfs.register_container(".pak") do |path|
  MyPakFormat.new(path)
end
vfs.mount("assets/levels.pak", "/levels", priority: 0)

Multi-key encryption

keyring = VirtualFs::Crypto::Keyring.new(
  {"team-a" => key_a, "team-b" => key_b},
  default: default_key,
)
VirtualFs::PakFormat.build("game.vpak", key_provider: keyring) do |b|
  b.add("proprietary.dat", data, flags: VirtualFs::EntryFlags::Encrypted, key_id: "team-a")
end
vfs.mount("game.vpak", "/", key_provider: keyring)

Observability

vfs.stats.snapshot # => {reads: 42, bytes_read: 123456, cache_hits: 7}

FUSE mounting

The FUSE adapter lives in its own repo: virtualfs_fuse — refer to it for usage, platform requirements, and the real-mount integration test.

Parallel builds

parallel: transforms chunks on a dedicated multi-threaded execution context. The default is the fork_join context (eager threads + work-stealing); you can supply any Fiber::ExecutionContext (e.g. the stdlib Parallel) to control thread lifetime yourself:

VirtualFs::PakFormat.build("game.vpak", parallel: 4) { |b| ... }

# or with a caller-owned context (stdlib or fork_join)
context = ForkJoin::ExecutionContext.new("build", 4)
VirtualFs::PakFormat.build("game.vpak", parallel: 4, context: context) { |b| ... }
context.shutdown

A Builder-created context is shut down automatically when the build finishes; caller-provided contexts stay open.

HTTP handler

vfs = VirtualFs::Vfs.new
vfs.mount("app.vpak", "/", priority: 10)
vfs.mount("dist/",    "/", priority: 20)

server = HTTP::Server.new([VirtualFs::StaticHandler.new(vfs)])
server.listen("127.0.0.1", 3000)

GET/HEAD, index.html for directories, single- and multi-range Range requests (streamed), MIME detection from the stdlib registry, ETag/Last-Modified conditional requests with 304 Not Modified, gzip content negotiation for compressible types, and an optional HTML directory listing (directory_listing: true). A cache_control: option sets Cache-Control on responses.

Architecture

Six layers, bottom-up. Cross-cutting features (compression, encryption, mmap, cache) live only in BlockStore — container drivers never decompress or decrypt.

┌─────────────────────────────────────────────────────────────────┐
│ 6. Vfs façade        mirrors Crystal stdlib: File / Dir / IO;    │
│                       streaming reads, copy/rename/append        │
├─────────────────────────────────────────────────────────────────┤
│ 5. Async + Events     AsyncPool (bounded), open_async, prefetch, │
│                       Watcher (live reload), on_change/on_error  │
├─────────────────────────────────────────────────────────────────┤
│ 4. MountStack         ordered mounts + write mount; PathTrie     │
│                       prefix index for O(depth) resolve/ls/glob  │
├─────────────────────────────────────────────────────────────────┤
│ 3. Index              per-mount file table: path, size, blob     │
│                       location, flags, checksums, metadata       │
│                       (+ PathTrie for subtree queries)           │
├─────────────────────────────────────────────────────────────────┤
│ 2. BlockStore         the cross-cutting I/O engine: mmap,        │
│                       decompress, decrypt, dedup, prefetch,      │
│                       range/stream reads, byte-budget LRU cache  │
├─────────────────────────────────────────────────────────────────┤
│ 1. Container drivers  Pak | Zip | Tar | OsDir | InMemory         │
│                       each yields an Index + raw byte ranges     │
└─────────────────────────────────────────────────────────────────┘

Adding a new container is cheap: produce an Index and serve raw byte ranges. The hard parts live in one testable place (BlockStore).

Error handling

All errors inherit from VirtualFs::Error < Exception and carry path and mount_id for diagnostics.

Security hardening

• Archive entry names are validated at scan time (PathValidator): absolute paths, .. segments, NUL bytes, and backslashes are rejected, closing zip-slip / tar-slip traversal. Read/write paths are normalized the same way.
• Index parsing is bounds-checked (entry lengths, varint width, chunk/tag counts) so hostile packages cannot force huge allocations or infinite loops.
• Decompression is capped per entry and per chunk (DecompressionBomb).
• Sequential reads of chunked compressed/encrypted entries stream chunk by chunk — each chunk is read and transformed at most once per pass.
• Whole-file SHA-256 + optional Ed25519 signatures authenticate content and origin; fsck verifies every entry, including encrypted ones with a key.

Testing on other platforms

The suite is fully cross-platform — run crystal spec on the OS you target. The watcher specs (spec/watcher_spec.cr) run every core scenario twice: against the platform-native backend (:auto) and the portable polling backend (:poll), so the OS-specific implementation is exercised simply by running the suite there:

spec/watcher_platform_spec.cr adds platform-specific coverage — unicode filenames, new-subdirectory discovery, rapid change batches, and handle release on restart — where only the block matching the current OS compiles and runs.

License

MIT

Contributing

1. Fork it (https://github.com/naqvis/virtualfs/fork)
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request

Contributors

• Ali Naqvi - creator and maintainer