Bakelite

Next-generation BakedFS, virtual filesystem, and binary container engine for Crystal.

Bakelite provides zero-copy compile-time asset embedding alongside streaming-capable, chunked-compressed storage and post-compilation binary overlay containers.

---

Features

• Dual Storage Paradigm:
  • bake: Inlines raw static bytes directly into the Crystal binary for zero-copy, instantaneous string reads. Ideal for shaders, configs, templates, and crucial YAML/JSON definitions.
  • store: Slices assets into compressed chunks (Deflate, Gzip, Zlib) streamed on demand.
• Constant Memory Streaming (Bakelite::FileIO):
  • Transparent IO implementation guaranteeing bounded memory usage ($O(\text{chunk_size})$) regardless of whether the file is 1MB or 2GB.
  • Supports random seeking (Seek::Set, Seek::Current, Seek::End) across chunk boundaries with active chunk decompression caching.
• Custom Volumes & Union Routing:
  • Isolate subsystems into named volumes (Volume) with custom mount points, priority shadowing, and unmounting.
  • Perfect for mod systems, DLC layering, and multi-tenant asset management.
• Compile-Time Content-Addressed Transforms:
  • Run built-in transformations (:crlf_to_lf, :minify_json, :strip_comments, :trim) or external CLI pipelines at compile time.
  • Results are cached by SHA-256 hash in .bakelite/cache to prevent redundant rebuild overhead.
• Large-Scale Streaming (100MB & 1GB Tested):
  • Verified with 100MB and 1GB scale suites: streaming 16,384 chunks with 64-bit offsets and bit-perfect CRC32 verification.
  • Compile-time size guard on bake warns developers when an inlined file exceeds 10MB, suggesting store to prevent binary executable bloat.
• Thread Safety & Multi-Fiber Concurrency (Bakelite::SynchronizedIO):
  • Container-level mutex synchronization guarantees race-free, atomic seek-and-read operations across concurrent fibers and OS background threads.
• Post-Compile Executable Overlay Container:
  • Pack assets into .bkl standalone archives or append them directly onto compiled executables (PE, ELF, Mach-O).
  • 32-byte fixed EOF trailer allows executables to inspect and mount their own embedded payload at runtime via mount_self!.
• Hardened Validation & Error Handling:
  • Typed exceptions (Bakelite::Error, CorruptContainerError, InvalidTrailerError, ChecksumMismatchError) with exact offsets and diagnostics.
• Ecosystem Integration:
  • Versioning powered by sol-vin/carbon.
  • ANSI TUI tables and CLI styling powered by sol-vin/opal.
  • Structured multi-track guide compiler powered by sol-vin/jasper.

---

Installation

Add bakelite to your shard.yml:

dependencies:
  bakelite:
    github: sol-vin/bakelite
    version: ~> 0.1.0

Run shards install.

---

Quick Start

1. Macro DSL Embedding

require "bakelite"

module Assets
  include Bakelite::FS

  # Direct inlined bake (zero-copy string/bytes)
  bake "config/app.yml", as_path: "app.yml"

  # Streamed chunked store (64KB chunks, Deflate compressed)
  store "media/soundtrack.ogg", chunk_size: 65536, compress: :deflate

  # Embed entire directories
  bake_folder "public/icons", prefix: "icons"
  store_folder "assets/models", prefix: "models"
end

# Read baked content directly
puts Assets["app.yml"].content

# Stream large files through standard Crystal IO
Assets.open("media/soundtrack.ogg") do |io|
  io.seek(1024)
  buffer = Bytes.new(512)
  io.read_fully(buffer)
end

2. Custom Volumes & Union Mounts

module Game
  include Bakelite::FS

  # Define a dedicated volume mounted at /mods with high priority
  volume :mods, mount: "mods", priority: 100 do
    bake "mods/pack1/rules.json", as_path: "rules.json"
  end
end

# Access files via mount prefix or direct volume reference
item = Game["mods/rules.json"]
mod_item = Game.volume(:mods)["rules.json"]

3. Declarative Multi-Volume Manifest (bake_manifest)

Declare multi-volume configurations cleanly in YAML with custom mount points, auto thresholds, and glob patterns with negative exclusions (!pattern). The manifest is processed in a single fast compile-time pass (<200ms):

# manifest.yml
volumes:
  engine:
    mount: ""
    default_chunk_size: 65536
    default_compression: deflate
    auto_threshold: 16384 # Files < 16KB use bake; >= 16KB use store
    files:
      - src/libgodot.cr
      - src/lapis.cr
      - src/libgodot/**/*.cr
      - src/bridge/**/*
      - shard.yml
      - godot-version.yml
    exclude:
      - src/main.cr
      - src/libgodot/docs/**
      - "**/*.uid"
  template:
    mount: "template"
    files:
      - template/**/*
  addon:
    mount: "addons/crystal_integration"
    files:
      - addons/crystal_integration/**/*

Bake into your application with a single call:

module EngineFS
  include Bakelite::FS

  bake_manifest "manifest.yml", base_dir: "."
end

4. Volume Extraction API

Extract isolated volumes or specific subfolders directly to disk with full overwrite protection:

# Extract the lean :engine volume into lib/lapis
EngineFS.extract_volume(:engine, "lib/lapis")

# Extract only the "scenes" folder from the template volume
EngineFS.extract_volume_folder(:template, "scenes", "my_project/scenes")

5. Programmatic Packaging API (Bakelite.pack)

Pack containers or append assets directly from your toolchain or scripts without spawning child processes:

# Pack a directory into a container or append to an executable
Bakelite.pack(
  target: "bin/game.exe",
  source_dir: "assets/",
  volume: :root,
  mount_point: "assets",
  append: true
)

# Pack an array of preconfigured volumes
vol = Bakelite::Volume.new(:levels)
# ... populate volume ...
Bakelite.pack("game_assets.bkl", volumes: [vol])

6. Appending Containers Post-Compilation

Compile your application normally:

crystal build src/main.cr -o bin/game.exe

Append assets into the binary using the bakelite CLI:

bakelite pack bin/game.exe assets/ --mount assets --append

Inside src/main.cr, mount the appended container on startup:

require "bakelite"

Bakelite.mount_self!

# All assets are now available transparently!
if item = Bakelite.get?("assets/textures/player.png")
  puts "Found asset: #{item.size} bytes"
end

---

CLI Reference

Bakelite ships with a complete CLI tool for managing containers and documentation:

# Pack a directory into a .bkl archive or append to an executable
bakelite pack <target> <dir> [--mount PATH] [--append] [--chunk-size 64KB] [--compress deflate]

# List files and volumes in a container with rich Opal tables
bakelite list <target>

# Inspect container trailer, start offset, index metadata, and mounted volumes
bakelite inspect <target>

# Verify CRC32 checksums for every chunk in a container
bakelite verify <target>

# Extract all files or a specific volume to disk
bakelite extract <target> <destination> [--volume NAME]

# Compile structured guide documentation with Jasper
bakelite docs [--src docs_src] [--out src/bakelite/docs]

# Display version information
bakelite version

---

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                       Bakelite::FS                          │
│                      (Union Router)                         │
└──────────────┬───────────────────────────────┬──────────────┘
               │                               │
       Priority 100                    Priority 0
┌──────────────▼──────────────┐ ┌──────────────▼──────────────┐
│       Volume (:dlc)         │ │       Volume (:root)        │
│   Mount: "content/dlc"      │ │       Mount: ""             │
└──────────────┬──────────────┘ └──────────────┬──────────────┘
               │                               │
       ┌───────┴───────┐               ┌───────┴───────┐
       │               │               │               │
┌──────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐
│  BakedItem  │ │ StoredItem  │ │  BakedItem  │ │ StoredItem  │
│ (Inlined)   │ │  (Chunked)  │ │ (Inlined)   │ │  (Chunked)  │
└─────────────┘ └──────┬──────┘ └─────────────┘ └──────┬──────┘
                       │                               │
              ┌────────▼────────┐             ┌────────▼────────┐
              │ Bakelite::FileIO│             │ Bakelite::FileIO│
              │ O(chunk) Memory │             │ O(chunk) Memory │
              └─────────────────┘             └─────────────────┘

---

Running Specs

# Run full test suite
crystal spec

# Or compile and run dedicated runner
crystal build spec/all_specs.cr -o bin/all_specs.exe
./bin/all_specs.exe

---

License

MIT License. Copyright (c) 2026 sol-vin.