kamil

High-Performance Decoupled Content Engine & Static Site Generator for Crystal

Overview

Kamil is a high-performance content engine and static site generator written in Crystal. It is engineered for blazing speed, concurrency safety, and zero-headache content management.

Kamil decouples content parsing from template rendering, guarantees atomic persistence via a single-writer worker fiber, suppresses filesystem feedback loops with an in-memory Anti-Echo Sentinel, and embeds a SQLite WAL database with FTS5 full-text search.

Key Design Principles & Architecture

                    ┌────────────────────────────────────────────────────────┐
                    │                      CLI Commands                      │
                    │   kamil init | build | serve | index | search          │
                    └───────────┬────────────────────────────────┬───────────┘
                                │                                │
                                ▼                                ▼
            ┌───────────────────────────────────────┐ ┌───────────────────────────────────────┐
            │         Kamil::Engine (Serve)         │ │       Kamil::Generator (Build)        │
            │  Kemal HTTP Server + /dev/reload SSE  │ │  SiteEmitter: Clean URLs + Assets     │
            └───────────────────┬───────────────────┘ └───────────────────┬───────────────────┘
                                │                                         │
                                ▼                                         ▼
            ┌───────────────────────────────────────┐ ┌───────────────────────────────────────┐
            │        Kamil::Watcher / Reconcile     │ │       Kamil::Content::Pipeline        │
            │   Recursive FS Monitor + Checksums    │ │  parse (Lifecycle 1: Ingest/Tokens)   │
            │                                       │ │  render (Lifecycle 2: Crinja/HTML)    │
            └───────────────────┬───────────────────┘ └───────────────────┬───────────────────┘
                                │                                         │
                                ▼                                         │
            ┌───────────────────────────────────────┐                     │
            │    Kamil::Persistence::Sentinel       │                     │
            │  Mutex-Guarded AntiEcho (10s TTL)     │                     │
            └───────────────────┬───────────────────┘                     │
                                │                                         │
                                ▼                                         │
            ┌───────────────────────────────────────┐                     │
            │   Kamil::Persistence::WritePipeline   │                     │
            │  Single-Writer Fiber (kamil-worker)   │                     │
            └───────────────────┬───────────────────┘                     │
                                │                                         │
                                ▼                                         ▼
            ┌─────────────────────────────────────────────────────────────────────────────────┐
            │                  Kamil::Persistence::SqliteStore (LibSQL WAL)                   │
            │      pages table (path PK, rowid, metadata, plain_body, JSON fields)           │
            │      pages_fts virtual table (FTS5 external content, triggers: ai/au/ad)        │
            └─────────────────────────────────────────────────────────────────────────────────┘

1. Decoupled 2-Stage Ingestion Pipeline:
   • Lifecycle 1 (Pipeline.parse): Ingests raw source, normalizes frontmatter metadata, extracts plaintext, tokenizes shortcodes into deterministic SHA-256 tokens, and computes CRLF-normalized SHA-256 checksums without layout or template overhead.
   • Lifecycle 2 (Pipeline.render): Evaluates Markdown through Markd with raw HTML passthrough (safe: false), recursively expands block/inline shortcodes, binds metadata to the Crinja template context, and produces static HTML.
2. Single-Writer Serialized Pipeline (WritePipeline):
   • Enqueues write mutations into a serialized channel queue processed by a dedicated kamil-write-worker fiber.
   • Executes co-located atomic temporary writes (.file.tmp.<pid>.<uuid>) with immediate File.rename on the same filesystem partition, eliminating race conditions, partial writes, and EXDEV cross-device link errors.
3. Anti-Echo Sentinel (Sentinel::AntiEcho):
   • Mutex-synchronized in-memory registry tracking impending internal writes by path and content checksum.
   • Suppresses redundant filesystem watcher events caused by internal file modifications while maintaining 10-second inline TTL eviction without auxiliary timer fibers.
4. Embedded LibSQL/SQLite WAL & FTS5 Search:
   • Enforces PRAGMA journal_mode = WAL, synchronous = NORMAL, and busy_timeout = 5000 to allow unlimited concurrent read operations alongside single-writer worker transactions.
   • Synchronizes full-text search index (pages_fts) using SQLite FTS5 external content triggers (AFTER INSERT, AFTER UPDATE, AFTER DELETE), BM25 relevance ranking, and highlighted search snippets.
5. Real-Time Development Server & SSE Live Reload:
   • High-throughput preview server built on Kemal and Freshen.
   • Multi-directory recursive file watcher monitoring content/, layouts/, and static/.
   • Debounced (50ms) Server-Sent Events hub (/dev/reload) with automatic dead-client stream reaping and automatic script injection.

Installation & Prerequisites

Prerequisites

• Crystal: 1.21.0 or later
• SQLite: 3.35.0 or later (with FTS5 support enabled)
• GNU Make: for build automation

Build from Source

# Clone the repository
git clone https://gitlab.com/renich/kamil.git
cd kamil

# Install shard dependencies
shards install

# Compile release binary
make build

# Verify build
./bin/kamil --version

Quickstart

# 1. Initialize a new Kamil site
kamil init my_site
cd my_site

# 2. Start the development server with live reload
kamil serve

# Open http://127.0.0.1:3000 in your browser.
# Edit content/index.md or layouts/default.html and watch the browser reload automatically!

# 3. Build for production
kamil build --clean
# Output is generated in public/ ready for deployment.

CLI Command Reference

kamil (Global)

Usage:
  kamil <command> [arguments] [options]

Commands:
  init [DIR]               Scaffold a new Kamil project directory structure and config
  build [DIR]              Compile content documents from database catalog to static HTML
  serve [DIR]              Start preview web server with Live-Reload SSE and file watcher
  index [DIR]              Ingest content documents into SQLite WAL database and FTS5 index
  search <QUERY>           Query full-text search index with BM25 ranking and snippets
  version                  Print Kamil version
  help                     Show this help message

Global Options:
  -v, --version            Show version information
  -h, --help               Show help for kamil or a specific subcommand

kamil init

Scaffolds a new Kamil project directory structure, default kamil.yml configuration, starter content, layout templates, static stylesheets, and initializes the SQLite WAL database.

kamil init [DIR] [options]

Generated Directory Structure:

my_site/
├── kamil.yml           # Project configuration
├── kamil.db            # SQLite WAL database & FTS5 catalog
├── content/
│   └── index.md        # Starter homepage document
├── layouts/
│   └── default.html    # Base Crinja HTML layout template
└── static/
    └── css/
        └── style.css   # Base styling

kamil build

Performs a reconciliation sweep of the content directory against the database catalog, compiles published documents into static HTML using clean URL hierarchy, and copies static assets to the output directory.

kamil build [DIR] [options]

Clean URL Mapping Rules:

• content/index.md $\rightarrow$ public/index.html
• content/about.md $\rightarrow$ public/about/index.html
• content/posts/hello.md $\rightarrow$ public/posts/hello/index.html
• content/posts/index.md $\rightarrow$ public/posts/index.html
• content/docs/api/v1/auth.md $\rightarrow$ public/docs/api/v1/auth/index.html

kamil serve

Boots the Kemal development preview web server, multi-directory file watcher, and debounced SSE live-reload hub.

kamil serve [DIR] [options]

kamil index

Reconciles the content directory with the SQLite WAL index and FTS5 catalog. Detects new, updated, unchanged, and orphaned files.

kamil index [DIR] [options]

kamil search

Queries the SQLite FTS5 inverted full-text index using BM25 relevance ranking and generates matching text snippets.

kamil search <QUERY> [options]

Example Output:

$ kamil search "crystal concurrency" --limit 2
Found 2 result(s) for 'crystal concurrency':

1. Concurrency Patterns in Crystal (/posts/concurrency-patterns/)
   Snippet: ...exploring <b>crystal</b> <b>concurrency</b> with execution contexts and channels...

2. Kamil Architecture Guide (/docs/architecture/)
   Snippet: ...demonstrates high-performance <b>crystal</b> <b>concurrency</b> models...

Configuration (kamil.yml)

Kamil uses kamil.yml in the project root directory. If omitted, safe defaults are applied.

# Kamil Site Configuration

site:
  title: "My Kamil Site"      # Site title
  base_url: ""               # Base URL for deployment (e.g. "https://example.com")
  language: "en"             # Site language code

paths:
  content: "content"         # Directory containing markdown/html content
  layouts: "layouts"         # Directory containing Crinja HTML layout templates
  static: "static"           # Directory containing static CSS, JS, images
  output: "public"           # Directory where static distribution is emitted

database:
  url: "sqlite3://kamil.db"  # SQLite database connection URL
  wal: true                  # Enable SQLite WAL (Write-Ahead Logging) mode
  busy_timeout: 5000         # SQLite busy timeout in milliseconds

server:
  host: "127.0.0.1"          # Preview server bind address
  port: 3000                 # Preview server port
  live_reload: true          # Enable Server-Sent Events live-reload hub
  debounce_ms: 50            # Live reload debounce coalescing window (ms)
  poll_interval_ms: 100      # Filesystem watcher poll interval (ms)

Precedence Hierarchy

1. Explicit CLI Flags (--port, --output, --content-dir, --db, etc.)
2. Explicit Config File (--config my_custom_config.yml)
3. Auto-Discovered Config File (./kamil.yml)
4. Built-in Defaults (Kamil::Config.default)

Content Authoring

Frontmatter & Supported Formats

Kamil supports Markdown (.md, .markdown) and raw HTML (.html, .htm) content files. Every content document supports YAML or JSON frontmatter enclosed within opening and closing --- fences (with automatic UTF-8 BOM tolerance):

---
title: "Getting Started with Kamil"
slug: "getting-started"
status: "Published"
published_at: 2026-08-22T10:00:00Z
updated_at: 2026-08-22T12:00:00Z
taxonomies:
  tags:
    - "crystal"
    - "ssg"
    - "fast"
  categories:
    - "guides"
custom_fields:
  author: "Rénich Bon Ćirić"
  layout: "default"
  featured: true
---

# Getting Started with Kamil

Welcome to the documentation for Kamil!

Automatic Fallback Rules

• Title: Uses explicit title $\rightarrow$ first level-1 heading (# Title) $\rightarrow$ filename stem (e.g. my-post.md $\rightarrow$ "my-post") $\rightarrow$ "Untitled".
• Slug: Uses explicit slug $\rightarrow$ slugified title $\rightarrow$ filename stem.
• Status: Defaults to Published (supports Draft, Published, Scheduled).
• Timestamps: Parses ISO8601, RFC3339, and standard date formats, normalized to UTC.

Layout Templates (Crinja)

Kamil uses Crinja (Jinja2 / Django template engine for Crystal) for layout rendering. Supported template extensions in layouts/ include .html, .htm, .j2, .jinja, .jinja2, and .crinja.

Template Example (layouts/default.html or layouts/default.j2):

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>{{ page.title }}</title>
  <link rel="stylesheet" href="/css/style.css">
</head>
<body>
  <header>
    <h1>{{ page.title }}</h1>
    {% if page.taxonomies.tags %}
    <ul class="tags">
      {% for tag in page.taxonomies.tags %}
      <li><span class="tag">{{ tag }}</span></li>
      {% endfor %}
    </ul>
    {% endif %}
  </header>

  <main id="content">
    {{ page.body }}
  </main>

  <footer>
    <p>Published on {{ page.published_at }} by {{ page.custom_fields.author }}</p>
  </footer>
</body>
</html>

Crinja Context Variables

• page.title (String): Document title
• page.slug (String): Clean URL slug
• page.body (String): Fully evaluated HTML content with expanded shortcodes
• page.format (String): Markup format (Markdown or HTML)
• page.status (String): Publication status (Draft, Published, Scheduled)
• page.published_at (String): RFC3339 UTC publication timestamp
• page.updated_at (String): RFC3339 UTC last updated timestamp
• page.checksum (String): Full-source normalized SHA-256 checksum
• page.taxonomies (Hash): Map of taxonomy names to string arrays (e.g. page.taxonomies.tags)
• page.custom_fields (Hash): Map of custom key-value pairs (e.g. page.custom_fields.author)

Shortcodes

Kamil provides a robust shortcode engine supporting positional tokenization, CommonMark Type-2 HTML block isolation, and recursive inner markdown evaluation.

Block Shortcodes

Block shortcodes enclose content between opening and closing tags. Inner content is compiled from Markdown to HTML recursively.

{{< alert type="warning" title="Important Notice" >}}
This is an alert box with **Markdown** formatting and a [link](https://crystal-lang.org).
{{< /alert >}}

Rendered HTML Output:

<div class="kamil-shortcode kamil-alert" data-title="Important Notice" data-type="warning">
<p>This is an alert box with <strong>Markdown</strong> formatting and a <a href="https://crystal-lang.org">link</a>.</p>
</div>

Inline Shortcodes

Inline shortcodes are self-closing:

Here is a badge: {{< badge text="v1.0" />}} and an icon: {{< icon name="star" />}}.

Rendered HTML Output:

Here is a badge: <span class="kamil-badge">v1.0</span> and an icon: <span class="kamil-icon" data-name="star"></span>.

Shortcode Isolation & Code Blocks

Shortcodes located within fenced code blocks (```) or inline backticks (`) are automatically pre-masked and preserved verbatim as code without being tokenized or expanded.

Development & Verification

Build Automation (GNUmakefile)

# Compile release binary
make build

# Run test suite
make spec

# Run Ameba static linter
make lint

# Auto-format codebase
make format

# Clean build artifacts and databases
make clean

Quality Guarantees

• Zero Unsafe Pragmas: 100% free of .not_nil!, # ameba:disable, and # ameba:ignore.
• SQL Injection Immune: 100% of SQLite database queries use parameterized SQL placeholders (?).
• Path Traversal Immune: Directory traversal boundary checks prevent arbitrary file reads/writes (../).
• 100% Spec Pass: Complete suite of 417+ unit, integration, and adversarial concurrency specs.

Support & Donation

If you find kamil or other tools in this ecosystem valuable, consider supporting continued FOSS development:

• Liberapay: liberapay.com/renich

Contributing

Contributions are welcome! Please review our Contributing Guidelines for instructions on our Red-Green-Refactor TDD workflow, coding standards, and pull request procedures.

Changelog

Detailed release history and changelogs are documented in CHANGELOG.rst.

License

GNU Affero General Public License v3.0 or later (AGPL-3.0-or-later).

Copyright © 2026 Rénich Bon Ćirić and Contributors.