shortcode

crystal parser commonmark markdown static-site-generator Crystal-lang ast shortcodes

High-performance, CommonMark AST-safe positional shortcode tokenizer and recursive Markdown expander for Crystal.

CommonMark AST-Safe Positional Shortcode Tokenizer & Recursive Markdown Expander for Crystal

Overview

shortcode enables Hugo-style shortcode tags (such as {{< badge text="Beta" />}} and {{< alert type="info" >}}Inner Markdown{{< /alert >}}) within Markdown content pipelines without corrupting the underlying CommonMark AST hierarchy.

Most naive shortcode implementations use regular expressions directly on rendered HTML or raw Markdown, breaking paragraph wrapping, corrupting code blocks, or mangling HTML attributes. shortcode solves this via a two-pass AST preservation engine.

How It Works (Two-Pass AST Architecture)

 Raw Source (with {{< ... >}})
               │
               ▼
   [ 1. Shortcode::Engine.tokenize ] ──► Extracts shortcodes & masks code blocks
               │                         Replaces with deterministic HTML comments:
               │                         <!--SHORTCODE:INLINE:hash--> & <!--SHORTCODE:BLOCK:hash-->
               ▼
   [ 2. CommonMark Engine (e.g. Markd) ] ──► Compiles Markdown to HTML unharmed
               │
               ▼
   [ 3. Shortcode::Engine.expand ] ────► Recursively evaluates inner Markdown bodies
               │                         and expands placeholders into final HTML
               ▼
          Final HTML

Tokenization Pass: Before Markdown compilation, raw shortcodes outside code fences and inline backticks are replaced with deterministic HTML comment placeholders ( and ). Code blocks and tilde fences are strictly masked to prevent accidental evaluation of code examples.
CommonMark AST Pass: The Markdown parser compiles the document into HTML, passing HTML comment placeholders through unharmed without creating unwanted <p> wrapper corruptions.
Expansion Pass: Placeholders are recursively substituted with rendered HTML structures. Inner Markdown bodies inside block shortcodes are compiled via a consumer-provided MarkdownCompiler interface.

Installation

Add shortcode to your shard.yml:

dependencies:
  shortcode:
    gitlab: renich/shortcode
    version: ~> 0.1.0

Run shards install.

Quick Start

require "shortcode"
require "markd"

# 1. Define your Markdown compiler adapter
class MarkdCompiler < Shortcode::MarkdownCompiler
  def compile_to_html(markdown : String) : String
    Markd.to_html(markdown)
  end
end

compiler = MarkdCompiler.new

source = <<-MD
# Modern Architecture

Here is an inline tag: {{< badge text="v1.0" color="green" />}}

{{< alert type="warning" >}}
**Warning**: This block contains *markdown* and {{< badge text="Nested" />}}!
{{< /alert >}}
MD

# 2. Tokenize shortcodes before Markdown compilation
tokenized_markdown, shortcodes = Shortcode::Engine.tokenize(source)

# 3. Compile to HTML via standard Markdown engine
compiled_html = compiler.compile_to_html(tokenized_markdown)

# 4. Expand shortcode placeholders recursively
final_html = Shortcode::Engine.expand(compiled_html, shortcodes, compiler)

puts final_html

Custom Renderers

Provide custom handlers for project-specific shortcodes:

expanded = Shortcode::Engine.expand(compiled_html, shortcodes, compiler) do |inv, inner_html|
  case inv.name
  when "youtube"
    video_id = inv.params["id"]? || "dQw4w9WgXcQ"
    %(<div class="video-wrapper"><iframe src="https://www.youtube.com/embed/#{video_id}" allowfullscreen></iframe></div>)
  when "badge"
    %(<span class="badge badge-#{inv.params["color"]? || "info"}">#{inv.params["text"]?}</span>)
  else
    Shortcode::Engine.render_shortcode(inv, inner_html)
  end
end

API Reference

`Shortcode::Engine`

Method	Description
`tokenize(source : String, ...)`	Scans source, masks code blocks, extracts shortcodes, and injects comment tokens
`expand(html, shortcodes, compiler, &block)`	Recursively substitutes comment tokens with rendered HTML components
`render_shortcode(inv, inner_html)`	Default semantic HTML fallback renderer for built-in shortcodes