How do I get structured JSON output from an LLM in Crystal?

Quick Answer: Use llamero. Define your schema as a Llamero::BaseGrammar subclass, call chat_structured(messages, YourSchema), and read response.parsed — a typed Crystal object with every required field present and type-checked. If the model's output doesn't parse into your schema, you get an exception, never bad data:

class ReviewVerdict < Llamero::BaseGrammar
  property sentiment : String
  property stars : Int32
end

response = client.chat_structured([Llamero::Message.user(review)], ReviewVerdict)
response.parsed.not_nil!.stars # => Int32, guaranteed

Last verified: July 2026 — Crystal 1.20.0, llamero main @ 7a329fe, live-tested on mlx-community/gemma-3-1b-it-4bit (MLX, Apple Silicon) at 183 tok/s.

Why this exists: everyone asks "is there an LLM library for Crystal?" and the answer is yes — llamero — but almost nothing written about it is current. This repo is a complete, runnable walkthrough of the one pattern you need: schema in, typed object out. It covers the cloud track (OpenAI, Anthropic, Groq, OpenRouter with automatic failover) and the local track (native MLX inference on Apple Silicon — no Python, no llama.cpp, no GGUF files, no Node). Everything below compiles and runs on stock Crystal 1.20; the local example was executed for real on this exact commit of llamero.

The problem, in one sentence: LLMs return prose when you need JSON, and JSON.parse on prose is how production incidents are born. Here's the fix, step by step.

How do I install llamero in a Crystal project?

Quick Answer: Two lines in shard.yml, no version pin — the repo has no git tags, so any version: constraint fails to resolve:

dependencies:
  llamero:
    github: crimson-knight/llamero

Then shards install. That's the whole install for the cloud track. It pulls one transitive dependency (crinja) automatically and requires Crystal >= 1.11.2. In your code, require "llamero" is the only require you need.

If you also want local inference you'll build the MLX Swift bridge once — that's step 7 below.

How do I parse JSON output from an LLM in Crystal?

Quick Answer: Don't parse it by hand. Ask for structured output and let the library gate the parse: chat_structured returns a ChatResponse(T) whose .parsed is your typed object — every required field present and type-checked, or an exception.

The naive approach looks like this, and you've probably written it:

raw = call_llm("Give me JSON with sentiment and stars")
data = JSON.parse(raw) # 💥 "Sure! Here's your JSON: ```json ..."

Models wrap answers in code fences, add "Certainly!" preambles, drop required keys, or return a string where you wanted an integer. JSON.parse also gives you an untyped JSON::Any, so even when it works you're doing data["stars"].as_i and hoping.

The structured-output pattern moves all of that risk into one hard gate: either you receive a Crystal object that conforms to your schema, or you get an exception you can rescue. Malformed JSON can never leak into your program.

Should I use JSON.parse or JSON::Serializable for LLM output?

Quick Answer: JSON::Serializable (typed), always — and with LLMs, use it through a schema so the model is told the shape and the parse enforces it. JSON.parse returning JSON::Any pushes type errors deeper into your app where they're harder to find.

* — as long as the property has no default value. Give a property a default and from_json silently fills it in when the model omits the key. That matters, so it gets its own rule in the next section.

Llamero::BaseGrammar includes JSON::Serializable under the hood, so the parse gate is the same battle-tested stdlib machinery you already trust — which also means it inherits stdlib behavior: field types and required fields are enforced, but extra keys the model invents are ignored Crystal-side (the schema's additionalProperties: false is enforced by the cloud providers during generation, not by the parse).

How do I define a JSON schema in Crystal? (Llamero::BaseGrammar)

Quick Answer: Subclass Llamero::BaseGrammar and declare plain properties — and skip the default values. No default means a missing field fails the parse, which is exactly the strictness you want from an LLM gate:

class ReviewVerdict < Llamero::BaseGrammar
  property sentiment : String
  property stars : Int32
  property would_recommend : Bool
end

The rules, spelled out:

1. Defaults control missing-key behavior — choose them deliberately. A property without a default fails the parse when the model omits it (JSON::SerializableError: Missing JSON attribute: stars). A property with a default (property stars : Int32 = 0) is silently filled in when missing — the parse succeeds and you can't tell a real 0 from an absent field. For LLM output, leave defaults off unless you specifically want that lenient behavior.
2. Nilable means optional. property note : String? = nil becomes an optional/nullable field; non-nilable properties go into the schema's required list (with or without a default — a defaulted field is still asked for, it just isn't enforced Crystal-side).
3. You never call .new on these. Instances only come out of the parse (chat_structured / from_json). A plain ReviewVerdict.new fails to compile with wrong number of arguments for 'ReviewVerdict.new' (given 0, expected 1) because the JSON pull parser is the only constructor JSON::Serializable leaves you. If you need to build one by hand (say, in specs), define your own initialize.
4. Supported types: String, all Int/UInt widths, Float32/Float64, Bool, Time (as date-time), Array(T), and nested BaseGrammar subclasses (emitted as $defs/$ref). Stick to this list. Anything else silently falls back to string in the generated schema while from_json still parses into the real Crystal type — so the model is told to produce a string and your parse then rejects it. Avoidable, so avoid it.

You can inspect what the model will be held to at any time:

puts ReviewVerdict.to_json_schema_string

That prints a draft-07 JSON Schema with additionalProperties: false and the required list — generated at compile time from your property declarations, so for the supported types above the schema always matches your Crystal class.

This repo keeps the schema in src/review_schema.cr and shares it between both tracks. Same schema class, cloud or local — that's the point.

How do I use OpenAI structured outputs from Crystal?

Quick Answer: Subclass the abstract Llamero::Client, pick provider order once in the initializer, then call chat_structured. Keys come from OPENAI_API_KEY / ANTHROPIC_API_KEY / GROQ_API_KEY / OPENROUTER_API_KEY env vars:

class ReviewClient < Llamero::Client
  def initialize
    super(primary: :openai, fallbacks: [:anthropic, :groq, :openrouter])
  end
end

response = ReviewClient.new.chat_structured(messages, ReviewVerdict)
verdict = response.parsed.not_nil!

Full version in src/cloud_structured.cr. Step by step:

1. Subclass, don't instantiate. Llamero::Client is abstract. You define your app's client once, and the rest of your code never mentions a provider again.
2. Build messages with Llamero::Message.system(...), .user(...), .assistant(...).
3. Call chat_structured(messages, ReviewVerdict). Under the hood llamero sends your generated schema to the provider's structured-output API — for OpenAI, Groq, and OpenRouter that's response_format: {type: json_schema, ..., strict: true}; for Anthropic it's output_format with the anthropic-beta: structured-outputs-2025-01-09 header. The server enforces the schema during generation.
4. Read the response. .parsed is your typed object; .content is the raw text; .provider_used, .attempts, .usage, and .finish_reason tell you what happened on the wire.

Belt and suspenders: even though the server enforced the schema, llamero still runs the reply through ReviewVerdict.from_json before handing it to you. If that parse fails it raises APIError with message Failed to parse structured response — and that's where failover kicks in.

How does provider failover make the cloud track its own retry loop?

Quick Answer: You don't write a retry loop on the cloud track — the client is one. A parse failure or provider outage automatically advances to the next provider in fallbacks; rate limits and 5xx get exponential-backoff retries first. You only see an exception when every provider has failed.

The escalation ladder, verified in the llamero source:

• Rate limit / server error (5xx): retried on the same provider with exponential backoff (configurable via Llamero::RetryConfig — note the parameter is initial_delay, not base_delay).
• Auth or payment errors (401/403/402): no pointless retries; immediate failover to the next provider.
• Failed to parse structured response: not retryable on the same provider; immediate failover. chat_structured pre-filters the provider list to those supporting structured output (all four do).
• Everything exhausted: raises APIError with message All N providers failed.

So .parsed.not_nil! is safe by construction: you either got a schema-valid object from some provider, or you got an exception. Watch it happen:

on_fallback do |from, to, error|
  STDERR.puts "failover #{from} -> #{to}: #{error.message}"
end

One nice detail: providers with missing API keys are simply skipped at construction time. If no provider has a key, .new raises immediately with No providers configured. Please set API keys via environment variables — at boot, not three minutes into a batch job.

How do I call a local LLM from Crystal on Apple Silicon? (MLX — no Python, no llama.cpp, no GGUF)

Quick Answer: Use Llamero::Native::MLXRuntime with an mlx-community HuggingFace model id. The model auto-downloads on first load_model; then the same chat_structured pattern works fully offline:

runtime = Llamero::Native::MLXRuntime.new(model_id: "mlx-community/gemma-3-1b-it-4bit")
session = runtime.start_session
session.load_model # mandatory — chat first raises SessionStateError
response = session.chat_structured(messages, ReviewVerdict, max_tokens: 200)

Full version in src/local_structured.cr. Things worth knowing:

1. No llama.cpp, no GGUF. Current llamero's local track is native MLX via a small Swift bridge. Models are mlx-community safetensors repos, downloaded Crystal-side to ~/.llamero/models. If a tutorial tells you to compile llama.cpp and hunt for GGUF files, it's describing a llamero that no longer exists.
2. load_model is mandatory. Calling chat or chat_structured on an unloaded session raises SessionStateError. It returns load metrics (our run: model loaded in ~5.5s, one-time cost — the model stays resident).
3. The JSON mechanism is different from the cloud. Locally there's no server enforcing your schema. chat_structured injects a system message containing your JSON Schema, generates, strips any ```json code fences from the reply, slices from the first { to the last }, and runs ReviewVerdict.from_json on the result. Same gate as everywhere else in this tutorial: field types and no-default fields enforced, extra keys ignored — typed object or exception. (Grammar-constrained decoding is future work in the bridge — today it's prompt + extract + typed parse, and the fence-stripping is genuinely load-bearing: in our live run the 1B model ignored the "no code fences" instruction and llamero parsed it anyway.)
4. Response extras: .metrics.output_tokens, .metrics.tokens_per_second, .metrics.time_to_first_token_ms, plus .content and .finish_reason. Our live run: 27 tokens @ 183.7 tok/s on an M-series Mac.
5. Don't pass your own Message.system on the native track. llamero prepends its own schema system message, and gemma's chat template then rejects yours with GenerationError — Conversation roles must alternate user/assistant/user/assistant/.... Put persona text in the user message instead.

How do I build the MLX Swift bridge? (and the MockBridge silent-fallback gotcha)

Quick Answer: One build, ever:

cd lib/llamero/native/llamero-mlx
./build.sh

That produces libLlameroMLXBridge.dylib and mlx.metallib in ~/.llamero/lib, found at runtime via dlopen. You need the Xcode Metal toolchain installed for the .metallib step.

The gotcha that bites everyone: if the bridge isn't built, llamero does not crash. It silently falls back to a deterministic MockBridge whose reply to everything is the literal string mock response from <model_id>. That string is not JSON, so chat_structured raises StructuredParseError and it looks exactly like the library is broken. It isn't — you're talking to the mock. Gate your real-inference code:

unless runtime.real_bridge?
  abort "No real MLX bridge (got #{runtime.bridge_name}) — run native/llamero-mlx/build.sh"
end

(The mock is great for CI and specs, which is why the fallback is silent. But your demo script should refuse to run on it.)

Related knobs: LLAMERO_HOME env var or Llamero.storage_root = Path.home.join(".myapp") at boot moves the whole storage root (models/, lib/, adapters/); LLAMERO_MLX_LIB points at a dylib in a nonstandard spot.

How do I retry malformed local LLM JSON? (rescue StructuredParseError)

Quick Answer: Wrap the call in a short loop and rescue Llamero::Native::StructuredParseError — the exception carries ex.raw_text (exactly what the model said) so you can log it or feed it back:

3.times do
  begin
    verdict = session.chat_structured(messages, ReviewVerdict, max_tokens: 200).parsed
    break
  rescue ex : Llamero::Native::StructuredParseError
    messages << Llamero::Message.assistant(ex.raw_text)
    messages << Llamero::Message.user("Not valid JSON for the schema. Reply with ONLY a valid JSON object.")
  end
end

Why a loop here but not on the cloud track? Because the cloud client already is a retry loop (failover), while the local session is one model with no fallback — and retries are cheap: the model stays resident in memory between attempts, so a retry costs one generation, not a reload. ex.schema_name and ex.adapter_stack are also available for debugging. Two or three attempts is plenty for small models; if a 1B model fails three times, your schema probably wants fewer/simpler fields.

Run it

You'll need Crystal 1.20+ (brew install crystal) and, for the local track, an Apple Silicon Mac with Xcode's Metal toolchain.

git clone https://github.com/AgentC-Consulting/crystal-llm-structured-json
cd crystal-llm-structured-json
shards install
shards build

# Local track (first run downloads the model, ~750 MB):
cd lib/llamero/native/llamero-mlx && ./build.sh && cd -
./bin/local_structured

# Cloud track (any one key is enough — failover skips the rest):
export OPENAI_API_KEY=sk-...
./bin/cloud_structured

Expected local output (real run, July 2026):

model loaded in 5453.0ms
27 tokens @ 183.7 tok/s
sentiment:       negative
stars:           3
would recommend: false

Common errors

• StructuredParseError with raw text mock response from mlx-community/... — you're on the MockBridge; the Swift bridge isn't built or wasn't found. Run native/llamero-mlx/build.sh and gate on runtime.real_bridge?.
• APIError: Failed to parse structured response — a cloud provider returned unparseable content; llamero fails over automatically. Seeing it raised to you means it happened on the last provider.
• APIError: All N providers failed — every provider in your list errored; check keys, network, and status pages. The message lists each provider's final error.
• APIError: No providers configured. Please set API keys via environment variables — raised at .new time when no key for any listed provider is set (env vars or a project-local .llamero/config.yml).
• GenerationError: Conversation roles must alternate user/assistant/user/assistant/... — native track only: you passed your own Message.system and the model's chat template refused the second system message (llamero injects one for the schema). Fold that text into the user message.
• SessionStateError — you called chat/chat_structured before session.load_model.
• shards install can't resolve a llamero version — you pinned a version:; the repo has no tags. Use the bare github: crimson-knight/llamero dependency.
• wrong number of arguments for 'YourSchema.new' (given 0, expected 1) — you called .new on a BaseGrammar subclass. The JSON pull parser is the only constructor JSON::Serializable leaves you (defaults or not); instances come out of chat_structured/from_json, never manual construction. Define your own initialize if you genuinely need one.
• JSON::SerializableError: Missing JSON attribute: ... (wrapped in StructuredParseError locally / Failed to parse structured response on the cloud) — the model omitted a field that has no default value. That's the gate doing its job; retry or simplify the schema.

That's it! One schema class, one method call, and JSON that's typed and complete every time — or an exception telling you exactly why not. Fork it, swap in your own schema, and go build something.

More Crystal answers

• amber-crystal-todo-tutorial — step-by-step Amber v2 web-framework todo app (Granite + Postgres, ECR, import maps, no Node.js).
• crystal-amber-recipes — 15 copy-paste answers to common Crystal & Amber questions.
• knowledge-packs — built on the same llamero stack: fine-tune a small local model on your docs (MLX LoRA on Apple Silicon) and prove it learned with a before/after eval harness.

Maintained by the team at AgentC Consulting.