Knowledge Packs

Fine-tune a small local model on your docs — and prove it actually learned something.

A knowledge pack is a versioned, portable directory that compiles a body of expertise into four things: curated sources, schema-validated generated training JSONL, a contamination-guarded held-out eval set, and a JSON manifest (hashes, provenance, recipe, base-model compatibility, license). You "install" a pack by LoRA-fine-tuning a small local model on Apple Silicon (MLX), and you prove it worked by running the same eval harness before and after.

packsmith is the Crystal CLI that builds, verifies, evaluates and diffs them.

Long-form guide with the measured numbers: https://agentc-consulting.github.io/knowledge-packs/

packsmith build packs/amber-framework      # chunk → split → generate → guard → emit
packsmith verify packs/amber-framework     # hashes, schemas, contamination — fail closed
packsmith eval packs/amber-framework --model BASE --out reports/base.eval.json
packsmith train packs/amber-framework --base BASE
packsmith eval packs/amber-framework --model BASE --adapter ADAPTER --out reports/adapter.eval.json
packsmith diff reports/base.eval.json reports/adapter.eval.json --fail-below-delta 0.10

The problem

Every local fine-tuning tutorial ends the same way: train a LoRA, run one prompt, eyeball the output, declare victory.

Here is what that hides. On our reference machine (M1 Max, 32 GB), we trained mlx-community/SmolLM-135M-Instruct-4bit twice on the identical 471-pair dataset for the identical 2000 iterations — only the learning rate differed. With mlx_lm's default lr (1e-5), validation loss walked down from 4.94 to 1.96 — a beautiful curve — and recall of trained facts barely moved: 2/20 vs the base model's 1/20. With lr 1e-4, validation loss reached 0.53 and recall of trained facts went to 11/20. Both per-item eval reports are checked in (live-run/reports/) so you can audit every answer.

Loss going down is not knowledge going in. If your pipeline can't measure the difference, you don't have a pipeline — you have a ritual.

The idea

Treat "teach a model X" as a build artifact with provenance, and make measurement part of the format itself:

my-pack/
  pack.manifest.json          format + provenance + split recipe + sha256 file table
  sources/curated/*.md        the source material (license + attribution recorded)
  sources/source_index.jsonl  every source unit: raw & normalized sha256, section, license
  generated/train.jsonl       schema-validated training records, each citing its source units
  generated/formats/…         mlx / alpaca / sharegpt rendered views
  eval/questions.jsonl        held-out questions (answer-free view)
  eval/answer_key.jsonl       the answers, quotes and scoring rules
  reports/*.eval.json         before/after eval reports, per-item, published as-is

Three rules make it honest:

1. Split before generate. Source units are assigned to train/valid/eval by a salted hash of the unit id, before any example is generated. Eval questions can only be generated from source sections the training data never saw.
2. Fail closed. The contamination guard (unit-ID exclusion, duplicate-question detection, verbatim-overlap check, n-gram Jaccard threshold) runs at build time and again on verify. Any violation is exit code 5, not a warning.
3. Same harness, both sides. packsmith eval runs the identical question set, prompts and scoring against the base model and the adapter. packsmith diff --fail-below-delta turns "did it actually learn?" into a CI assertion.

Quickstart

You need stock Crystal (tested on 1.20.0). No models, no API keys — the deterministic path runs anywhere Crystal builds, including CI.

git clone https://github.com/AgentC-Consulting/knowledge-packs
cd knowledge-packs
shards install
crystal build src/packsmith.cr -o bin/packsmith

# build + verify the demo pack (Amber web framework docs)
./bin/packsmith build packs/amber-framework
./bin/packsmith verify packs/amber-framework

# run the eval harness end-to-end against Llamero's deterministic mock bridge
./bin/packsmith eval packs/amber-framework --model test-model --runtime mock \
  --out reports/smoke.eval.json

That last command runs the full harness — structured multiple-choice with typed parsing, bounded retries, invalid-rate accounting — against a mock model that answers deterministic nonsense. It scores ~0. That's the point: the plumbing is provable without any model on disk.

Build a pack from your own docs

./bin/packsmith init my-pack --id you.my-docs --name "My Docs"
# drop curated markdown into my-pack/sources/curated/  (every heading-scoped
# section becomes one provenance-tracked source unit)
./bin/packsmith build my-pack
./bin/packsmith verify my-pack

Generation has two honest modes:

• --mode deterministic (default, zero-cost): template synthesis — source-grounded flashcards, section-recall, cloze questions with sibling-term distractors, exact-command probes. Its quality ceiling is stated plainly: it tests memorization and lookup-like recall only — no conceptual synthesis, no troubleshooting judgment. It exists so the toolchain runs end-to-end with no model and no API key, and it supplies the exact-recall probes.
• --mode structured-model --generator openai:gpt-4.1-mini (or anthropic:…, groq:…, openrouter:…, or mlx:<model-id> for fully local): each source unit is sent to the generator with a JSON schema, and every reply is parsed into typed Crystal classes and validated — wrong enum values, missing fields, or an answer_quote that doesn't actually appear in the source unit all get the record rejected and logged. Train and eval use separate prompts and separate schemas. To be precise about the mechanism: this is schema-prompted generation plus Crystal-side typed validation, not grammar-constrained decoding.

Either way, the generated JSONL is the artifact of record. The recorded recipe (model, seed, temperature, prompt hashes, source hashes, full generation log) is reproducibility evidence, not a promise of byte-identical regeneration against remote models.

Train

./bin/packsmith train packs/amber-framework --base mlx-community/SmolLM-135M-Instruct-4bit

This wraps scripts/train_mlx.sh: a uv venv with mlx-lm and transformers<5 (transformers 5.x breaks current mlx-lm imports), running python -m mlx_lm lora with the measured defaults — 300 iterations, batch 2, 8 layers, lr 1e-4. Those defaults are a measured starting point for SmolLM-135M on one machine, not universal constants. The training config is written next to the adapter, and adapters never bump the pack version — an adapter is a downstream artifact keyed to (pack id + version, base model, training config).

Prove it (or disprove it)

./bin/packsmith eval packs/amber-framework --model BASE --out reports/base.eval.json
./bin/packsmith eval packs/amber-framework --model BASE --adapter artifacts/adapters/… \
  --out reports/adapter.eval.json
./bin/packsmith diff packs/amber-framework/reports/base.eval.json \
  packs/amber-framework/reports/adapter.eval.json \
  --out packs/amber-framework/reports/diff.json --fail-below-delta 0.10

The eval harness scores two probe types for headline numbers:

• Multiple choice — the model must answer as JSON; the reply is parsed Crystal-side into an enum (A|B|C|D). Invalid JSON or an out-of-set answer gets one bounded retry (the retry names the failure and repeats the schema); still-invalid scores wrong and the invalid rate is reported separately, per item.
• Exact recall at temperature 0 — commands, class names, config keys, scored by normalized exact match or an answer-key regex.

Reports carry counts before percentages (74/120, then the ratio), per-item records, and the eval-set hash — diff refuses to compare reports from different eval sets.

Measured results (packsmith-native)

This table is populated only from checked-in reports/*.eval.json + diff.json files produced by the harness above — never typed in by hand. Nothing is published here until a before/after run on record produces it, and if an adapter fails to improve its pack's eval, that failure gets published too. Until then: run the harness yourself — the commands above are the whole procedure.

The demo pack's manifest lists its base models with role target (expected to work) rather than tested (before/after run on record) for exactly this reason.

One measured before/after run IS on record — it predates packsmith (a standalone Python harness following the same split-before-generate and fail-closed rules, but with simpler scoring: first-letter MC parse with no retry, normalized-substring recall — deviations enumerated in live-run/RESULTS.md). It doesn't populate the table above, but the per-item reports, eval sets, training data and write-up are checked in verbatim under live-run/. Its two eval sets measure different things: the memorization set probes trained facts under held-out question templates (a fact-injection probe — deliberately not a source-held-out eval), while the heldout_sources control set draws only from source units that contributed no training example. Signal: recall of trained facts (normalized-substring match at temp 0) went 1/20 → 11/20 while the control set showed no gain (9/50 → 7/50) — fact injection is real, and the flat control is consistent with the adapter learning the trained facts rather than "the framework" (a 50-item synthetic control bounds that claim; it cannot prove it). Questions were synthetic and not hand-reviewed: lab notes, not a headline. Full caveats in live-run/RESULTS.md.

That run's artifacts are also on Hugging Face: the training data and both eval sets as crimson-knight/amber-framework-knowledge-pack, and the exact adapter the numbers were measured on — sha256-matched to RESULTS.md — as crimson-knight/SmolLM-135M-Instruct-4bit-amber-lora, with the before/after per-item reports included verbatim.

The honest fine print

Small models stay small. Published verbatim, because it's true:

This does not turn a small model into a reliable framework expert. It tests whether a local adapter measurably improves source-specific recall and simple usage knowledge under a disclosed eval harness.

Things this toolchain will never claim: general framework mastery, production coding reliability, benchmark superiority, cross-model transfer, or grammar-constrained decoding. And the two hardest failure modes are designed against, not wished away:

• Self-flattering synthetic evals (same generator style for train and eval measures pattern learning, not knowledge): split-before-generate, separate train/eval prompts and schemas, deterministic exact-fact probes, per-item published reports, identical harness both sides, fail-closed guard. Any headline number we ever publish must additionally include 30–50 manually reviewed held-out questions — synthetic-only numbers don't headline.
• Silent training failure (the lr 1e-5 measurement above): before/after eval is the built-in happy path, and diff --fail-below-delta makes it a gate, not a suggestion.

Exit codes

Appendix: the measured MLX recipe

Everything below was measured on one M1 Max (32 GB); treat it as a reference point.

• Setup: uv venv && uv pip install mlx-lm 'transformers<5' — the pin matters: with mlx-lm 0.31.3, transformers 5.x broke the import (4.57.6 works).
• python -m mlx_lm lora --model mlx-community/SmolLM-135M-Instruct-4bit --train --iters 2000 --batch-size 2 --num-layers 8 --learning-rate 1e-4 on the 471-pair Amber dataset → 62.8 s wall, ~33 it/s, 0.42 GB peak. A 300-iteration run takes ~11 s: cheap enough that skipping the before/after eval saves you nothing.
• Default lr 1e-5, same data, same 2000 iterations: val loss 4.94 → 1.96 with recall of trained facts at 2/20 vs the base model's 1/20 (lr 1e-4 reached 11/20). Loss fell; almost nothing went in. This is the whole pitch for making before/after eval the default path. Reports: live-run/reports/.
• GGUF export (scripts/export_gguf.sh): mlx_lm fuse --dequantize (not --de-quantize; with mlx-lm 0.31.3 in our run, fuse --export-gguf produced a truncated file — use the two-step path) → llama.cpp convert_hf_to_gguf.py --outtype f16 (~9 s, valid 258 MB for SmolLM-135M). Inference via llama-simple-chat / llama-completion.
• Bases: SmolLM-135M-Instruct-4bit (cached, exercised), Qwen3-0.6B-4bit (target). In our July 2026 runs (mlx-lm 0.31.3), LoRA adapters on Gemma e-series "elastic" checkpoints trained but showed no visible inference effect; we did not isolate the cause, so we stick to dense bases. A local observation, not an upstream verdict.

Where this is going

Today: build packs from your own docs and prove them locally. The direction we're building toward — stated as ambition, not as something that exists — is a marketplace of installable, signed knowledge packs: buy the "Amber framework" pack or publish your own "internal-API" pack, each shipping its eval set so every claim is checkable on your machine before you trust it. The manifest's signatures[] field and the strict format versioning are already shaped for that future. Image-generation LoRA marketplaces prove people pay for portable model deltas; we have not found a text-model knowledge equivalent with measurement built in. We think measurement is the product.

Crystal + Llamero

The toolchain is written in Crystal on top of Llamero: native MLX sessions and API clients with one structured-output interface, typed schema classes, and a deterministic MockBridge that lets the entire spec suite and CI run green with no model on disk.

See also, from the same team: getting typed, schema-checked JSON out of an LLM in Crystal is its own topic — covered hands-on in crystal-llm-structured-json. And if the Amber framework itself (the demo pack's subject) is new to you: amber-crystal-todo-tutorial is a full walkthrough, with quick copy-paste answers in crystal-amber-recipes.

Repository layout

src/            packsmith CLI + the knowledge-pack format implementation
spec/           pure-logic + MockBridge specs (stock crystal, no models)
schemas/        training_item.v1.json, eval_item.v1.json, pack.manifest.schema.json
scripts/        train_mlx.sh, export_gguf.sh (the measured recipes)
packs/          amber-framework (flagship demo), llamero-docs (CI fixture)

License

MIT — see LICENSE. The demo pack's source units are condensed and rewritten from the Amber Framework documentation, with attribution recorded in the pack manifest. The Amber framework itself is MIT; its docs repository carries no explicit LICENSE file, which is why the pack ships rewritten units rather than verbatim copies.

Built by the team at AgentC Consulting — https://agentc.consulting?ref=kp-toolchain