needle

Crystal bindings for Needle 3, Cactus Compute's on-device model for tool calling, structured extraction and text embedding.

Text goes in, a JSON tool call comes back. A byte-level grammar compiled from your schemas constrains every token, so the call always parses and always matches the schema you declared. An off-topic request returns no calls rather than a guess.

The engine is a sub-1 MB native library loaded with dlopen; the weights are a ~35 MB needle3.cact archive the engine maps at startup. Both are fetched once from the Hugging Face Hub and cached. Nothing is compiled locally, and no inference call ever reaches the network.

Needle 3 only. A Needle 2 archive is rejected by name rather than handed to an engine that cannot run it; rebuild it with needle build against a Needle 3 checkpoint.

Requirements

• Crystal >= 1.21.0
• Linux or macOS. Windows is rejected at compile time: the engine is loaded through dlopen.
• curl and awk for setup.sh, plus unzip or python3 to unpack the wheel.

Install

Add the shard:

dependencies:
  needle:
    github: shpeckman/needle

Then fetch the engine and the base weights for this machine:

./setup.sh install

That prints the library path on stdout and caches both files under ~/.cache/cactus-needle/v3/3.0.1/. Nothing downloads implicitly at runtime: if either file is missing, construction raises and names the command to run.

Quick start

require "needle"

tools = [
  Needle::Tool.new("set_lights", "Turn a room's lights on or off and set brightness.", %({
    "type": "object",
    "properties": {
      "room": {"type": "string", "description": "Room name, e.g. 'living room'."},
      "on": {"type": "boolean", "description": "Whether the lights should be on."},
      "brightness": {"type": "integer", "description": "Brightness from 0 to 100."}
    },
    "required": ["room", "on"],
    "additionalProperties": false
  })),
]

engine   = Needle::Engine.new(tools, system: "You control a smart home.")
response = engine.complete("dim the living room to 30")

if response.refusal?
  puts "no declared tool fits that request"
else
  response.tool_calls.each { |call| puts "#{call.name} #{call.arguments}" }
end

examples/minimal.cr is the above; examples/home_automation.cr drives a full multi-turn loop with a dispatch table, a confidence floor and grounding checks.

Tools

A Tool is a name, a description and a JSON Schema for its arguments. The schema is accepted as a string or as a parsed JSON::Any.

Needle::Tool.new("lock_door", "Lock or unlock a door.", %({
  "type": "object",
  "properties": {
    "door": {"type": "string"},
    "locked": {"type": "boolean"}
  },
  "required": ["door", "locked"],
  "additionalProperties": false
}))

The fourth argument is an optional list of triggers: case-insensitive regexes that route a matching request to this tool and require a call. A triggered call ships even below the engine's confidence floor.

Needle::Tool.new(name, description, schema, [%(\b(lock|unlock)\b.*\bdoor\b)])

Design them the way the model expects: one tool per action, names a user would say, closed sets as enums, bounds on every number, and five tools or fewer per session where you can manage it. Past five, the engine's retrieval head renders only the top five per turn; pass tool_index_path: to persist those embeddings across runs.

Responses

Every turn returns one Needle::Response.

Four predicates cover the branches worth handling:

• refusal? — the engine wanted to call and nothing fits. There is no free-text fallback, so always handle this case.
• held_back? — a call was produced and withheld, because confidence fell under 0.1 or a grounding gate fired. It is in suppressed_calls. Show it and ask, or treat the turn as a refusal.
• done? — the loop is finished; the answer is the tool results you collected.
• ungrounded? / negated? — see below.

ToolCall#arguments is the raw JSON text and parsed_arguments is the parsed object, which raises Needle::Error if the engine ever emits something unparseable. Read only the keys the request evidenced: optional fields with no evidence are omitted, not guessed.

Grounding

response.validation carries the engine's verdict on whether each argument is evidenced in the request. ungrounded lists tool.field paths whose value is not, and negation marks a request whose own clause excludes what was called. ungrounded_for(tool_name) strips the prefix so a dispatch loop can refuse just the flagged calls:

response.tool_calls.each do |call|
  flagged = response.validation.ungrounded_for(call.name)
  next unless flagged.empty?
  execute(call)
end

Confidence

The score is the minimum of a calibration head and the decode probability of the call. Pick a threshold per product: act at or above it, confirm below it. The calibration holds for the base model only, so an engine constructed with weights: reports confidence as nil.

Conversations

Repeated complete calls on one engine continue a single conversation, and later arguments can depend on earlier tool results. Feed each result back as the next complete:

response = engine.complete("dim the living room to 30")
while response.kind.call? && !response.tool_calls.empty?
  results  = response.tool_calls.map { |call| dispatch(call) }
  response = engine.complete(results.size == 1 ? results.first : "[#{results.join(',')}]")
end

reset rewinds the conversation and keeps the tools loaded. To change tools, construct a new engine.

Embeddings

vector = engine.embed("book a table for two at eight")

Returns the sentence embedding as Array(Float32), for local search, matching and routing without a second model.

Fine-tuned weights

needle build produces a .cact archive. Pass it as weights:, or load it into an existing engine:

engine = Needle::Engine.new(tools, weights: "tuned.cact")
engine.load_weights("other.cact")

The archive's format tag is read before anything is loaded, so the engine and the archive can never disagree about generation.

The engine cannot unload weights. Once a tuned archive is bound, constructing or calling a base-model engine raises instead of silently answering with those weights. Construct base-model engines before tuned ones, or run them in separate processes. State is shared per generation across the whole process and guarded by a mutex, so concurrent complete calls serialise rather than corrupt each other.

Resolution order

The engine library is resolved without touching the network:

1. library: passed to Needle::Engine.new
2. NEEDLE3_LIB_PATH
3. ~/.cache/cactus-needle/v3/3.0.1/libneedle.so (.dylib on macOS)

The base weights are resolved from ~/.cache/cactus-needle/v3/3.0.1/needle3.cact. Both raise a Needle::Error naming the fix when they are missing. HF_ENDPOINT redirects every Hub request at a mirror or proxy.

To download explicitly from Crystal:

Needle::Engine.install         # engine library into the cache
Needle::Engine.install_weights # needle3.cact into the cache

CLI

shards build needle produces the needle binary.

download <platform> fetches a prebuilt runner for another device — one of macos-arm64, linux-x86_64, linux-arm64, linux-armv7, linux-riscv64, linux-mipsel, windows-x86_64, windows-arm64, android-arm64, android-armv7, android-riscv64, ios-arm64, ios-sim-arm64, tvos-arm64, watchos-arm64, wasm, wasm-component — and drops needle3.cact beside it.

Training data

generate-data talks to an OpenRouter-compatible chat endpoint. Set OPENROUTER_API_KEY, and OPENROUTER_URL for a different gateway.

needle generate-data --tools schemas.json --num-samples 500 --output data.jsonl
needle generate-data --augment data.jsonl --num-samples 200

Examples are JSONL, one object per line with query, tools and answers; an off-topic example has answers: []. The same pipeline is available as Needle::Generate if you would rather drive it from Crystal.

Fine-tuning

LoRA training and .cact export are JAX-based and live in the reference Python package, so finetune and build locate that stack and forward to it:

pip install "cactus-needle[train]"
needle finetune data.jsonl --epochs 10 --out adapter.safetensors
needle build --lora adapter.safetensors --layers 8 --out tuned.cact

--layers N exports any rung from 2 to 20 of the ladder, so a subnetwork tuned on one product's tools runs on far smaller hardware than the full model needs. --platform <folder> writes a runnable folder with the engine and the archive. --upload pushes the result to $NEEDLE_HF_REPO and needs HF_TOKEN.

Pinning the engine

digests.lock records the SHA-256 of every published engine wheel, one line per platform:

<generation> <engine-version> <platform-tag> <sha256>

It is read at compile time and checked before a wheel is unpacked, so a mismatch raises before anything is written to disk. ./setup.sh install verifies against an existing entry and otherwise records what it saw (trust on first use) unless --frozen is given.

./setup.sh lock      # hash every published wheel, leave existing entries alone
./setup.sh relock    # overwrite entries, reporting anything that changed
./setup.sh verify    # re-download and report drift, changing nothing
./setup.sh weights   # the base archive only
./setup.sh path      # the cache path for this machine
./setup.sh clean     # remove the cached engines and weights

Commit the file. A changed line means upstream replaced a published wheel and deserves review before you accept it.

Environment

Tests

crystal spec

The suite covers path resolution, the lockfile, URL construction, response parsing and the data-generation helpers. It does not touch the network or need an installed engine.

License

MIT. Needle itself is built by Cactus Compute and published under Apache-2.0.