nodedb-lt.cr

A Crystal shard that embeds NodeDB-Lite as an in-process, multi-model database by binding its C FFI (nodedb-lite-ffi, a cdylib). No network hop, no server process — the engine runs inside your Crystal process.

Module name: NodeDB::Lite. require "nodedb-lt" gives you NodeDB::Lite.open(...).

The two shards together give a "dev on lite, prod on origin" pattern for Crystal apps: develop and test against the embedded engine, deploy against a NodeDB Origin, and treat the divergence matrix in this README as the list of places behavior is not identical.

Installation

dependencies:
  nodedb-lt:
    github: emanzx/nodedb-lt.cr
    version: ~> 0.1.0

This shard links against a prebuilt libnodedb_lite_ffi.so. See Building the cdylib and Linking below before running shards install — there is no postinstall build step; the library has to already exist at a path you control.

Building the cdylib

The cdylib is not vendored or built by this shard. Build it once from the nodedb-lite source tree and point the shard at the resulting .so.

cd /home/system/rnd/nodedb-lite-main/nodedb-lite-ffi   # lite main @ ee9ccdd

# The 0.5 nodedb-* crates this worktree depends on are unpublished. The
# worktree's .cargo/config.toml path-patches all 14 of them to a local
# checkout of nodedb-main @ v0.5.0 — that patch has to be in place before
# this build will resolve.

CARGO_TARGET_DIR=/mnt/nodedb-cache/cargo-target-interop \
  cargo build -p nodedb-lite-ffi --release

This produces /mnt/nodedb-cache/cargo-target-interop/release/libnodedb_lite_ffi.so.

The library exports no version symbol, so a build-time hash is the only skew detection available. After building, verify against the recorded hash in docs/lib-build-hash.txt:

sha256sum /mnt/nodedb-cache/cargo-target-interop/release/libnodedb_lite_ffi.so
# compare against docs/lib-build-hash.txt

docs/lib-build-hash.txt also records the exact source provenance (lite main commit + patched nodedb-main version) the recorded hash was built from. If your hash doesn't match, you are linking a different build than this shard was verified against — the contract spec (below) will catch a missing symbol, but it cannot catch a same-symbols, different-behavior rebuild. Re-verify against the design spec's ground-truth section before trusting a mismatched build.

Linking

Source scripts/env.sh before any crystal build / crystal run / crystal spec invocation:

source scripts/env.sh

It sets:

• LIBRARY_PATH / LD_LIBRARY_PATH — pointed at the directory containing libnodedb_lite_ffi.so (defaults to the CARGO_TARGET_DIR used above; override NODEDB_LITE_LIB_DIR if you built elsewhere).
• GLIBC_TUNABLES=glibc.rtld.optional_static_tls=2097152 — the cdylib is large enough that its static thread-local-storage requirements exceed glibc's default static TLS surplus at dlopen time. Without this tunable, loading the library can fail at runtime with a static-TLS allocation error even though the symbols themselves are fine. This is a property of the library's size, not of any one machine.

Two failure modes to know about, both surfaced by the contract spec (spec/contract_spec.cr) before any other spec runs:

• Missing library — the dynamic loader aborts the process at start, before a single spec runs. LD_LIBRARY_PATH isn't pointed at a directory containing libnodedb_lite_ffi.so.
• Stale library — a library that loads but is missing an expected symbol fails lazily, at the first call into that symbol, not at load time. The contract spec dlopens the resolved library directly and asserts all 23 bound symbols resolve — run it first when link errors are confusing; a failure there names exactly which symbol is missing, before any other spec gets a chance to fail on it indirectly.

The contract spec only checks symbol presence, not behavior — it cannot tell a correct build from a same-symbols rebuild with different semantics. That's what the manual sha256sum comparison in Building the cdylib, above, is for; the two checks are complementary, not the same check twice.

Quick start

require "nodedb-lt"

db = NodeDB::Lite.open(":memory:")
db.execute_sql("CREATE COLLECTION notes (id TEXT PRIMARY KEY, body TEXT) WITH (engine='document_strict')")
db.execute_sql("INSERT INTO notes (id, body) VALUES ('n1', 'hello')")
result = db.execute_sql("SELECT body FROM notes WHERE id = 'n1'")
# document_strict SELECT ignores the projection list and returns every
# schema column, so `result.columns` is `["id", "body"]` here, not just
# `["body"]` — index by name rather than assuming position.
result.rows.first[result.columns.index!("body")] # => "hello"
db.close

The open matrix

NodeDB::Lite.open mirrors the FFI's explicit-opt-out design for encryption at rest — there is no implicit "just write plaintext" path for a persistent database:

NodeDB::Lite.open(":memory:")                      # ok — plaintext tempdir, no opt-out needed
NodeDB::Lite.open("./data", passphrase: "secret")  # encrypted at rest
NodeDB::Lite.open("./data", plaintext: true)        # conscious opt-out
NodeDB::Lite.open("./data")                         # raises ArgumentError — in Crystal, before the FFI

A bare persistent path with no passphrase: and no plaintext: true is rejected in Crystal, before it ever reaches the FFI, with a message explaining the choice. This exists because the underlying library's own convention refuses a NULL passphrase against a real path (to stop silent plaintext persistence) — but that refusal comes back as an opaque NULL handle with no cause attached, so the shard front-loads the validation to give you an actionable message instead. Passing both passphrase: and plaintext: true together is also an ArgumentError — pick one.

NodeDB::Lite.open_discarding_corrupt_store!(path, ...) is destructive: if the store at path is corrupt, it is renamed aside and an empty database is returned in its place. The data at path is not recovered by this call. Only reach for it when you have already decided data loss is acceptable.

Blocking semantics

Every FFI call blocks the calling thread — the underlying library runs rt.block_on on whichever thread calls in, for every one of the 23 bound exports. There is no async surface at this layer.

Under Crystal's default single-threaded scheduler, this means every call into NodeDB::Lite freezes every fiber in the process for the duration of that call, not just the calling fiber. A slow query does not just block its own coroutine — it stalls your whole event loop.

Guidance:

• Keep individual queries short.
• Build with -Dpreview_mt to get multi-threaded fiber scheduling, so a blocking call on one thread doesn't freeze fibers scheduled on others.
• For latency-sensitive applications, dispatch database calls to a dedicated worker thread/fiber and communicate results back over a channel, rather than calling in from request-handling fibers directly.

An async wrapper over the blocking FFI is out of scope for v0.1.

:memory: is a plaintext on-disk tempdir

NodeDB::Lite.open(":memory:") does not open a purely in-memory database. It opens a plaintext store in a temporary directory under $TMPDIR. That directory is deleted only on a clean close — a crash, a killed process, or an unclosed handle at exit leaves the data on disk.

Practical consequences:

• Do not put secrets into a :memory: database on the assumption that it disappears when the process dies. It doesn't, unless close actually runs.
• :memory: databases still do real disk I/O; they are not a speed trick distinct from a tempdir-backed persistent database.
• See Explicit close, below — a :memory: database that never gets closed leaks its tempdir the same way a persistent database can lose unflushed writes.

crystal-db driver

Registering require "nodedb-lt" also registers a nodedb-lite: URI scheme with crystal-db:

require "db"
require "nodedb-lt"

DB.open("nodedb-lite://memory") { |db| ... }
DB.open("nodedb-lite://./path/to/data?plaintext=true") { |db| ... }
DB.open("nodedb-lite://./path/to/data?passphrase=secret") { |db| ... }

nodedb-lite://memory maps to :memory:; a host+path maps to that filesystem path; passphrase and plaintext query params map to the matching open keyword arguments.

One shared embedded instance per target, not one per pool connection. The driver keeps a registry keyed by the normalized open target (path + passphrase + plaintext, or the memory sentinel), and every DB::Connection the pool opens against that target is a facade over the same underlying Database. This is mandatory, not an optimization: the underlying library has no file lock, so N independent opens against one directory would race each other's background flush/compaction, and N independent :memory: opens would each get a different database — the classic pool-and-:memory: bug. Pool settings (max_pool_size, etc.) are therefore inert; setting max_pool_size=1 is recommended purely for clarity, since the pool's size does not change how many embedded databases actually exist.

The shared-instance key includes the passphrase/plaintext query params, not just the path. Two DB.open calls against the same persistent path but with different URI params (e.g. one with plaintext=true, one with no params) are treated as different targets and each opens its own, independent embedded handle over the same directory — with no lock to prevent them from racing each other. Always open a given file with identical URI params everywhere in your application.

Typed columns require a document_strict collection — the driver does not special-case this, it's a property of the underlying engine (see the divergence matrix below):

DB.open("nodedb-lite://memory") do |db|
  db.exec("CREATE COLLECTION users (name TEXT, age INT, id TEXT PRIMARY KEY) WITH (engine='document_strict')")
  db.exec("INSERT INTO users (id, name, age) VALUES ($1, $2, $3)", "u1", "alex", 30)
  name = db.query_one("SELECT name FROM users WHERE age = $1", 30, as: String)
end

Query parameters ($1, $2, ...) are inlined client-side as SQL literals, using the same quoting rules as nodedb.cr. This carries the same known limitation: a $n-shaped token inside a string literal in your SQL text is also substituted, since substitution happens on the raw SQL string before it reaches the engine. Keep placeholder syntax out of string literals.

Driver-level failures are re-wrapped where the crystal-db interface demands it; engine failures surface as NodeDB::Lite::OperationError.

Divergence matrix vs nodedb.cr

Same SQL dialect on both shards. Different type fidelity and a handful of SQL-surface gaps, documented here so the parity claim stays honest — "same dialect, documented divergences," never "identical results."

Decode table (JSON value → Crystal type)

Both shards decode a TIMESTAMP column to Time, but by different paths: nodedb.cr parses Time from the wire's text representation by OID; nodedb-lt.cr builds Time from {"micros": n}. Same Crystal type, different source shape — a divergence in mechanism, not in the type the application sees.

Other type-fidelity divergences: a REAL column loses Float32-ness (the JSON boundary only carries Float64); UUID/Decimal values are typed by OID on the wire but arrive as untyped strings here; Bytes values arrive as JSON number arrays here rather than a distinct binary type.

Collection engine and SQL-surface divergences

• Schemaless is the default. A bare CREATE COLLECTION maps to the schemaless document engine. Its SELECT always collapses to exactly two columns — id and a JSON-string document blob holding the row's fields — regardless of the projection list.
• Typed columns require document_strict. CREATE COLLECTION ... (col TYPE, ...) WITH (engine='document_strict') returns real per-field columns from SELECT, but it still ignores the projection list: every schema column comes back, in schema declaration order, id first, regardless of which columns the SELECT actually named. SELECT body FROM notes on a notes (id TEXT PRIMARY KEY, body TEXT) collection returns both id and body — index the row by result.columns.index!("body") rather than assuming the projection list controls column order or count.
• No SQL path round-trips an array. On the schemaless engine, an array literal (e.g. [1.0, 2.0, 3.0]) parses but the field silently comes back null — data loss, not a decode divergence, with no error raised. On a document_strict collection, declaring a FLOAT[] column is rejected outright at DDL time. Vector data has no SQL path in this build; use the vector namespace (db.vector.insert / db.vector.search / db.vector.delete), which reaches the engine through dedicated FFI calls instead of execute_sql.
• kv collections accept SQL writes but not SQL reads. INSERT and UPDATE against a WITH (engine='kv') collection succeed and report rows_affected normally. SELECT against the same collection — including SELECT * — always returns zero columns and zero rows; there is no SQL read path for kv storage, the same situation as document and vector data, which the shard reaches through dedicated FFI calls instead.
• nodedb.cr builder shapes that this engine rejects: the TIME_KEY timeseries column marker, the upcased BITEMPORAL collection flag inside the column list, GRAPH INSERT EDGE / GRAPH TRAVERSE / graph algorithm statements as SQL text (graph operations only reach this engine through the dedicated FFI graph namespace), and SHOW COLLECTIONS. DROP COLLECTION and DESCRIBE are both accepted.

All of the above is pinned by assertions in spec/parity_spec.cr, run against the real embedded engine — if the underlying library's behavior ever drifts from what's documented here, that spec fails loudly rather than the divergence quietly rotting out of date.

Explicit close

Always call db.close when you're done with a database. Crystal does not guarantee that finalize runs at process exit — the shard registers finalize as a backstop, not as the primary teardown path.

Consequences of relying on the finalizer instead of an explicit close:

• A persistent database that never gets an explicit close may lose unflushed writes.
• A :memory: database that never gets an explicit close leaks its backing tempdir (see :memory: semantics, above) — the temporary directory is only removed on a clean close.

db = NodeDB::Lite.open(":memory:")
at_exit { db.close }

close is idempotent — calling it more than once, or on an already-closed database, is safe. Any other call against a closed Database raises NodeDB::Lite::ClosedError.

Not bound in v0.1

• The 6 ndb_array_* exports (ndb_array_create, ndb_array_put_cell, ndb_array_read_coord, ndb_array_slice, ndb_array_delete_cell, ndb_array_gdpr_erase_cell) are out of scope for this release. That array engine speaks zerompk — a hand-rolled MessagePack codec over internal Rust types with no published schema — and binding it correctly means byte-exactly reimplementing that contract in Crystal. This is planned as its own v0.2 effort, built against Rust-generated test vectors rather than reverse-engineered from behavior.

• Sync is fire-and-forget, with no stop and no status surface. db.sync.start(url, jwt) returning successfully means the background retry loop was spawned — it does not mean the origin is reachable. A bad URL still "succeeds" at the call site; connectivity failures retry invisibly with no feedback into Crystal.

  This carries a teardown warning that matters more than the missing status surface: the retry thread this spawns cannot be stopped or joined. Once sync.start has been called, keep that database open for the remainder of the process. Closing the database (or letting it fall through to the finalizer) while the retry thread is still alive races teardown inside the underlying library and can crash the process with an invalid memory access. This is a limitation of the underlying library, not something this shard can guard against — do not close a database that has an active sync running.

License

BSD-2-Clause. See LICENSE.