Dual-Write Gap Lab

The row committed. The event never existed.

This interactive Crystal 1.21 lab shows what happens when one business action must write durable state to both a database and an event broker without a shared transaction. It runs one deterministic order stream through four handoffs:

1. database commit, then best-effort broker publish;
2. broker publish, then best-effort database commit;
3. a transactional outbox with an unguarded consumer; and
4. an observable transactional outbox with consumer deduplication.

At the default workload, database-first loses 3,529 committed-order events. Broker-first creates 1,364 phantom events for transactions that rolled back. The unguarded outbox loses neither, but uncertain acknowledgements produce 2,175 duplicate downstream effects. The observable outbox keeps 100% event fidelity, detects relay age in 30 seconds, and correlates each retry and dedupe decision.

Why this lab exists

Ordering two writes does not make them atomic.

• If the database commits and the publish fails, downstream systems never learn about durable business state.
• If the event publishes and the database rolls back, downstream systems act on a state that does not exist.
• If an outbox relay loses an acknowledgement, safe at-least-once delivery can send the same event again.

A transactional outbox stores the business row and event row in one database transaction. A separate relay publishes committed outbox rows. Because relay delivery can repeat, consumers must make the message ID idempotent.

This lab makes the transitions visible through:

• committed orders, broker deliveries, and consumer effects;
• missing, phantom, and duplicate-effect counts;
• event fidelity;
• pending and peak outbox depth;
• oldest outbox age;
• detection and recovery delay; and
• per-order correlated state ledgers.

The deterministic state machine lives in src/model.cr. The HTTP service, JSON serialization, and health client use only Crystal's standard library; there are no runtime shard dependencies.

Run it

Requires Crystal 1.21.

make run

Open http://127.0.0.1:8080.

Choose another bind address or port:

HOST=0.0.0.0 PORT=9000 make run

Print the default model:

make json | jq

With Docker:

docker build -t dual-write-gap-lab .
docker run --rm -p 8080:8080 dual-write-gap-lab

The multi-stage image compiles a static Crystal binary and runs it as a non-root user in a pinned Alpine 3.23.3 image.

Verify it

make check

The gate verifies Crystal formatting, runs deterministic state-machine specs, builds the release binary, validates JavaScript syntax, and exercises the live HTTP routes, untrusted-input clamps, non-finite values, assets, methods, security headers, and environment validation. CI separately builds and health-checks the non-root static-binary container.

API

GET /api/simulate accepts:

Example:

curl 'http://127.0.0.1:8080/api/simulate?relay_pause_seconds=180&consumer_dedupe_minutes=0' | jq

Telemetry recipe

Record every durable transition with the same business key:

order.id
db.transaction.id
db.committed
outbox.id
outbox.created_at
outbox.age_ms
messaging.message.id
messaging.publish.attempt
messaging.acknowledged
consumer.dedupe.hit
consumer.effect.applied
trace.id

With telemetry.sh, these fields let you query directly for committed-but-unpublished orders, published events without a commit, duplicate delivery, duplicate suppression, and the exact relay attempt holding back an outbox row.

Model boundaries

This is an educational deterministic model, not a broker or database benchmark. It isolates the semantics of the transactional outbox pattern:

• the business update and outbox row commit atomically;
• a separate process relays committed rows;
• relay delivery is at least once and can repeat after an uncertain acknowledgement; and
• consumers use the message identity to suppress duplicate effects.

See AWS Prescriptive Guidance on the transactional outbox pattern for the production architecture and its duplicate-message considerations.

License

MIT