Karma

Karma is a small TCP service for high-throughput, day-bucketed limit usage.

Use it when an application needs fresh usage totals for many subjects on every request and an analytical database is too heavy for that hot path. Karma keeps usage counters in memory, persists accepted writes through snapshots and a write-ahead log (WAL), and speaks newline-delimited JSON over TCP.

Russian version: README.ru.md.

What It Is For

Typical flow:

application loads accounts, users, or workspaces
  -> application asks Karma for current limit usage by subject id
  -> response returns fresh pre-aggregated usage totals

Karma is a focused read model for limit usage, not a general-purpose time-series database.

It supports:

• unsigned 64-bit usage counters grouped by limit name, subject id, and UTC day bucket;
• single and batch reads/writes;
• idempotent writes for at-least-once producers;
• large rebuilds through streaming ingest;
• snapshots, WAL replay, and restore verification;
• asynchronous master-to-slave replication by snapshot bootstrap and WAL polling;
• health, statistics, and Prometheus-style metrics.

It does not provide automatic leader election, quorum writes, multi-master replication, arbitrary time-series tags, or ad-hoc analytical queries.

Quick Start

Requirements:

• Crystal 1.17.1
• Shards

Build and run:

shards build --release
bin/karma \
  --bind=127.0.0.1 \
  --port=8080 \
  --directory=.karma-data \
  --restore=true \
  --wal=true

Write and read usage for a limit:

printf '{"v":2,"op":"counter.increment","series":"api_requests","key":42,"value":1}\n' \
  | nc 127.0.0.1 8080

printf '{"v":2,"op":"counter.sum","series":"api_requests","key":42}\n' \
  | nc 127.0.0.1 8080

Response envelope:

{"protocol_version":2,"success":true,"response":1,"error_code":null}

With Docker:

docker build -t karma:local .
docker run --rm \
  -p 8080:8080 \
  -v karma-data:/data \
  karma:local \
  --bind=0.0.0.0 \
  --port=8080 \
  --directory=/data \
  --restore=true \
  --wal=true \
  --wal-fsync=true

For production, use a persistent volume, WAL enabled, --wal-fsync=true, health checks, metrics scraping, and regular snapshot.create_all or SIGUSR1 snapshots.

Data Model

Read commands never create missing series. A missing series returns not_found; a missing key inside an existing series returns zero or an empty result.

Protocol

Karma 1.0 accepts only protocol v2 requests:

• one request is one JSON object followed by \n;
• one response is one JSON object followed by \r\n;
• every request must include "v": 2 and an op field.

Example:

{"v":2,"op":"counter.increment","series":"api_requests","key":42,"bucket":20260505,"value":1}

Error response:

{
  "protocol_version": 2,
  "success": false,
  "response": "Field tree or series is required",
  "error_code": "validation_error"
}

Stable error codes:

If --auth-token is set, every client request must include token. If --read-auth-token is set, that token can run read-only commands. Tokens are never written to the WAL.

Common Operations

Limit Usage

{"v":2,"op":"tree.create","series":"api_requests"}
{"v":2,"op":"counter.increment","series":"api_requests","key":42,"value":1}
{"v":2,"op":"counter.increment","series":"api_requests","key":42,"bucket":20260505,"value":10}
{"v":2,"op":"counter.decrement","series":"api_requests","key":42,"bucket":20260505,"value":1}
{"v":2,"op":"counter.sum","series":"api_requests","key":42}
{"v":2,"op":"counter.sum","series":"api_requests","key":42,"range":{"from":20260501,"to":20260505}}
{"v":2,"op":"counter.series","series":"api_requests","key":42,"range":{"from":20260501,"to":20260505}}

Batch Reads and Writes

{"v":2,"op":"counter.batch_sum","series":"api_requests","keys":[41,42,43]}
{"v":2,"op":"counter.batch_sum","series":"api_requests","keys":[41,42,43],"range":{"from":20260501,"to":20260505}}
{"v":2,"op":"counter.multi_sum","items":[{"series":"api_requests","key":101},{"series":"emails_sent","key":101}]}
{"v":2,"op":"series.batch_add","series":"api_requests","items":[[42,20260505,10],[43,20260505,3]]}
{"v":2,"op":"series.batch_set","series":"api_requests","items":[[42,20260505,10],[43,20260505,0]]}

series.batch_set writes exact bucket values. A zero value deletes that bucket. Large requests must fit --max-request-bytes.

Series Inspection and Maintenance

{"v":2,"op":"tree.list"}
{"v":2,"op":"tree.info","series":"api_requests"}
{"v":2,"op":"tree.keys","series":"api_requests","limit":1000,"cursor":0}
{"v":2,"op":"tree.summary","series":"api_requests","range":{"from":20260501,"to":20260505}}
{"v":2,"op":"tree.top","series":"api_requests","limit":100}
{"v":2,"op":"series.delete_before","series":"api_requests","before":20260401}
{"v":2,"op":"series.compact","series":"api_requests"}
{"v":2,"op":"system.compact"}

Deletes and Resets

{"v":2,"op":"counter.reset","series":"api_requests","key":42}
{"v":2,"op":"counter.batch_reset","series":"api_requests","keys":[41,42,43]}
{"v":2,"op":"tree.reset","series":"api_requests"}
{"v":2,"op":"counter.delete_range","series":"api_requests","key":42,"range":{"from":20260501,"to":20260505}}
{"v":2,"op":"counter.batch_delete_range","series":"api_requests","keys":[41,42,43],"range":{"from":20260501,"to":20260505}}
{"v":2,"op":"tree.delete_range","series":"api_requests","range":{"from":20260501,"to":20260505}}

Idempotent Writes

Mutating commands can include idempotency_key. Karma stores the first successful response for that key. Repeating the same payload returns the saved response with "idempotent": true; reusing the key with a different payload returns idempotency_conflict.

Example:

{"v":2,"op":"counter.increment","series":"api_requests","key":42,"bucket":20260505,"value":1,"idempotency_key":"usage-event-123"}

Eligible commands:

• counter.increment, counter.decrement;
• series.batch_add, series.batch_set;
• counter.reset, counter.batch_reset;
• counter.delete_range, counter.batch_delete_range;
• tree.reset, tree.delete_range.

Idempotency records are persisted through WAL and snapshots. Retention is controlled by --idempotency-max-records, --idempotency-max-age-seconds, and:

{"v":2,"op":"idempotency.prune","before":"2026-05-29T00:00:00Z","limit":10000}

Streaming Ingest

Streaming ingest is for rebuilds, backfills, and large imports. Supported modes:

Example:

{"v":2,"op":"ingest.begin","stream_id":"import-20260505","mode":"add","granularity":"day"}
{"v":2,"op":"ingest.chunk","stream_id":"import-20260505","series":"api_requests","chunk_seq":1,"items":[[42,20260505,10]]}
{"v":2,"op":"ingest.commit","stream_id":"import-20260505"}

Abort:

{"v":2,"op":"ingest.abort","stream_id":"import-20260505"}

Duplicate chunks are skipped. Out-of-order chunks are rejected before they are applied. A stream is bound to the series used by its first chunk. Committed streams are remembered durably, so a repeated replace_series commit cannot replace the series again after restart, snapshot restore, or replication bootstrap.

Persistence and Recovery

Karma persists data through:

• snapshots: MessagePack .tree files, one per series;
• WAL: newline-delimited JSON entries in karma.wal.

The active WAL rotates at 64 MiB by default. Rotated files are named karma.wal.<first_lsn>.segment and have optional *.segment.idx sidecar indexes that map LSN to byte offset for fast replication catch-up.

Snapshot commands:

{"v":2,"op":"snapshot.create","series":"api_requests"}
{"v":2,"op":"snapshot.create_all"}
{"v":2,"op":"snapshot.list"}
{"v":2,"op":"snapshot.info"}
{"v":2,"op":"snapshot.verify"}

Fetch or load snapshots:

{"v":2,"op":"snapshot.load","file":"1777925811_api_requests.tree"}
{"v":2,"op":"snapshot.fetch","file":"1777925811_api_requests.tree"}
{"v":2,"op":"snapshot.fetch_chunk","file":"1777925811_api_requests.tree","offset":0,"limit":262144}

New WAL entries use a v2 LSN envelope:

{"v":2,"lsn":1,"entry":{"v":2,"op":"counter.increment","series":"api_requests","key":42,"bucket":20260505,"value":1}}

Startup with --restore=true loads the latest snapshot per series and replays WAL entries after the snapshot LSN. snapshot.create_all writes atomic snapshots, fsyncs them, truncates WAL after successful snapshotting, and prunes old snapshots according to --dump-retention-per-tree.

Recovery markers for external pipelines:

{"v":2,"op":"recovery.checkpoint","source":"usage-export","offset":"export-2026-05-05","event_id":"batch-42"}
{"v":2,"op":"recovery.status"}
{"v":2,"op":"recovery.status","source":"usage-export"}
{"v":2,"op":"reconciliation.report","checked_points":1000,"mismatch_count":2,"absolute_drift":15,"max_abs_delta":10}

Replication

Karma supports asynchronous master-to-slave replication. A slave can bootstrap from master snapshots and then poll WAL entries.

Start a slave:

bin/karma \
  --role=slave \
  --port=8081 \
  --directory=/var/lib/karma-slave \
  --restore=true \
  --replication-source-host=127.0.0.1 \
  --replication-source-port=8080 \
  --replication-token=read-secret

Useful requests:

{"v":2,"op":"replication.status"}
{"v":2,"op":"replication.entries","after_lsn":120,"limit":1000}

Operational boundaries:

• replication is asynchronous;
• slave nodes reject direct mutating client commands;
• failover is manual;
• stop the old master before promoting a slave;
• rebuild remaining slaves from the promoted master.

Detailed runbook: docs/replication-operations-runbook.md.

Configuration

Command-line options override environment variables. Boolean values are true/false. Timeout values are seconds unless the option name ends with -ms.

Health and Metrics

{"v":2,"op":"system.ping"}
{"v":2,"op":"system.health"}
{"v":2,"op":"system.stats"}
{"v":2,"op":"system.metrics"}

Metrics include uptime, role, memory use, series/key counts, WAL size and LSN, command counts and latency, batch counters, ingest counters, idempotency counters, recovery/reconciliation counters, and replication lag/polling status.

Watch these in production:

• karma_replication_lag_entries
• karma_replication_poll_errors_total
• karma_replication_last_poll_success_unix
• karma_errors_total
• karma_query_timeouts_total

Clients

Crystal client:

dependencies:
  karma_client:
    path: clients/crystal

KarmaClient.with_client do |karma|
  karma.record_usage("api_requests", subject_id: 42, amount: 1, day: Time.utc)
  karma.usage("api_requests", subject_id: 42)
end

See clients/crystal.

Ruby/Rails client:

gem "karma_client", path: "clients/ruby"

The client uses the v2 TCP JSON protocol, maps stable Karma error codes to Ruby exceptions, supports explicit connect/read/write timeouts, and includes a small connection pool for Puma/Sidekiq workloads.

Minimal Ruby request:

require "json"
require "socket"

socket = TCPSocket.new("127.0.0.1", 8080)
socket.write({v: 2, op: "counter.sum", series: "api_requests", key: 42}.to_json + "\n")
puts socket.gets
socket.close

Performance Checks

Local results depend on CPU, disk, filesystem, runtime, network, and workload mix. Treat these as local regression checks, not universal promises.

Last recorded local results: 2026-06-06.

Additional local checks from the same run:

• idempotent counter.increment, WAL off, prebuilt JSON: about 506,918 ops/sec without idempotency_key, about 205,914 ops/sec with unique keys;
• 100,000 keys with 7 buckets each: counter.batch_sum read about 1,505,471 keys/sec;
• 50,000 keys with 356 buckets each: counter.batch_sum read about 1,946,673 keys/sec;
• replication load test ended with zero lag and matching master/slave totals;
• a 1,000,000-entry segmented WAL read a cold page from an indexed segment in 83.23 ms versus 253.36 ms without the sidecar index.

Reproduce the main checks:

crystal build --release scripts/load_test.cr -o bin/karma_load_test
bin/karma_load_test

crystal build --release scripts/tcp_load_test.cr -o bin/karma_tcp_load_test
bin/karma_tcp_load_test --clients=4 --wal=true --wal-fsync=false

More scripts are in scripts.

Signals

• SIGINT: stop accepting new TCP clients, snapshot all series, truncate WAL after successful snapshots, and exit with status 0.
• SIGUSR1: snapshot all series, truncate WAL after successful snapshots, and keep running.

Development

Developer documentation in Russian: docs/development.ru.md.

crystal spec
crystal spec spec/bucketed_counter
shards build --release

The Karma::BucketedCounter data structure lives in src/bucketed_counter.cr and src/bucketed_counter.

License

MIT