mqhole

mqhole is a small Crystal CLI for moving bytes between two parties that know the same CloudAMQP team API key and wormhole name. It provisions or reuses one shared LavinMQ CloudAMQP subscription, then uses AMQP(S) durable queues as the transport.

This is a transport tool, not a PAKE/encryption implementation like upstream magic-wormhole. The shared name is hashed into a queue name; anyone with the CloudAMQP API key can access the same broker.

Requirements

• Crystal 1.20.2 or newer
• A CloudAMQP team API key or an existing AMQP broker URL

Install dependencies and build:

make

The binary is written to bin/mqhole.

CloudAMQP

Set the API key in the environment:

export CLOUDAMQP_API_KEY=...

The send and receive commands default to scaleway::nl-ams, which CloudAMQP currently reports as supporting shared plans. List regions with shared plans and pass one with --region if you want a different broker location:

bin/mqhole regions --shared-only
bin/mqhole regions scaleway

mqhole creates an instance named after the selected region, for example mqhole-lavinmq-scaleway-nl-ams, and reuses it on later runs.

To use an existing RabbitMQ or LavinMQ broker, set AMQP_URL to its amqp:// or amqps:// connection URL. This bypasses CloudAMQP provisioning and does not require an API key. Keep credentials in the environment rather than command arguments.

Usage

Send stdin:

printf 'hello\n' | bin/mqhole send demo

Receive and echo to stdout:

bin/mqhole receive demo

By default, receive exits after one transfer. Use --listen to keep receiving until you stop it, for example with Ctrl-C:

bin/mqhole receive demo --listen

In listen mode, transfer errors such as a bad decryption passphrase are logged and the failed transfer is dropped so later transfers can still be received.

Send and receive a file:

bin/mqhole send demo --file ./payload.bin

bin/mqhole receive demo --output ./received.bin --no-echo

Send encrypted data:

bin/mqhole send demo --encrypted --file ./payload.bin

The sender prints a generated passphrase as a logfmt line:

at=info event=encryption_passphrase passphrase=...

On an interactive terminal, send updates transfer progress in place on stderr. Use --verbose to write progress as logfmt lines instead. File and --data payloads include a known total size; stdin payloads report bytes and rate without a total:

at=info event=send_progress transfer_id=... bytes=... total=... percent=... rate=... rate_human=".../s"
at=info event=receive_progress transfer_id=... bytes=... total=... percent=... rate=... rate_human=".../s"

Receive progress is reported in verbose mode.

Receive encrypted data:

bin/mqhole receive demo --encrypted --output ./received.bin --no-echo

The receiver prompts for the passphrase without echoing it, then prints *** after the passphrase has been read. If encrypted data is received without --encrypted, or if the passphrase is wrong, mqhole exits with a clear error and leaves the AMQP messages unacknowledged so the receive can be retried.

Run a hook with the temporary payload path:

bin/mqhole receive demo --hook 'sha256sum' --hook-mode file --no-echo

Run a hook with the payload as one process argument:

bin/mqhole receive demo --hook 'printf %s' --hook-mode argument --no-echo

Argument hook mode is intended for text payloads. It rejects payloads containing NUL bytes because those cannot be represented safely as process arguments.

Payloads sent to the same name must be sent sequentially. The current protocol expects each transfer's AMQP messages to stay contiguous in the queue, so concurrent sends to the same name can fail with a transfer id mismatch. Use different names if you need overlapping transfers.

Publishes use a bounded window of 32 outstanding broker confirmations. A send reports success only after every message has been confirmed; persistent messages and acknowledgement after successful output/hook delivery remain in use. Receivers subscribe instead of polling and discard chunk buffers after writing them to the temporary file. Broker prefetch is unlimited because acknowledgements are deferred until the entire transfer is delivered; the local delivery buffer is bounded. Acknowledgement metadata still grows with the number of chunks.

Benchmarks

See the benchmark report for comparisons with tailcat, AMQP, MQTT, and streams on local brokers and free CloudAMQP plans. It includes small and large files, many-file transfers, encryption, memory use, and added network delay, with raw measurements and reproduction instructions.

Development

Run the checks used by CI:

make check

To run the broker integration spec against a disposable local broker:

MQHOLE_TEST_AMQP_URL=amqp://guest:guest@localhost crystal spec

The live smoke test used during development sent and received data through a real CloudAMQP LavinMQ instance in scaleway::nl-ams.