parquet-cr

A Crystal port of apache/arrow-rs — specifically the parquet crate, the Apache Parquet implementation from apache/arrow-rs.

Upstream is vendored as a git submodule at vendor/arrow-rs, pinned to release 60.0.0 (commit ef1fa157977633f0ba21aa921f9ef3d5c669235b). The upstream parquet/ directory is the source of truth for all ported behavior.

Documentation

• Resuming work — start here in a new session
• Architecture
• Asynchronous storage reads and writes
• Development
• Coding Guidelines
• Testing
• PR Workflow
• Changelog

Supported workflows

The port includes synchronous file/column and Arrow RecordBatch readers and writers, nested data, compression, page indexes, bloom filters, modular encryption, Variant/JSON operations and GeoParquet support. Parity is still in progress: see the current status and documented divergences for scope and limitations.

Runnable examples and benchmark

From a configured repository checkout:

crystal run examples/custom_page_index.cr
crystal run examples/read_with_rowgroup.cr
crystal run examples/object_store.cr
crystal build --release bench/parquet_round_trip.cr -o /private/tmp/parquet-round-trip
/private/tmp/parquet-round-trip --smoke --samples=3

The object-store example uses memory storage and needs no cloud credentials. The benchmark validates values before timing all 46 source cases; its CSV reports first and warm samples. See its guide for dimensions, measurement boundaries and differences from upstream Criterion measurements.

Rewrite Parquet files

crystal build bin/parquet-rewrite.cr -o bin/parquet-rewrite
bin/parquet-rewrite -i input.parquet -o output.parquet --compression zstd
bin/parquet-rewrite --help

The command preserves values and user metadata while changing encoding, compression, row groups, pages, statistics and bloom settings. See command behavior and pinned Rust interoperability checks.

Import CSV or TSV

crystal build bin/parquet-fromcsv.cr -o bin/parquet-fromcsv
bin/parquet-fromcsv -s schema.txt -i input.csv -o output.parquet -h
bin/parquet-fromcsv --help

Provide a Parquet message schema in schema.txt; -h skips the header row. Use -f tsv for tab-separated input. See command behavior and pinned Rust checks.

Upstream README Highlights

The upstream parquet crate provides a Rust implementation of the Apache Parquet columnar file format, including file metadata, encodings, compression, and Arrow interoperability. See the submodule README for the original project description and feature matrix. Upstream tests in parquet/tests and inline #[cfg(test)] modules are treated as the normative behavior for this port.

Installation

Requires Crystal 1.22.0 or newer. The current dependency configuration uses arrow-cr from the sibling path ../arrow-cr; prepare that checkout before installing. See development setup and the dependency requirements.

1. Add the dependency to your shard.yml:

   dependencies:
     parquet-cr:
       github: dsisnero/parquet-cr
   

2. Run shards install with the sibling arrow-cr checkout available.

Usage

require "parquet-cr"

Development

See docs/development.md for checkout, parity tracking, and environment notes.

Contributors

• Dominic Sisneros - creator and maintainer