crystal-csv-gz-test

Crystal port of https://github.com/mostlygeek/go-csv-gz-test

This is a micro-benchmark project to see how fast Crystal can filter S3 inventory .csv.gz files

Other implementations:

• mostlygeek's Go implementation
• mythmon's Rust implementation
• peterbe's Python implementation + blog

Usage

# get some working data, downloads 1GB from S3 into testdata/ subdirectory
> ./download.sh

# Processing using a one file at a time
> crystal run ./filter.cr

My results (on my late 2016 13" MBP)

Total: 31521045, Matched: 710093, Ratio: 2.25%
Time: 00:28:44.910567000