pack.cr

This Crystal library provides macros that transform simple values to and from byte sequences according to compile-time format strings, based on Perl and Ruby's pack and unpack functions. Packing enforces full type safety and unpacking directly returns extracted values in their specified types.

The library is still under early development.

Usage

Pack.pack

require "pack"

# `Pack.pack` returns a new writable `Bytes`
Pack.pack "csl>", 42_i8, -1000_i16, 1 << 31 # => Bytes[42, 24, 252, 128, 0, 0, 0]

# `Pack.pack_to` writes to an `IO` instance
File.open("my.bin", "rb") do |f|
  version = 1_u8
  total_songs = 5_u8
  first_song = 1_u8
  Pack.pack_to f, "U4CCCCS*",
    {'N', 'E', 'S', 'M'}, 0x1A, version, # 0x1A allowed due to auto-casting
    total_songs, first_song, [0x8000, 0xE000, 0xE003] of UInt16
end

Pack.unpack

require "pack"

# The following:
Pack.unpack buf, "c2S!>a*"

# roughly expands to:
def unpack(buf : Bytes)
  byte_offset = 0

  sz = sizeof(Int8)
  value1 = StaticArray(Int8, 2).new do |i|
    IO::ByteFormat::SystemEndian.decode(Int8, buf[byte_offset + sz * i, sz])
  end
  byte_offset += sz * 2

  sz = sizeof(UInt16)
  value2 = IO::ByteFormat::BigEndian.decode(UInt16, buf[byte_offset, sz])
  byte_offset += sz

  elem_count = buf.size - byte_offset
  value3 = buf[byte_offset, elem_count]
  byte_offset += elem_count

  Tuple.new(value1, value2, value3)
end

# `Pack.unpack` returns a tuple of extracted values
x1, x2, x3 = Pack.unpack Bytes[0x01, 0xC8, 0x03, 0x04], "cCs"
x1 # => 1_i8
x2 # => 200_u8
x3 # => 1027_i16

# No need for further casts
typeof(x1) # => Int8
typeof(x2) # => UInt8
typeof(x3) # => Int16

# Repeat counts and globs become `StaticArray`s and `Array`s
x1, x2 = Pack.unpack Bytes[1, 0, 2, 0, 3, 0], "c2s>*"
x1 # => StaticArray[1_i8, 0_i8]
x2 # => [512_i16, 768_i16]

# Binaries become `Bytes`, UTF-8 values become `Char`s and `String`s
x1, x2, x3 = Pack.unpack Bytes[0x41, 0x42, 0x43, 0x31, 0x32, 0x33, 0x34], "a2U2U*"
x1 # => Bytes[65, 66]
x2 # => "C1"
x3 # => "234"

Current features

• Packing
  • Fixed-size integers (c C s S l L q Q n N v V)
  • Native integers (i I j J)
  • Native size modifiers (_ !)
  • Endianness modifiers (< >)
  • Floating-point values (d f F e E g G)
  • BER-compressed integers (w)
  • Binary strings (a A Z)
  • UTF-8 characters / strings (U U*)
  • Bitstrings and hexstrings (b B h H)
  • Raw pointers and slices (p P)
  • UU-encoded strings (u)
  • Base64-encoded strings (m M)
  • String lengths (/)
  • Offset directives (@ x X)
  • Aligned offsets (x! X!)
  • Repeat counts and globs (*)
• Unpacking
  • Fixed-size integers (c C s S l L q Q n N v V)
  • Native integers (i I j J)
  • Native size modifiers (_ !)
  • Endianness modifiers (< >)
  • Floating-point values (d f F e E g G)
  • BER-compressed integers (w)
  • Binary strings (a A Z)
  • UTF-8 characters / strings (U U*)
  • Bitstrings and hexstrings (b B h H)
  • Raw pointers and slices (p P)
  • UU-encoded strings (u)
  • Base64-encoded strings (m M)
  • String lengths (/)
  • Offset directives (@ x X)
  • Aligned offsets (x! X!)
  • Repeat counts and globs (*)
  • Unpacking directly from readable & rewindable IO?

Non-features (probably)

• Runtime format strings
• Long double (D)
• Signed modifier (!) for n N v V
• Endianness modifiers (< >) for d f F
• Checksums (%)
• Command groups (( ) .)

Installation

1. Add the dependency to your shard.yml:

   dependencies:
     pack.cr:
       github: HertzDevil/pack.cr
   

2. Run shards install

See also

• crystal-lang/crystal#276
• "Crystal equivalent of Ruby’s open(url).read[].unpack?"
• Prior attempt
• perlpacktut
• Ruby's String#unpack
• Ruby's Array#pack

Contributing

1. Fork it (https://github.com/HertzDevil/pack.cr/fork)
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request

Contributors

• Quinton Miller - creator and maintainer