tokenizer v1.0.1

Simplified binary stream tokenization for crystal lang

Crystal Lang Tokenizer

Build Status

A tool for buffering and tokenizing streaming inputs.


Consider a binary protocol such as the one used by the Harman BSS DSP.

It uses 0x03 to indicate the end of a message.

require "socket"
require "tokenizer"

# Connect to the device
connection ="", 1023)
connection.tcp_nodelay = true

# Messages terminate with 0x03, so we are looking for this byte
token_buffer =, 0x03))

while !connection.closed?
    raw_data =
    bytes_read =
    break if bytes_read == 0 # Connection was closed

    token_buffer.extract(raw_data[0, bytes_read]).each do |message|
        # Process messages here, messages are of type Bytes

        # If the data was a string, it's simple to convert
        # (assuming we want to ignore the start and stop bytes)
        message =[1, message.size - 2])

        # Do something with the message
        process message

Supported tokenization strategies

  • Message Length - i.e. all messages are 12 bytes in size
  • Delimiter - i.e. all messages end with [0x03, 0x00]
  • Abstract - i.e. message header determines message length

Message Length

Messages are a fixed length, optionally starting with some indicator bytes.

# Message length 4 bytes, including the indicator bytes
buffer =, "GO")

# So a string like "GO12, GO56, G" has 2 complete messages
# "GO12" and "GO56"

messages = buffer.extract("GO12, GO56, G") # => [Bytes, Bytes]
messages = { |bytes| }
messages # => ["GO12", "GO56"]

The example above uses strings however you would typically use this binary data that can't be represented by strings


Messages are variable length, however there is a byte or bytes that represent the end of the message.

# Messages end with \n
buffer ="\n")

# So a string like "Hello.\nHow are you?\nWha" has 2 complete messages
# "Hello.\n" and "How are you?\n"

messages = buffer.extract("Hello.\nHow are you?\nWha")
messages = { |bytes| }
messages # => ["Hello.\n", "How are you?\n"]


Messages are split by some arbitrary logic. i.e.

  • A header specifies the length of a message
  • or a successful CRC check indicates the message end

A callback is used for the application to define when a complete message has been received.

# A message header indicates the length of the message
buffer = do |io|
    bytes = io.peek # for demonstration purposes
    string = io.gets_to_end

    string[0].to_i + 1

# So a string like "7welcome2to5hu" has 2 complete messages
# "7welcome" and "2to"

messages = buffer.extract("7welcome2to5hu")
messages = { |bytes| }
messages # => ["7welcome", "2to"]

  • The block is expected to return the number of bytes in the next message
  • Returning anything <= 0 means the message is not complete
  • You can return the message size even if the message has not completely buffered. (i.e. if the header is completely buffered)
Github statistic:
  • 0
  • 0
  • 2
  • 0
  • 4 months ago


MIT License