flatbuffers

FlatBuffers for Crystal. Schemas are compiled at build time, either from .fbs files or from a Crystal macro DSL, into plain Crystal types that read buffers in place and build them with a single reusable builder.

• Reads are zero-parse: an accessor computes a position and loads a value.
• Every read is bounds-checked. A truncated or corrupt buffer raises; it never reads outside the buffer.
• The generated code and the builder are verified against flatc 25.2.10 in both directions.

Developed and tested on Crystal 1.21.1.

Contents

• Installation
• Quick start
• Defining a schema
• Type mapping
• Generated API
• Object API
• 64-bit offsets and vectors
• Runtime API
• FlexBuffers
• Supported schema features
• Limitations
• Performance notes
• Development

Installation

Add the shard to shard.yml, replacing the placeholder with the repository location:

dependencies:
  flatbuffers:
    github: <owner>/flatbuffers

Then run shards install and require "flatbuffers".

Quick start

monster.fbs, next to the Crystal file that uses it:

namespace Game;

enum Color : byte { Red, Green, Blue }

struct Vec3 {
  x: float;
  y: float;
  z: float;
}

table Weapon {
  name: string (key);
  damage: short;
}

table Shield {
  armor: int;
}

union Equipment { Weapon, Shield }

table Monster {
  name: string (required);
  hp: short = 100;
  level: int = null;
  pos: Vec3;
  color: Color = Blue;
  inventory: [ubyte];
  weapons: [Weapon];
  tags: [string];
  equipped: Equipment;
}

root_type Monster;
file_identifier "MONS";

Building a buffer:

require "flatbuffers"

FlatBuffers.schema "monster.fbs"

builder = FlatBuffers::Builder.new
sword = Game::Weapon.create(builder, name: "Sword", damage: 3)
axe = Game::Weapon.create(builder, name: "Axe", damage: 5)

orc = Game::Monster.create(builder,
  name: "Orc",
  hp: 300,
  level: 7,
  pos: Game::Vec3.new(1, 2, 3),
  color: Game::Color::Red,
  inventory: Bytes[1, 2, 3],
  weapons: Game::Weapon.sorted_vector(builder, [sword, axe]),
  tags: ["fierce", "green"],
  equipped_type: Game::Equipment::Weapon,
  equipped: axe,
)
bytes = Game::Monster.finish(builder, orc)

Reading it back:

monster = Game::Monster.root(bytes)

monster.name                        # => "Orc"
monster.name_bytes                  # => Bytes[79, 114, 99]
monster.hp                          # => 300
monster.level                       # => 7
monster.pos                         # => Game::Vec3(@x=1.0, @y=2.0, @z=3.0)
monster.color                       # => Game::Color::Red
monster.inventory.try &.to_a        # => [1, 2, 3]
monster.weapons.try &.map(&.name)   # => ["Axe", "Sword"]
monster.tags.try &.to_a             # => ["fierce", "green"]
monster.equipped_type               # => Game::Equipment::Weapon

case item = monster.equipped
when Game::Weapon then item.damage  # => 5
when Game::Shield then item.armor
end

weapons = monster.weapons.not_nil!
Game::Weapon.find(weapons, "Sword").try &.damage # => 3
Game::Weapon.find(weapons, "Bow")                # => nil

Fields that were not written return their schema default, or nil where there is none:

bare = FlatBuffers::Builder.new
empty = Game::Monster.root(bare.finish(Game::Monster.create(bare, name: "Bare")))

empty.hp       # => 100
empty.level    # => nil
empty.pos      # => nil
empty.color    # => Game::Color::Blue
empty.weapons  # => nil

Defining a schema

Both front ends feed the same generator and produce the same Crystal API.

From .fbs files

FlatBuffers.schema "monster.fbs"
FlatBuffers.schema "a.fbs", "b.fbs"

• Paths are relative to the file containing the call.
• Every type reachable through include is generated. Put files that share an include in one call so the shared types are generated once.
• Schema errors fail the build with file:line:column: message.

From the macro DSL

FlatBuffers.define do
  namespace Game

  enum_ Color, Int8 do
    Red
    Green
    Blue
  end

  struct_ Vec3 do
    x : Float32
    y : Float32
    z : Float32
  end

  table Weapon do
    field name : String, key: true
    damage : Int16
  end

  table Shield do
    armor : Int32
  end

  union_ Equipment do
    Weapon
    Shield
  end

  table Monster do
    field name : String, required: true
    hp : Int16 = 100
    level : Int32?
    pos : Vec3
    color : Color = Color::Blue
    inventory : Array(UInt8)
    weapons : Array(Weapon)
    tags : Array(String)
    equipped : Equipment
  end

  root Monster
  identifier "MONS"
end

This declares the same schema as the .fbs file above.

Fields:

• name : Type declares a field; name : Type = value gives it a default.
• name : Type? makes a scalar or enum optional. Structs, tables, strings and vectors are always nilable and must not be written with ?.
• Attributes need the field prefix: field name : String, required: true, field hp : Int16 = 100, id: 2.
• Block-level attributes follow the name: struct_ Vec3, force_align: 16 do, enum_ Flags, UInt8, bit_flags: true do.
• Defaults accept numbers, booleans, an enum member (Color::Blue), a quoted string of combined bit flags ("Fly Swim"), Float64::NAN and Float64::INFINITY.
• A string field takes a string default, motto : String = "grr". A vector field takes the empty default, loot : Array(Int32) = [] of Int32.
• 64-bit fields use the attributes of the same name: field blob : Array(UInt8), vector64: true, field title : String, offset64: true.

Errors are reported against the Crystal source line of the offending statement.

FlatBuffers.define(emit: "schema/monster.fbs") do ... end also writes the equivalent .fbs file, relative to the calling file, for use from other languages. The file is rewritten only when its content changes.

Type mapping

Names:

• Namespaces and type names are camel-cased: my_game.Sample becomes MyGame::Sample.
• Field names are underscored: maxHp becomes max_hp.
• Enum members are camel-cased: NONE becomes None, BIT_ONE becomes BitOne.
• A field whose name is a Crystal keyword or core method gets a trailing underscore: class becomes class_.
• A union member given as a qualified type is named after the full path: Game.Common.Vec3 becomes GameCommonVec3.
• A declared field always keeps its name. A generated accessor (_bytes, _type, _nested_root, _flexbuffer_root) whose name is already taken by a declared field gets a trailing underscore: with fields name and name_bytes, the zero-copy reader of name is name_bytes_. The same name is used for the matching create argument.

Generated API

Tables

For table Monster:

For a union or vector of unions with a string member, <field>_bytes returns the same members with Bytes in place of String, viewing the buffer without copying.

Reader return types:

create arguments are named and all optional, except required fields:

Fields marked offset64 or vector64 take an Offset64 in place of an Offset; see 64-bit offsets and vectors.

Vector elements may be values (scalars, enums, structs), Strings, or Offsets. A struct or string used as a union member is written first with builder.struct or builder.string.

Every child object must be created before the create call of the table that refers to it. create handles strings and vectors passed as Crystal values itself.

For the root type with a file_identifier:

For a table with a key field:

A string key field is implicitly required.

Structs

A struct is a Crystal value type with one getter per field and a positional constructor in declaration order:

point = Game::Vec3.new(1, 2, 3)
point = Game::Vec3.new(x: 1, y: 2, z: 3)
point.x # => 1.0

A vector of structs is copied as one block when the struct has no padding and no force_align; Vec3.wire_dense? tells whether that holds.

The same type is used for reading and for building. Vec3.wire_size and Vec3.wire_align give its size and alignment in the buffer.

Enums and unions

An enum becomes a Crystal enum with the declared underlying type. A value unknown to the schema is preserved, not rejected.

A union generates an enum of the same name whose first member is None. The reader returns the member matching the selector, or nil for None and for selectors unknown to the schema. Union members may be tables, structs or strings.

Services

rpc_service generates a module holding a METHODS constant:

Game::Storage::METHODS[:store][:request]   # => Game::Monster
Game::Storage::METHODS[:store][:streaming] # => "none"

Each entry has name, request, response and streaming. No transport or stubs are generated.

Object API

Every table also generates a mutable class named after it with a T suffix. unpack copies a table out of a buffer and pack writes the object back:

object = Game::Monster.root(bytes).unpack # => Game::MonsterT
object.hp = 50
object.weapons.try &.<<(Game::WeaponT.new(name: "Bow", damage: 2))
object.equipped_type = Game::Equipment::Shield
object.equipped = Game::ShieldT.new(armor: 4)

builder = FlatBuffers::Builder.new
bytes = Game::Monster.finish(builder, object.pack(builder))

fresh = Game::MonsterT.new(name: "Imp", hp: 10)

• The constructor takes named arguments. Every one is optional except required fields; the others start at their schema default or nil.
• Each field is a property.

pack writes the selector stored in <field>_type; keep it in step with the value when changing a union. A vector of unions cannot be packed while it holds a nil element.

A schema that declares both Monster and MonsterT is rejected.

64-bit offsets and vectors

offset64 on a string or vector field stores a 64-bit offset to it. vector64 on a vector field implies offset64 and also widens the vector's length to 64 bits; such a field reads as FlatBuffers::Vector64(T). Vectors with either attribute may only hold scalars or structs.

table Archive {
  title: string (offset64);
  blob: [ubyte] (vector64);
  count: int;
}

The format requires all 64-bit data to be written before anything else. When the table is the first object in the buffer, create takes plain values:

archive = Game::Archive.create(builder, title: "far", blob: bytes, count: 1)

Otherwise write the 64-bit data first and pass the Offset64s on:

blob = builder.vector64(bytes, length64: true)
title = builder.string64("far")
child = Game::Weapon.create(builder, name: "Sword")
archive = Game::Archive.create(builder, title: title, blob: blob, count: 1)

• vector64(items, length64: true) matches a vector64 field; vector64(items) matches an offset64 vector.
• Writing 64-bit data after any other object raises FlatBuffers::Error.
• The whole buffer is still limited to 2 GiB; see Limitations.

Runtime API

FlatBuffers::Builder

The slice returned by finish and to_slice points into the builder's buffer and is valid until the next reset or write. Copy it with dup to keep it.

start_table, slot, end_table, prep, place, push, push_offset, push_offset64, ref and ref64 are the low-level interface used by generated code.

FlatBuffers::Vector(T), FlatBuffers::Vector64(T) and FlatBuffers::UnionVector(E, T)

All include Indexable(T), so size, [], each, map, to_a and the rest of that interface are available.

Vector#bytes and Vector64#bytes return the raw element bytes as a read-only slice without copying. For a [ubyte] field this is the payload itself.

Other

Reading outside the buffer raises IndexError.

FlexBuffers

FlatBuffers::Flex decodes schema-less FlexBuffers in place. Nothing is parsed up front; every accessor is bounds-checked.

root = FlatBuffers::Flex::Reference.root(bytes)
root["name"].as_s            # => "orc"
root["hp"].as_i              # => 300
root["list"][1].as_s         # => "two"
root["xyz"].as_vector.map(&.as_f) # => [1.0, 2.0, 3.0]
root["missing"]?             # => nil
root.as_map.each { |key, value| puts key }

A [ubyte] table field marked flexbuffer gains <field>_flexbuffer_root, which returns the Reference or nil when the field is absent.

• The number accessors convert between integers, floats and bools, and read null as zero. Any other type raises FlatBuffers::Error, as does asking a value for a vector, map or string it is not.
• as_vector covers untyped, typed and fixed-size vectors, and the values of a map.
• Flex::Map is Enumerable({String, Reference}) and offers size, keys, values, [key] (raises KeyError) and [key]?. Lookup is a binary search that compares the key bytes in place. A key containing a zero byte is never found.

Supported schema features

• Namespaces, includes, user attribute declarations.
• Enums, including bit_flags.
• Unions of tables, structs and strings, with aliases; vectors of unions.
• String defaults (= "text") and empty vector defaults (= []).
• offset64 and vector64.
• Structs with nested structs, enums, fixed-size arrays and force_align.
• Tables with scalar defaults, optional scalars, explicit ids, and vectors of scalars, enums, structs, strings and tables.
• required, deprecated, key, shared, nested_flatbuffer, flexbuffer, original_order, force_align.
• root_type, file_identifier, file_extension, size-prefixed buffers.
• rpc_service, as metadata only.

The attributes hash, streaming, idempotent, private, csharp_partial, and the cpp_* and native_* families are accepted and have no effect on the generated Crystal code.

Limitations

• No verifier. Reads are bounds-checked, but a buffer is not validated up front.
• No in-place mutation; change a buffer through the object API.
• A buffer cannot exceed 2 GiB, the size limit of a Crystal Bytes. offset64 and vector64 are supported for compatibility with the wire format, not for larger buffers.
• 64-bit strings and vectors must be created before any other object in the buffer.
• The only vector default is [].
• FlexBuffers can be decoded but not built.
• The object API cannot pack a vector of unions holding a nil element.
• Little-endian targets only, as Crystal itself assumes.

Performance notes

• Reuse one builder with reset. A new builder allocates and regrows its buffer for every message.
• Read strings through _bytes on hot paths. The String readers allocate on every call.
• Pass arrays or slices of numbers, bools, enums or padding-free structs to vector fields; they are copied as one block.
• find on a string key compares bytes in place and allocates nothing.
• Vector validates its extent once, so iterating scalars and structs does no per-element bounds check.
• Tables written with the same fields set normally share one vtable, so many small tables of one kind cost little more than their own bytes.
• sorted_vector allocates a temporary copy to sort. Write the tables in key order and use builder.vector when the order is already known.
• The object API copies everything it touches. Use it to edit a buffer, not to read one.

Indicative figures from one run of make bench on a single pinned core, with Crystal 1.21.1. The workload is one table holding 1,024 small tables, 1,024 keyed tables, 1,024 strings and several scalar vectors, 105,736 bytes as a FlatBuffer and 139,398 bytes as JSON.

make bench reproduces these on your machine. make profile and make perf break the cost down per scenario.

Development

make bench runs every group at 8 and at 1,024 elements, except flexbuffers, which runs once on a fixed document. It prints the time per call, its spread, the bytes allocated per call and, for whole-vector groups, the time per element. Before timing anything it checks that the schema file and the macro DSL build byte-identical buffers.

The lookup rows search for each key once per call, in a fixed shuffled order, so the branch predictor cannot settle on one path; read the per-element column as the time per find.

make bench GROUPS="lookup object" runs only the groups whose title contains one of the words.

make profile and make perf run one scenario at a time through bench/profile.cr: encode, decode_bytes, decode_strings, vectors, lookup, object, flex and archive. A round of flex or archive is 64 operations, so that every scenario does a comparable amount of work.

• make profile needs valgrind. It prints the hottest functions of each scenario, then a table of instruction totals and instructions per round with the cost of the setup scenario subtracted. The callgrind.out.<scenario> files are kept for callgrind_annotate or KCachegrind.
• make perf needs Linux perf and permission to use it (kernel.perf_event_paranoid). It prints perf stat counters (cycles, instructions, branches and their misses, cache references and misses) and the hottest symbols of each scenario. The perf.data.<scenario> files are kept, recorded with frame-pointer call graphs, for perf report. On a hybrid CPU the counters are restricted to the performance cores, which is where the run is pinned.
• Both accept SCENARIOS="encode lookup" to choose scenarios. ROUNDS sets the rounds per scenario under callgrind (default 300) and PERFROUNDS under perf (default 20000).

Layout:

The compiler caches the generator binary and does not track .ecr templates. After editing a template, run crystal clear_cache or touch src/flatbuffers/compiler/generator.cr; otherwise the old output is reused.

License

MIT