treesitter

Crystal bindings to the Tree-sitter C library: grammars loaded from shared libraries at run time, incremental parsing from a string or a chunked reader, syntax nodes, and queries with the common predicates.

Requirements

• Crystal 1.21 or newer
• libtree-sitter 0.25 or newer, found through pkg-config (Fedora: tree-sitter-devel; or build it from the repository with make && sudo make install)
• Grammars as shared libraries, for example the ones nvim-treesitter installs, or built with tree-sitter build

Installation

dependencies:
  treesitter:
    github: shpeckman/treesitter

require "treesitter"

Usage

json   = TreeSitter::Language.load("/usr/lib64/libtree-sitter-json.so", "json")
parser = TreeSitter::Parser.new(json)
text   = %({"a": [1, 2]})
tree   = parser.parse(text)

root = tree.root
root.type                                  # => "document"
pair = root.named_child(0).not_nil!.named_child(0).not_nil!
pair.field("key").not_nil!.range           # => 1...4
pair.field("value").not_nil!.start_point   # => TreeSitter::Point.new(0, 6)

Languages

Language.load(name) finds and loads a grammar from Language.directories. Lookup checks each directory in order for libtree-sitter-<name>.so and <name>.so; Language.find(name) returns the matching path or nil. The default directories are ~/.local/lib, /usr/local/lib, /usr/local/lib64, /usr/lib, and /usr/lib64. Replace, append to, or prepend to the array to configure lookup:

TreeSitter::Language.directories.unshift("/opt/tree-sitter")
json = TreeSitter::Language.load("json")

Language.load(name) raises TreeSitter::Error when no library is found. Language.load(path, name, symbol = "tree_sitter_<name>") opens an explicit grammar library and calls its entry function; dashes in the name become underscores in the default symbol. Both forms raise TreeSitter::Error when the library or symbol is missing, or when the grammar's ABI is outside Language::MIN_ABI..Language::MAX_ABI (13 to 15). A grammar linked into the program can be wrapped directly: Language.new(tree_sitter_json(), "json"). Loaded libraries stay open for the life of the process.

Parsing

A chunk must stay unchanged until the next call; the parser keeps the last one alive. Raises Error when the parse does not finish.

To reparse after a change, describe it with Tree#edit and pass the old tree:

tree = parser.parse("[1, 2]")
tree.edit(TreeSitter::Edit.new(start_byte: 5, old_end_byte: 5, new_end_byte: 8,
  start_point: TreeSitter::Point.new(0, 5), old_end_point: TreeSitter::Point.new(0, 5),
  new_end_point: TreeSitter::Point.new(0, 8)))
tree = parser.parse("[1, 2, 3]", tree)

new_tree.changed_regions(old_tree) lists the Regions (bytes and points) whose syntax differs between the edited old tree and the new one; text edited inside an unchanged structure is not among them.

parser.included_regions = regions limits the following parses to those regions of the text, in order and not overlapping, for a language embedded in another one (a <script> in HTML, a fenced block in Markdown); an empty array parses the whole text again. Node#region gives a node's region.

parser.included_regions = [script.region]
inner = parser.parse(text)

Offsets are bytes into UTF-8 text. A Point is a zero-based row and a byte column. Tree#copy makes an independent copy, cheap to make, for use on another fiber.

Nodes

Node is a value tied to its Tree, and it keeps the tree alive. Every navigation method returns nil where there is no node.

Queries

query = TreeSitter::Query.new(json, <<-SCM)
  (pair key: (string) @key)
  ((string (string_content) @keyword) (#any-of? @keyword "if" "else"))
  SCM

query.each_capture(root, text) do |capture|
  capture.name             # "key"
  capture.node.range       # bytes
  capture.pattern          # index of the pattern that matched
end

query.each_match(root, text, 0...40) do |match|
  match["key"]             # first node captured as key
  match.nodes("key")       # every one, for quantified captures
end

each_capture yields captures in text order, which is what highlighting needs. each_match yields whole matches. Both take an optional byte range. The text for predicates comes from a TreeSitter::Source (text(from, to)); passing a String wraps it in a StringSource. Text held elsewhere, such as a line buffer, includes Source so a query never builds the whole document.

A query the grammar rejects raises QueryError, which carries its kind (Syntax, NodeType, Field, Capture, Structure, Language) and byte offset.

Predicates

Each predicate has a not- form (#not-eq?, …). A capture missing from the match passes. Directives (#set!, #make-range!), unknown predicates and a #match? whose expression does not compile are skipped, so the match is reported as if they were absent.

query.properties(pattern) holds the #set! directives of a pattern as key → value ("" when a key is set without one): (#set! injection.language "lua") gives {"injection.language" => "lua"}. Read it with match.pattern or capture.pattern.

Language detection

require "treesitter/detect" (not loaded by require "treesitter") adds TreeSitter::Detector. It guesses the language of text that has no file name, such as a pasted snippet. It works in two steps:

1. An rlcd model, trained with a proper scoring rule, ranks the languages with calibrated probabilities. It hashes words, symbol runs and character 2–5-grams of the text.
2. The checked most probable languages are parsed with their grammars (Language.find). Each one's probability is multiplied by exp(-penalty * error_ratio), where error_ratio is the share of the text that falls in ERROR nodes, with one byte counted per missing node. The checked candidates then share out their original total again. A language without a loadable grammar is not reweighted.

Labels are grammar names, so the detected name goes straight to Language.load.

require "treesitter/detect"

detector = TreeSitter::Detector.train({
  "json"   => json_snippets,
  "python" => python_snippets,
  "ruby"   => ruby_snippets,
}, epochs: 30)
detector.save("languages.rlcd", precision: RLCD::Precision::I8)

detector  = TreeSitter::Detector.load("languages.rlcd")
detection = detector.detect("def greet(name):\n    print(name)\n")
detection.language                 # => "python", or nil below the threshold
detection.probability              # => 0.97
detection.candidates               # => [Candidate("python", 0.97, 0.0), Candidate("ruby", 0.02, 0.4), ...]
detection.language.try { |name| TreeSitter::Language.load(name) }

Detector.train(samples, dim: 4096, seed: 42, hidden: 0, epochs: 60, lr: 0.01, validation: 0.15, rng:) takes grammar name → snippets, for at least two languages. It trains a linear head (or a hidden layer) with the log score. A temperature fitted on the validation share of each language's snippets calibrates the result. save writes JSON for a .json path and the compact rlcd binary otherwise, with precision F64, F32 or I8. Detector.load detects the format and raises RLCD::FormatError for a malformed file. detector.model is the RLCD::Model, and it carries the optimizer state, so training can resume with RLCD::Trainer. A detector reuses one parser, so give each fiber its own.

Measured on this machine: 1500 chunks of 25 lines per language for C, JavaScript, JSON, Python, Ruby and Rust, with 8-line chunks of other files held out. Detection was 96.0% accurate with ECE 0.020 without the parse check (0.5 ms per snippet), and 96.1% with ECE 0.017 with it (1.3 ms). The I8 model, optimizer state included, is 221 KB.

Spec fixture

require "treesitter/fixture" (not loaded by require "treesitter") adds TreeSitter::Fixture. It compiles the bundled JSON grammar (fixtures/json, MIT, from tree-sitter-json) with cc into a temporary directory on first use:

json = TreeSitter::Fixture.language("json")
TreeSitter::Fixture.library("json")   # path of the compiled .so
TreeSitter::Fixture.clean             # removes the compiled libraries

Man pages

man/ holds man pages generated from this README and the API doc comments: treesitter(7cr) for the overview and one section 3 page per type, such as TreeSitter::Node(3cr). Read them from the installed shard:

man -M lib/treesitter/man TreeSitter::Parser
man -l lib/treesitter/man/man3/TreeSitter::Parser.3cr

License

MIT. The bundled JSON grammar keeps its own MIT license in fixtures/json/LICENSE.