cdp.cr

A Chrome DevTools Protocol client for Crystal.

This is the low-level half of a two-shard stack: cdp.cr speaks the protocol, and a Playwright-style API on top of it is being written as a separate shard. Nothing here knows about locators or auto-waiting — it is the transport, the session tree, and typed bindings for every command and event Chrome understands.

Status: usable. It starts a browser, runs typed commands, delivers typed events and cleans up after itself. The auto-waiting layer above it has not been written yet.

require "cdp"

CDP.launch do |browser|
  root = browser.connection.root

  target = root.execute(CDP::Protocol::Target::CreateTargetRequest.new(url: "about:blank"))
  page = root.attach(target.target_id)
  page.execute(CDP::Protocol::Page::EnableRequest.new)

  loaded = page.expect(CDP::Protocol::Page::LoadEventFiredEvent)
  page.execute(CDP::Protocol::Page::NavigateRequest.new(url: "https://example.com"))
  loaded.wait(10.seconds)

  title = page.execute(CDP::Protocol::Runtime::EvaluateRequest.new(
    expression: "document.title", return_by_value: true))
  puts title.result.value
end

What is generated

The whole protocol, from a pinned schema:

A command's request type names its own response type, so there is nothing to cast and nothing to look up:

response = session.execute(CDP::Protocol::Page::NavigateRequest.new(url: url))
response.frame_id # => typed

Events are subscribed to by class, which is what makes the payload typed and what keeps the ones nobody wants from being parsed at all:

session.on(CDP::Protocol::Target::AttachedToTargetEvent) do |event|
  event.target_info.type
end

Passing something that is not a command to execute, or something that is not an event to on, is a compile error that says so.

Safe by default

CDP.launch with no arguments is the safe configuration. Everything that trades one of these away has to be asked for by name.

No debugging port exists. The browser is driven over a pipe. The protocol has no authentication of any kind — whatever can reach the endpoint owns the browser, its cookies, and the filesystem through file:// — so the strongest available statement about that port is that there is not one. lsof on a launched browser shows nothing listening.

The browser cannot outlive the program. A browser started on a pipe exits when the pipe closes, so our process dying, however violently, takes it with it. There is no orphan-sweeping heuristic here because none is needed.

The sandbox stays on. --no-sandbox never appears by default. Running as root with the sandbox enabled fails with an explanation instead of quietly disabling it, which is the usual way this protection disappears from a project. Passing --no-sandbox through args is refused: it has to be CDP.launch(sandbox: false), so the decision is visible at the call site.

The profile is private and temporary. A fresh directory created 0700, removed when the browser closes.

Messages have a size limit. A page decides how large some responses are — a screenshot arrives as base64 inside JSON — and the standard library's WebSocket has no ceiling. The limit is checked against the declared length before anything is allocated.

The protocol log is redacted. Turning on cdp.protocol logging is otherwise the moment a program starts writing every cookie and every typed password to a file. Values are masked and the structure is kept, so the log still answers what it was opened to answer. See CDP::Redaction.

An optional navigation policy, off by default, refuses file://, chrome:// and addresses on the local network including the cloud metadata address. Its documentation is explicit that it is defence in depth and not a boundary.

Design notes

The bindings are generated ahead of time and committed. Generating them in a macro was considered and rejected: the standard library warns against long-running macro programs, and macro-built types are invisible to crystal docs and to editor completion, which is most of the reason for having typed bindings at all. A CI job re-runs the generator and fails on a dirty tree, so what is in git cannot drift from the schema it claims to describe.

Every response and event tolerates unknown fields. Chrome ships protocol fields ahead of the published schema constantly. Every generated response and event includes JSON::Serializable::Unmapped, so a newer browser neither breaks parsing nor silently loses data — the extra fields survive in json_unmapped and round-trip back out.

Enums convert explicitly. 347 wire values contain a dash, which is not a valid Crystal identifier, so each enum gets a generated from_wire/to_wire pair. The alternative some shards reach for — monkey-patching Enum.parse? globally — quietly changes the meaning of every enum in the program that includes them. An unrecognised value raises CDP::UnknownEnumValue, which names the enum and the value rather than reporting a JSON error at a byte offset.

Two types are classes, not structs. DOM.Node and Runtime.StackTrace refer to themselves, and a struct cannot contain itself. The generator finds them by walking the reference graph rather than by hardcoding, so a third one appearing in a future schema is handled without anyone noticing.

The schema is vendored, not downloaded. vendor/ holds both JSON files, the upstream revision, and their checksums. Nothing reaches the network during a build, and a spec verifies the checksums still match.

The WebSocket frame codec is written here. The standard library's is :nodoc:, and the public HTTP::WebSocket wrapper offers no message size limit, no read timeout and no way to set the Host header — the three things the threat model needs. The codec accepts only the subset CDP uses and refuses everything else: reserved bits, unknown opcodes, binary frames, masked frames from a server, oversized or fragmented control frames.

Every command carries a deadline, and a command that times out removes itself from the pending table. This is the gap in every prior-art shard: a browser that stops answering must surface as an error, not as a fiber nobody ever wakes up.

Subscriber code never runs on the fiber reading the socket. Events go through a queue drained by a separate fiber. This is not tidiness: a handler that issues a command and waits for the reply would deadlock if the reader were the fiber that has to deliver it. There is a spec that does exactly that.

Development

shards install
crystal run tools/generate.cr        # regenerate bindings from vendor/
crystal spec                         # everything, including a real browser
crystal spec --tag "~integration"    # no browser needed
CDP_SPEC_PARALLEL=8 crystal spec     # with the runtime's parallelism on
crystal tool format --check
./bin/ameba
crystal docs

tools/generate.cr is a thin command over tools/generator.cr; the split exists so the golden specs can feed a hand-written schema straight into the generator and compare the output.

About the specs

Four suites, failing in deliberately different ways.

spec/generator/golden_spec.cr runs minimal hand-written schemas — one per decision the generator makes — against exact expected output. A diff here tells you which decision changed. Accept an intended change with UPDATE_GOLDEN=1 crystal spec, and read the diff before committing it.

spec/generated_spec.cr is itself generated: it parses a schema-conformant document for all 1,950-odd types and round-trips every enum member, 2,191 exercises in total. It exists because crystal build proves almost nothing about generated code — the compiler never typechecks a method body nobody calls, so a binding tree full of nonsense compiles perfectly happily. That was verified by planting a deliberately broken method in a domain file and watching the build stay green.

spec/connection_spec.cr and friends run against a scriptable fake browser (spec/support/fake_cdp_server.cr) that will do what a real one will not: answer nothing, answer twice, drop the connection with commands in flight, send a message past the limit, split one across fifty frames.

spec/launcher_spec.cr and spec/protocol_drift_spec.cr start a real browser and assert with lsof and ps — "we open no port" is a claim about the process table, and the honest way to check it is to look there.

On tests that cannot fail

Every load-bearing spec here has been checked by breaking the thing it covers and confirming it goes red. Two were found to be worthless that way and rewritten:

• The obvious proof that sends need a lock is "concurrent sends tear a frame in half". They do not, measurably — this runtime serialises writes to one file descriptor — so that spec passed with the lock removed. The real proof is a thousand commands from eight fibers with CDP_SPEC_PARALLEL=8, where removing the lock produces hundreds of mismatched replies. The default run cannot catch it, which is why CI runs the suite twice.
• The same thousand-command spec did not prove replies are matched by id: handing each reply to the oldest waiting caller passed it, because replies arrive in the order the commands were sent. Proving it needed a separate spec that answers three commands in reverse.

License

MIT.