Crystal fibers and the MSVC C++ exception state, on Windows

Experiments around one fault. On Windows with the MSVC toolchain, a Crystal fiber that waits inside an ensure while another fiber on the same thread raises can end the process with 0xC0000409 when the ensure finishes. No rescue sees it, and nothing is printed. Crystal 1.21.0 and 1.22.2 both do it.

This repository holds a reproduction, two candidate remedies, and the tests that judge them. They run on GitHub's Windows runners (.github/workflows/windows.yml), or on any Windows machine with run.ps1.

What is going on

From Microsoft's runtime source and the 14.44 and 14.51 binaries:

• vcruntime keeps each thread's C++ exception state in one structure, __vcrt_ptd (128 bytes on x64), held in a fiber-local storage (FLS) slot. Windows' own fibers each get their own, since SwitchToFiber switches FLS. Crystal's fibers switch stacks themselves, so every Crystal fiber on a thread shares one.
• With MSVC, Crystal runs an ensure body inside the catch funclet and then rethrows with _CxxThrowException(NULL, NULL).
• Each catch pushes a record onto the chain at offset 0x58 of that structure. _CreateFrameInfo links a new record only if it lies below the head, which on one stack always holds.
• A fiber waiting in an ensure is inside a catch, with its record at the head. If another fiber, whose stack lies above, catches meanwhile, its record starts a new chain, and popping it leaves the chain empty. When the first fiber's ensure ends, _FindAndUnlinkFrame does not find its record and calls abort, which ends with __fastfail: 0xC0000409.

So the account predicts a crash only when the raiser's stack lies above the waiter's. T1's controls check that prediction.

Two remedies tried here

• R0 (r0/cxx_fiber_state.cr): at every Crystal fiber switch, save the outgoing fiber's __vcrt_ptd and install the incoming one's, through vcruntime's own FLS slot, as SwitchToFiber would. A dead fiber's structure goes to a reaper thread, which hands it back to vcruntime's own FLS callback by setting it in a Windows fiber and deleting that fiber. It hooks Fiber.swapcontext, sets itself up from Crystal.init_runtime, and stops the process (exit code 70, with a line on standard error) when it cannot run safely. It builds only for Crystal 1.21.0 to 1.22.2 (-Dcxx_fiber_state_unverified to build anyway), and admits only the vcruntime140.dll builds whose layout has been read: 14.44.35211.0 and 14.51.36247.0. CXX_FIBER_STATE=off turns it off; CXX_FIBER_STATE=trust-runtime admits another build. r0/cxx_fiber_detector.cr adds checks of R0's invariant, for tests.
• P2 (p2/): a change to Crystal 1.22.2's compiler, src/compiler/crystal/codegen/exception.cr, that leaves the catch funclet before the ensure body runs and throws the saved exception again after it, so that no Crystal code runs inside a catch funclet. p2/exception.cr is that file with the change, and p2/ensure-outside-funclet.patch the change alone.

The experiments

run.ps1 prints each result as PASS, FAIL, RECORD (measured, not judged) or NEWS (against the prediction) in results\<date-time>\summary.txt.

Running

On a Windows machine with Visual Studio's C++ tools and Crystal for Windows (MSVC) in C:\crystal\1.22.2 and C:\crystal\1.21.0:

powershell -ExecutionPolicy Bypass -File run.ps1                 # stages A and B
powershell -ExecutionPolicy Bypass -File run.ps1 -Stages E       # elevated: page heap, a share, owners

The workflow runs on every push to main, and by hand from the Actions tab:

• parse: parses the PowerShell with 5.1 and 7.
• linux: T6 on Linux, every program type-checked for Windows, and T13, with 1.21.0 and 1.22.2.
• windows: stages A, B and E on windows-2022 and windows-2025, with each compiler. Each job's summary page shows its summary.txt, and its results are an artifact.
• p2: downloads LLVM 23.1.1 for MSVC, builds the patched compiler, and runs P2.

Limits of the runners: they run elevated, so nothing here runs as a plain user; they are Windows Server, not Windows 10 or 11; and timings on shared machines are noisy, so P5 takes several runs each way and gives the spread.

Notes

• p2/exception.cr is a file of Crystal's compiler (Apache-2.0) with the change above.
• No Microsoft code is kept here or uploaded. P1 reads vcruntime140.dll's disassembly on the runner and writes only facts about it; the crash dump stays on the runner too.
• This is an experiment, not a library.