Alys

Alys is a Crystal memory usage tracer, tracking memory allocs, re-allocs, and frees for later analysis.

THIS IS ALPHA QUALITY SOFTWARE!

Installation

1. Add the dependency to your shard.yml:

   dependencies:
     alys:
       git: https://git.sr.ht/~refi64/alys
   

2. Run shards install

Usage

Setting Up Tracing

require "alys"

Alys.setup_from_env

Now, to enable tracing run your application with:

ALYS_TRACING=file crystal myapp.cr

This will trace to a file TIME.alys, where TIME is an ISO 8601 timestamp. To change the file's name, use:

ALYS_TRACING=file:myfile.alys ./myapp

Examing Traces with alys_converter

The .alys file is in a custom binary format, but it can be converted into JSON, Go pprof, folded stacks (for use with flamegraph), or a direct flamegraph via:

# NOTE: passing `--symbolize` with the binary is *required* on systems where
# ALYS_BACKTRACE_TYPE=addr by default (see below).

# Outputs myfile.alys.json:
bin/alys_converter --symbolize myprog myfile.alys
# You can rename the output file:
bin/alys_converter --symbolize myprog myfile.alys myfile.json
# and/or format & indent the JSON file:
bin/alys_converter --symbolize myprog --indent myfile.alys
bin/alys_converter --symbolize myprog --indent myfile.alys myfile.json

# Outputs myfile.pb.gz (pprof format):
bin/alys_converter --symbolize myprog -f pprof myfile.alys
# You can rename the output file:
bin/alys_converter --symbolize myprog -f pprof myfile.alys myfile.pb.gz

# Outputs myfile.folded (folded stacks):
bin/alys_converter --symbolize myprog -f folded-stacks myfile.alys
# You can rename the output file:
bin/alys_converter --symbolize myprog -f folded-stacks myfile.alys myfile.folded

# Outputs myfile.svg (flamegraph, via inferno-flamegraph):
bin/alys_converter --symbolize myprog -f inferno-flamegraph myfile.alys
# You can rename the output file:
bin/alys_converter --symbolize myprog -f inferno-flamegraph myfile.alys myfile.svg

(Note that .alys files are only compatible with an identical alys_converter version!)

JSON

alys_converter's JSON format is simply an array of event objects like so:

{
  "id": 29,
  "time": 1.009839517,
  "kind": "alloc",
  "addr": 281471443894272,
  "size": 96,
  "stack": [
    {
      "ip": 4721576,
      "line": 279,
      "file": "/home/ryan/code/alys/src/alys.cr",
      "name": "record_alloc"
    },
    ...
  ]
}

• id is a unique ID assigned to each allocation, with reallocation & free calls using the same ID as the original allocation. In other words, you can use this to link together allocation and free calls.
• time is the number of seconds since program start that this occurred.
• kind is one of alloc, realloc, or free.
• addr is the memory address that was allocated or freed.
• size is the number of bytes allocated.
• For realloc events, both of the above point to the new size and address, and two additional keys store the previous values, prev_addr and prev_size.
• stack is an array of stack frames, each containing (only ip is guaranteed to be present):
  • ip is the instruction pointer of this frame.
  • file is the full filename
  • line is the line number
  • name is the function name

You can see some (ugly) example usage of the JSON file for analysis at this Colab notebook)

pprof

pprof files can be visualized with Google's pprof CLI tool. Four sample indexes are available (with similar meanings as to Go's):

• alloc_objects: number of objects allocated over the course of the program
• alloc_space (default): amount of bytes allocated over the course of the program
• inuse_objects: number of objects allocated at the time of program termination
• inuse_space (default): amount of bytes allocated at the time of program termination

Most likely, you'd want to analyze the data with the pprof web UI; to start it on PORT, use:

pprof -http localhost:PORT myfile.pb.gz

Folded stacks and Inferno

Folded stacks files are usually converted to flamegraphs flamegraphs; you can also go straight from .alys -> flamegraph via the inferno-flamegraph format (requires Inferno to be installed). In the latter case, additional options can be passed to the inferno-flamegraph CLI via --inferno-opt:

# Pass --flamechart and --title=test to inferno-flamegraph:
bin/alys_converter -f inferno-flamegraph myfile.alys \
  --inferno-opt flamechart --inferno-opt title=test

Backtraces

By default, Alys will also capture a backtrace on each allocation or reallocation, which can result in significant slowdowns. You can control this via ALYS_BACKTRACE_TYPE:

• ALYS_BACKTRACE_TYPE=none will capture no backtrace at all.
• ALYS_BACKTRACE_TYPE=addr will create backtraces that only contain the function's memory addresses (i.e. no names, files, or line numbers). A backtrace like this, however, is not very useful, so alys_converter can resolve them when running alys_converter via:

  bin/alys_converter --symbolize ./myapp myfile.alys
  

  There are a few things to note:
  • This only works on specific platforms:
    • Linux.
    • (untested) macOS x64, when compiled with -Wl,-no_pie. M1 macs do not support this option. PIE randomizes the memory location of the executable at runtime, so it's impossible to know what was where after the fact.
  • llvm-symbolizer must be installed. This is the default on non-macOS systems.
• ALYS_BACKTRACE_TYPE=name will create backtraces that contain the function names but no files or line numbers. This is usually only slightly slower than ALYS_BACKTRACE_TYPE=addr, and it supports position-independent executables on macOS, but the symbol names aren't as detailed as using alys_converter's symbolization. This is the default on macOS.
• ALYS_BACKTRACE_TYPE=full will create backtraces that contain the function names, as well as the source file paths and line numbers.

TODO: add support for & document flipping Alys on / off at runtime

Performance Notes

Sampling intervals

Instead of writing an event for every trace, you can have Alys write an event for every N bytes allocated by setting ALYS_SAMPLE_INTERVAL=bytes:N. This should significantly increase the performance of allocations at the cost of some granularity.

Release Builds

Gathering backtraces on release builds is significantly slower than debug builds!

This is because release builds omit the frame pointer / base pointer, which is used to quickly go back up the call stack. Without this, backtrace creation needs to look into the embedded unwind information, which is significantly slower.

Basic Benchmarks

These were gathered by running the Lucky website locally, then timing:

wget --recursive --max-redirect 0 127.0.0.1:5000

In other words, this would run the Lucky website and then download every single page on the site.

Build Type	Tracing Enabled?	Backtrace Type	Backtrace Limit	Time
debug	N	N/A	N/A	1.2s
debug	Y	none	N/A	2.7s
debug	Y	addr	5	7s
debug	Y	addr	10	10s
debug	Y	addr	unlimited	23s
debug	Y	name	5	9s
debug	Y	name	10	14s
debug	Y	name	unlimited	33s
debug	Y	full	5	1m8s
debug	Y	full	10	3m25s
debug	Y	full	unlimited	8m47s
release	N	N/A	N/A	0.9s
release	Y	none	N/A	2.2s
release	Y	addr	5	34s
release	Y	addr	10	1m16s
release	Y	addr	unlimited	1m21s
release	Y	name	5	36s
release	Y	name	10	1m18s
release	Y	name	unlimited	1m25s
release	Y	full	5	8m4s
release	Y	full	10	timeout
release	Y	full	unlimited	timeout

libunwind

On Linux systems, the default glibc unwind implementation is significantly slower than alternatives, which will result in significant performance loss if backtraces are enabled. As a result, Alys will try to configure your program to use LLVM's libunwind if available, printing a warning when shards is run otherwise.

IMPORTANT NOTE: There are multiple implementations of libunwind available that may not be compatible. In particular, there's nongnu libunwind, which is incompatible with the standard libunwind and will not work. Alys will automatically avoid using this libunwind, but you may have some confusion if you think you installed "libunwind" but Alys is not using it.

Installing LLVM's libunwind

After running any of the below steps, you can run lib/alys/tools/detect_libunwind.sh to ensure that LLVM libunwind is detected.

Any Distro

Alys can build and use its own local copy of libunwind. In order to do this, just run:

$ lib/alys/tools/build_libunwind.sh

Fedora 36+

$ sudo dnf install llvm-libunwind

Debian 12+

# Find the libunwind version corresponding to the Debian release's primary
# LLVM package.
$ LIBUNWIND_VERSION=$(apt-cache depends llvm | egrep -o 'llvm-[0-9]+' | cut -d- -f2)
$ sudo apt install libunwind-$LIBUNWIND_VERSION

Debian/Ubuntu

Add the official LLVM APT repo, then run:

$ sudo apt install libunwind-13

Alpine

$ sudo apk add llvm-libunwind

Development

TODO

Contributing

Please see the guide for submitting patches on git.sr.ht. (If you choose to use git send-email, the patches should be sent to ~refi64/alys-devel@lists.sr.ht.)ODO

Contributors

• Ryan Gonzalez - creator and maintainer