onnxruntime.cr

ONNX Runtime bindings for Crystal

Installation

1. Install ONNX Runtime

   Download and install the ONNX Runtime from the official releases.

   For Linux:

   # Example for Linux
   wget https://github.com/microsoft/onnxruntime/releases/download/v1.21.0/onnxruntime-linux-x64-1.21.0.tgz
   tar -xzf onnxruntime-linux-x64-1.21.0.tgz
   export ONNXRUNTIME_DIR=/path/to/onnxruntime-linux-x64-1.21.0
   

   For macOS:

   # Example for macOS (arm64)
   curl -L https://github.com/microsoft/onnxruntime/releases/download/v1.21.0/onnxruntime-osx-arm64-1.21.0.tgz -o onnxruntime-osx-arm64-1.21.0.tgz
   tar -xzf onnxruntime-osx-arm64-1.21.0.tgz
   export ONNXRUNTIME_DIR=/path/to/onnxruntime-osx-arm64-1.21.0
   

2. Add the dependency to your shard.yml:

   dependencies:
     onnxruntime:
       github: kojix2/onnxruntime.cr
   

3. Run shards install

Usage

require "onnxruntime"

# Load a model
model = OnnxRuntime::Model.new("path/to/model.onnx")

# Print model inputs and outputs
puts "Inputs:"
model.inputs.each do |input|
  puts "  #{input[:name]}: #{input[:type]} #{input[:shape]}"
end

puts "Outputs:"
model.outputs.each do |output|
  puts "  #{output[:name]}: #{output[:type]} #{output[:shape]}"
end

# Prepare input data
input_data = {
  "input_name" => [1.0_f32, 2.0_f32, 3.0_f32]
}

# Run inference
result = model.predict(input_data)

# Process results
result.each do |name, data|
  puts "#{name}: #{data}"
end

MNIST Example

Download the MNIST model: mnist-12.onnx (raw

require "onnxruntime"

# Load the MNIST model
model = OnnxRuntime::Model.new("mnist-12.onnx")

# Create a dummy input (28x28 image draw 1)
input_data = Array(Float32).new(28 * 28) { |i| (i % 14 == 0 ? 1.0 : 0.0).to_f32 }

# Run inference
result = model.predict({"Input3" => input_data}, ["Plus214_Output_0"], shape: {"Input3" => [1_i64, 1_i64, 28_i64, 28_i64]})

# Get the output probabilities
probabilities = result["Plus214_Output_0"].as(Array(Float32))

# Find the digit with highest probability
predicted_digit = probabilities.index(probabilities.max)
puts "Predicted digit: #{predicted_digit}"

# Explicitly release resources
model.release
OnnxRuntime::InferenceSession.release_env

Memory Management

Currently, you must explicitly release resources when you're done with them:

# Create and use model
model = OnnxRuntime::Model.new("path/to/model.onnx")
result = model.predict(input_data)

# When finished, explicitly release resources
model.release
OnnxRuntime::InferenceSession.release_env

For long-running applications like web servers, consider setting up signal handlers to ensure resources are properly released on shutdown:

Signal::INT.trap do
  puts "Shutting down..."
  model.release
  OnnxRuntime::InferenceSession.release_env
  exit
end

See the examples directory for more detailed implementations.

Why do you need to manually free memory?

Previously, the following error was displayed on macOS.

libc++abi: terminating due to uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument
Program received and didn't handle signal ABRT (6)

This seems to be related to multithreading and mutex. According to the AI, this is difficult to solve with finalize, so we tried to solve it by creating a reference counter, but we were unable to solve it in the end. If you can solve this problem, please create a pull request!

Development

The code is generated by AI and may not be perfect. Please feel free to contribute and improve it.

Contributing

1. Fork it (https://github.com/kojix2/onnxruntime.cr/fork)
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request