awscr-s3

A Crystal shard for S3 and compatible services.

Installation

Add this to your application's shard.yml:

dependencies:
  awscr-s3:
    github: taylorfinnell/awscr-s3

Examples

Examples

Documentation

Documentation

Usage

require "awscr-s3"

Creating a Client

client = Awscr::S3::Client.new("us-east-1", "key", "secret")

For S3 compatible services, like DigitalOcean Spaces or Minio, you'll need to set a custom endpoint:

client = Awscr::S3::Client.new("nyc3", "key", "secret", endpoint: "https://nyc3.digitaloceanspaces.com")

If you wish you wish to you version 2 request signing you may specify the signer

client = Awscr::S3::Client.new("us-east-1", "key", "secret", signer: :v2)

List Buckets

resp = client.list_buckets
resp.buckets # => ["bucket1", "bucket2"]

Delete a bucket

client = Client.new("region", "key", "secret")
resp = client.delete_bucket("test")
resp # => true

Create a bucket

client = Client.new("region", "key", "secret")
resp = client.put_bucket("test")
resp # => true

Put Object

resp = client.put_object("bucket_name", "object_key", "myobjectbody")
resp.etag # => ...

You can also pass additional headers (e.g. metadata):

client.put_object("bucket_name", "object_key", "myobjectbody", {"x-amz-meta-name" => "myobject"})

Delete Object

resp = client.delete_object("bucket_name", "object_key")
resp # => true

Check Bucket Existence

resp = client.head_bucket("bucket_name")
resp # => true

Raises an exception if bucket does not exist.

Batch Delete Objects

resp = client.batch_delete("bucket_name", ["key1", "key2"])
resp.success? # => true

Get Object

resp = client.get_object("bucket_name", "object_key")
resp.body # => myobjectbody

# Or stream the object (recommended for large objects)
client.get_object("bucket_name", "object_key") do |obj|
  IO.copy(obj.body_io, STDOUT) # => myobjectbody
end

List Objects

client.list_objects("bucket_name").each do |resp|
  p resp.contents.map(&.key)
end

Upload a file

uploader = Awscr::S3::FileUploader.new(client)

File.open(File.expand_path("myfile"), "r") do |file|
  puts uploader.upload("bucket_name", "someobjectkey", file)
end

You can also pass additional headers (e.g. metadata):

uploader = Awscr::S3::FileUploader.new(client)

File.open(File.expand_path("myfile"), "r") do |file|
  puts uploader.upload("bucket_name", "someobjectkey", file, {"x-amz-meta-name" => "myobject"})
end

Creating a Presigned::Form.

form = Awscr::S3::Presigned::Form.build("us-east-1", "access key", "secret key") do |form|
  form.expiration(Time.unix(Time.now.to_unix + 1000))
  form.condition("bucket", "mybucket")
  form.condition("acl", "public-read")
  form.condition("key", SecureRandom.uuid)
  form.condition("Content-Type", "text/plain")
  form.condition("success_action_status", "201")
end

You may use version 2 request signing via

form = Awscr::S3::Presigned::Form.build("us-east-1", "access key", "secret key", signer: :v2) do |form|
  ...
end

Converting the form to raw HTML (for browser uploads, etc).

puts form.to_html

Submitting the form.

data = IO::Memory.new("Hello, S3!")
form.submit(data)

Creating a Presigned::Url.

options = Awscr::S3::Presigned::Url::Options.new(
   aws_access_key: "key",
   aws_secret_key: "secret",
   region: "us-east-1",
   object: "test.txt",
   bucket: "mybucket",
   additional_options: {
  "Content-Type" => "image/png"
})

url = Awscr::S3::Presigned::Url.new(options)
puts url.for(:put)

You may use version 2 request signing via

options = Awscr::S3::Presigned::Url::Options.new(
  aws_access_key: "key",
  aws_secret_key: "secret",
  region: "us-east-1",
  object: "test.txt",
  bucket: "mybucket",
  signer: :v2
)

Support 3rd party services

For S3-compatible services like DigitalOcean Spaces, Minio, or Backblaze B2, you'll need to set a custom endpoint.

Minio (local)

To use Minio locally, ensure your Minio server is running and configure the endpoint as follows:

options = Awscr::S3::Presigned::Url::Options.new(
  endpoint: "http://127.0.0.1:9000",
  region: "unused",
  aws_access_key: "admin",
  aws_secret_key: "password",
  bucket: "foo",
  force_path_style: true,
  object: "/test.txt"
)

url = Awscr::S3::Presigned::Url.new(options)
puts url.for(:get) # => "http://127.0.0.1:9000/foo/test.txt?X-Amz-Expires=86400&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=admin%2F20250313%2Funused%2Fs3%2Faws4_request&X-Amz-Date=20250313T235624Z&X-Amz-SignedHeaders=host&X-Amz-Signature=ce2b33af1dbfd0e132f31d3f9d716eb5d66f4d19fbdcb691e816f7033e345bce"

DigitalOcean Spaces

For DigitalOcean Spaces, the bucket is included in the subdomain:

options = Awscr::S3::Presigned::Url::Options.new(
  endpoint: "https://ams3.digitaloceanspaces.com",
  region: "unused",
  aws_access_key: "ACCESSKEYEXAMPLE",
  aws_secret_key: "SECRETKEYEXAMPLE",
  bucket: "foo",
  object: "/test.txt"
)

url = Awscr::S3::Presigned::Url.new(options)
puts url.for(:get) # => "https://foo.ams3.digitaloceanspaces.com/test.txt?X-Amz-Expires=86400&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ACCESSKEYEXAMPLE%2F20250313%2Funused%2Fs3%2Faws4_request&X-Amz-Date=20250313T235511Z&X-Amz-SignedHeaders=host&X-Amz-Signature=e7b17f3c335ed615bf68845f81c2091814f856b61d05cb5aae3ad664de0f1b6e"

Using a Custom HTTP Provider

By default, awscr-s3 creates a new standard HTTP::Client for each request. However, you may want more advanced connection management features, such as pooling or connection reuse. You can accomplish this by supplying your own HttpClientFactory to Awscr::S3::Http or Awscr::S3::Client.

Below is a minimal (and not production-ready) demonstration:

require "awscr-s3"
require "http"

class PoolingHttpClientFactory < Awscr::S3::HttpClientFactory
  getter pool : Array(HTTP::Client)
  @created_count : Int32 = 0

  def initialize(@pool_size : Int32 = 3)
    @pool = [] of HTTP::Client
  end

  def acquire_raw_client(endpoint : URI) : HTTP::Client
    if @pool.size > 0
      @pool.pop
    elsif @created_count < @pool_size
      @created_count += 1
      HTTP::Client.new(endpoint)
    else
      raise "No available clients in pool (limit of #{@pool_size} reached)"
    end
  end

  def release(client : HTTP::Client?)
    return unless client
    @pool << client
  end
end

Developing awscr-s3

To develop and run tests, use the following commands:

Run All Tests

To run all types of tests (unit and integration) by default, use:

crystal spec

Unit Tests

To run only the unit tests and exclude integration tests, use the --tag filter to exclude the integration tests:

crystal spec --tag '~integration'

Integration Tests

Integration tests test awscr-s3 against local S3 implementations, such as RustFS or SeaweedFS.

1. Start the RustFS server and any required services using Container Solution Compose:

   docker compose -f spec/integration/compose.yml up -d
   

2. Once the dependencies are up, run the integration tests:

   crystal spec --tag 'integration'