crig-sqlite

Crystal vector store powered by SQLite and sqlite-vec.

Port status: ✅ Complete - Full vector search capabilities with sqlite-vec extension

Quick Start

Installation

1. Add to your shard.yml:

dependencies:
  crig-sqlite:
    github: dsisnero/crig-sqlite
  crig:
    github: dsisnero/crig

2. Install dependencies and extension:

make deps
# or separately:
# shards install
# make install-extension

Basic Usage

In-Memory Database (Fast, No Persistence)

require "crig-sqlite"

# Open database with vector extension (auto-downloads if needed)
db, loaded = CrigSqlite::Extensions.open_with_vec(":memory:")

if loaded
  # Create embedding model
  model = SimpleEmbeddingModel.new

  # Create vector store
  store = CrigSqlite::SqliteVectorStore(SimpleEmbeddingModel, Article).new(db, model)
  store.create_table(model.ndims)

  # Add documents with embeddings
  articles = [
    Article.new("1", "Introduction to AI", "AI is transforming the world"),
    Article.new("2", "Machine Learning", "ML algorithms learn from data"),
  ]
  embeddings = model.embed_texts(articles.map(&.content))
  store.add_rows(articles.zip(embeddings.map { |e| ::Crig::OneOrMany(::Crig::Embedding).one(e) }))

  # Search
  index = store.index(model)
  request = ::Crig::VectorStore::VectorSearchRequest(CrigSqlite::SqliteSearchFilter).builder
    .query("artificial intelligence")
    .samples(5_u64)
    .build

  results = index.top_n(request, Article)
  results.each do |score, id, article|
    puts "Score: #{score.round(4)} | #{article.title}"
  end
end

db.close

File Database (Persistent Storage)

require "crig-sqlite"

# Open file database - data persists between runs
db, loaded = CrigSqlite::Extensions.open_with_vec("my_vectors.db")

if loaded
  model = SimpleEmbeddingModel.new
  store = CrigSqlite::SqliteVectorStore(SimpleEmbeddingModel, Article).new(db, model)

  # Create tables only if not exists
  store.create_table(model.ndims)

  # Add documents (only if empty)
  if db.scalar("SELECT COUNT(*) FROM articles").as(Int64) == 0
    articles = [
      Article.new("1", "Introduction to AI", "AI is transforming the world"),
      Article.new("2", "Machine Learning", "ML algorithms learn from data"),
    ]
    embeddings = model.embed_texts(articles.map(&.content))
    store.add_rows(articles.zip(embeddings.map { |e| ::Crig::OneOrMany(::Crig::Embedding).one(e) }))
  end

  # Search persists across runs
  index = store.index(model)
  request = ::Crig::VectorStore::VectorSearchRequest(CrigSqlite::SqliteSearchFilter).builder
    .query("artificial intelligence")
    .samples(5_u64)
    .build

  results = index.top_n(request, Article)
  results.each do |score, id, article|
    puts "Score: #{score.round(4)} | #{article.title}"
  end
end

db.close
# Database file "my_vectors.db" still exists with all data

How It Works

crig-sqlite uses the sqlite-vec extension to enable vector similarity search in SQLite. The extension is automatically downloaded when you run make install-extension or when you first use open_with_vec.

Extension Loading

1. Automatic: When you call CrigSqlite::Extensions.open_with_vec, it:

   • Checks if the extension is already installed
   • Downloads the correct pre-built extension for your platform
   • Loads it using SQLite's extension loading mechanism

2. Manual: You can also download and install manually:

   make install-extension
   

3. Environment variable: Set SQLITE_VEC_PATH to point to your extension:

   export SQLITE_VEC_PATH=/path/to/vec0.dylib
   

Supported Platforms

macOS Setup

macOS requires Homebrew SQLite for extension loading:

brew install sqlite
export LIBRARY_PATH="$(brew --prefix sqlite)/lib:$LIBRARY_PATH"

The Makefile automatically detects and uses Homebrew SQLite.

Development

# Clone and setup
git clone https://github.com/dsisnero/crig-sqlite
cd crig-sqlite
make deps

# Run tests
make test

# Format and lint
make format
make lint

# Package for release
make package

API Reference

CrigSqlite::Extensions

• Extensions.open_with_vec(path) : Tuple(DB::Database, Bool) - Open database with extension loaded
• Extensions.find_sqlite_vec_extension : String? - Find extension path
• Extensions.sqlite_vec_loaded?(db) : Bool - Check if extension is loaded

CrigSqlite::SqliteVectorStore(E, T)

• SqliteVectorStore.new(db, embedding_model) - Create store
• #create_table(dims) - Create vector table
• #add_rows(documents) - Add documents with embeddings
• #index(model) - Create search index

CrigSqlite::SqliteVectorIndex(E, T)

• #top_n(request, type) - Search for similar documents
• #top_n_ids(request) - Search returning only IDs

Contributing

1. Fork it (https://github.com/dsisnero/crig-sqlite/fork)
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request

License

MIT - See LICENSE for details.

Acknowledgments

• sqlite-vec - The SQLite vector extension
• Rig - The upstream Rust implementation