Kalubampa: Polyglot High-Performance Web Crawling & Parsing Architecture

Kalubampa is a large-scale web crawling and data extraction system designed using a Polyglot Microservices architecture. The system combines the strengths of three different programming languages (Ruby, Crystal, and Rust) unified by a Message Broker (Redis) and a Foreign Function Interface (FFI) to achieve high performance, maximum concurrency, and robust memory safety.

1. Architecture Overview

The system is divided into three main components, each playing a specialized role analogized as The Brain, The Muscle, and The Shield:

graph LR
    subgraph "The Brain — Ruby on Rails"
        A["Admin Dashboard :3000"] --> B["Sidekiq Result Collector"]
        A -->|LPUSH tasks| R["Redis :6379"]
        B -->|Fetch results| R
        B -->|Bulk Insert| P["PostgreSQL :5432"]
    end

    subgraph "The Muscle — Crystal Worker"
        C["Crystal Fibers"] -->|BRPOPLPUSH| R
        C -->|HTTP GET| W["Target Websites"]
        C -->|RPUSH results| R
        C -->|FFI C-ABI| D
    end

    subgraph "The Shield — Rust Parser"
        D["Rust Parsing Engine"]
    end

Core Component Details

1. The Brain: Ruby on Rails (PostgreSQL)

   • Function: A modern monolithic application providing the Admin Dashboard, Internal APIs, relational database management, authentication, and scheduling (Cron/Sidekiq).
   • Rationale: Excellent developer velocity and a highly mature ActiveRecord ecosystem to manage complex business logic.

2. The Muscle: Crystal (Kemal / Redis Worker)

   • Function: The driving engine for network I/O. Responsible for launching millions of HTTP requests concurrently without overloading the operating system, powered by Crystal Fibers.
   • Rationale: Features an elegant Ruby-like syntax but is compiled Ahead-Of-Time (AOT) to machine code. Significantly lower memory footprint compared to Node.js and more readable than Go.

3. The Shield & Scalpel: Rust (via C-ABI)

   • Function: The HTML parsing and extraction engine (CPU-bound). It processes malformed HTML DOM arrays and extracts targeted data points using regular expressions or CSS selectors.
   • Rationale: Provides C/C++ level computational speed and utilizes a Borrow Checker that guarantees immunity against Segfaults and Buffer Overflows. This is highly essential when processing random, potentially corrupted data from the internet.

2. System Interaction Diagram

The following diagram visualizes the data flow and inter-service interactions from the user interface down to low-level extraction:

[ Admin Web UI ] <---(WebSockets)---> [ Ruby on Rails (Port 3000) ]
                                            |   ^
                         (Bulk Insert)      |   | (Fetch Results via Sidekiq)
                                            v   |
                                       [ PostgreSQL ]
                                            
                                       [ Redis Broker ]
                                (Port 6379 / In-Memory RAM)
                                      ^               ^
             (LPUSH Tasks)            |               |  (RPUSH Results)
             [ Rails ] ---------------+               +--- [ Crystal Workers ]
                                                                  |
                                                                  v (HTTP GET)
                                                           [ Target Websites ]
                                                                  |
                                                                  v (Raw HTML)
                                                          [ FFI Bridge (C-ABI) ]
                                                                  |
                                                                  v (Unsafe Pointers)
                                                          [ Rust Parsing Engine ]

3. Workflow & System Execution Path

1. Task Initiation by User (The Brain)

   • The user interacts with the Ruby on Rails Admin Dashboard interface to create a new campaign or upload a list of target URLs.
   • Rails pushes these tasks in bulk (LPUSH) into the Redis Message Broker queue under the kalubampa:task_queue key.

2. Concurrent Network Processing (The Muscle)

   • Fail-Safe Task Pickup: The Crystal Worker fetches target URLs from Redis using the atomic BRPOPLPUSH command, moving the active task to a processing queue to ensure no data is lost if the worker crashes unexpectedly.
   • HTML Ingestion: The Crystal Worker executes high-concurrency mass HTTP GET requests to the target websites using non-blocking Fibers to retrieve the raw HTML content.

3. Secure Data Extraction (The Shield)

   • FFI Bridge: Crystal sends the raw HTML text to the Rust Parsing Engine over a Foreign Function Interface (FFI) using the C-ABI standard (Unsafe Pointers). Crystal interacts entirely using C layout conventions.
   • DOM Dissection: Rust parses and extracts the requested data via Regex or CSS Selectors securely without risking memory segmentation faults.
   • Memory Cleanup Protocol: Once extraction completes and data is passed back, Crystal must immediately call a dedicated Rust cleanup routine (e.g., free_product) to release the memory allocated on the Rust side.

4. Bulk Storage & Final State

   • Network Isolation Zone: Crystal Workers are strictly restricted from directly interacting with PostgreSQL or the Rails API. The extracted JSON payload is sent (RPUSH) back into Redis at kalubampa:result_queue.
   • Result Collector: A background Sidekiq Worker on the Rails side continuously pops data from kalubampa:result_queue.
   • Bulk Insert: Rails batch-processes the incoming JSON using the ActiveRecord insert_all method directly into PostgreSQL for high-speed, write-heavy throughput. The processed results are broadcasted in real-time to the Admin Dashboard.

4. Monorepo Directory Structure

The project is organized as a monorepo with the following layout:

kalubampa/
├── dashboard/                 # [RUBY] Rails App (The Brain)
│   ├── app/
│   │   ├── controllers/
│   │   ├── models/
│   │   ├── jobs/              # Sidekiq Workers (Result Collector)
│   │   └── views/
│   ├── config/
│   └── Gemfile
├── worker/                    # [CRYSTAL] Concurrency App (The Muscle)
│   ├── src/
│   │   ├── kalubampa_worker.cr
│   │   ├── ffi_bindings.cr
│   │   ├── redis_client.cr
│   │   └── fetcher.cr
│   └── shard.yml
├── parser/                    # [RUST] HTML Extraction Engine (The Shield)
│   ├── src/
│   │   └── lib.rs
│   ├── include/
│   │   └── rust_parser.h
│   └── Cargo.toml
├── docker/                    # Infrastructure & Environment Configurations
│   ├── docker-compose.yml
│   ├── Dockerfile.rails
│   └── Dockerfile.worker
└── Makefile

5. Docker Infrastructure & Service Management

The system leverages Docker Compose to orchestrate all essential underlying infrastructure dependencies with the following configurations:

• PostgreSQL 16 Alpine: Serves as the primary relational store optimized for high write-heavy ingestion rates.
• Redis 7 Alpine: Functions as the in-memory message broker handling the kalubampa:task_queue and kalubampa:result_queue structures.
• Additional Operational Features:
  • Configured persistent volumes to guarantee data reliability for both PostgreSQL and Redis.
  • An isolated application bridge network called kalubampa_net.
  • Integrated service health checks ensuring all database and caching sub-systems are fully healthy and ready to accept connections before the primary application runtimes initialize.