Logarithm: Anomaly Detection Agent

|Crystal| |GPLv3| |Version|

Logarithm is a self-learning diagnostics agent for GNU/Linux systems that uses machine learning to detect anomalies in system logs in real-time.

Built with Crystal for performance and reliability, it trains an autoencoder on normal log patterns to identify potential issues, security threats, or unusual behavior.

Features

• Multi-source Monitoring: Simultaneous ingestion from systemd journal and syslog files
• Real-time Detection: Continuous anomaly detection with configurable thresholds
• Machine Learning: TF-IDF vectorization and autoencoder-based unsupervised learning
• Incremental Retraining: Flexible retraining modes to adapt to evolving log patterns
• Security: AES-256 encryption, audit logging, and input validation
• Resilience: Retry logic, circuit breakers, and comprehensive error handling
• CLI Tools: Simple command-line interface for training and monitoring

Quick Start

Prerequisites: Crystal 1.17.1+, GNU Make, systemd dev libs (for journald)

.. code-block:: bash

git clone https://gitlab.com/renich/logarithm.git
cd logarithm
make release

Train the model (24 hours on default logs):

.. code-block:: bash

bin/logarithm train

Monitor for anomalies:

.. code-block:: bash

bin/logarithm monitor

Advanced usage: Use ./bin/logarithm --help for options. Supports multi-source monitoring, custom durations, and configuration files. See config.example.yml for configuration options.

Advanced Training Options

Logarithm supports flexible retraining strategies to adapt to evolving log patterns:

Incremental retraining (default, loads existing models and trains on new logs):

.. code-block:: bash

bin/logarithm train --retrain-mode incremental

Full retraining (ignores existing models, starts fresh training):

.. code-block:: bash

bin/logarithm train --retrain-mode full

Hybrid retraining (loads models but forces vocabulary expansion):

.. code-block:: bash

bin/logarithm train --retrain-mode hybrid

Expand vocabulary (add new terms to existing vectorizer vocabulary):

.. code-block:: bash

bin/logarithm train --expand-vocab

Rollback to previous model version (revert to backup models):

.. code-block:: bash

bin/logarithm train --rollback

Configuration

Logarithm supports configuration via YAML files, environment variables, and command-line flags. Settings are applied in this order of precedence:

1. Command-line flags (highest priority)
2. Environment variables
3. Configuration file
4. Built-in defaults (lowest priority)

Example config file:

.. code-block:: yaml

data_dir: ~/.local/share/logarithm
threshold: 0.85
duration: 48h
vocab_size: 100
batch_size: 10000
max_batches: 5

Environment variables:

• LOGARITHM_DATA_DIR: Model storage directory
• LOGARITHM_THRESHOLD: Anomaly detection threshold
• LOGARITHM_VOCAB_SIZE: TF-IDF vocabulary size
• LOGARITHM_BATCH_SIZE: Training batch size
• LOGARITHM_MAX_BATCHES: Maximum training batches

GPL v3 License

This project is licensed under the GPL v3 License - see the LICENSE_ file for details.

Authors

• Rénich Bon Ćirić - Creator and maintainer - renich@evalinux.com

Acknowledgments

• Built with Crystal <https://crystal-lang.org/>_
• Machine learning powered by num.cr <https://github.com/crystal-data/num.cr>_
• CLI framework by admiral <https://github.com/jwaldrip/admiral.cr>_

Documentation

• API Documentation: Generated from source code using make docs (uses README.rst)
• User Guide: See USER_GUIDE.rst for practical usage examples
• Fakelogs Guide: See tools/README.rst for testing tool documentation

.. |Crystal| image:: https://img.shields.io/badge/crystal-1.17.1+-blue.svg .. |GPLv3| image:: https://img.shields.io/badge/license-GPLv3-green.svg .. |Version| image:: https://img.shields.io/badge/version-0.8.0-orange.svg

.. _LICENSE: LICENSE