Totem

Crystal configuration with spirit. Inspired from Go's viper. Totem Icon by lastspark from Noun Project.

Configuration file formats is always the problem, you want to focus on building awesome things. Totem is here to help with that.

Totem has following features:

• Reading from JSON, YAML, dotenv formats config files or raw string.
• Reading from environment variables.
• Reading from remote key-value store systems(redis/etcd).
• Provide a mechanism to set default values for your different configuration options.
• Provide an alias system to easily rename parameters without breaking existing code.
• Write configuration to file with JSON, YAML formats.
• Convert config to struct with builder.

And we keep it minimize and require what you want with adapter and remote provider! No more dependenices what you do not need. Only JSON and YAML adapters were auto requires.

Uses the following precedence order. Each item takes precedence over the item below it:

• alias
• override, explicit call to set
• env
• config
• kvstores
• default

Totem configuration keys are case insensitive.

• Installation
• Quick Start
  • Operating configuration
  • Loading configuration
    • From raw string
    • From file
• Usage
  • Load configuration with multiple paths
  • Set Alias and using alias
  • Working with nested key
  • Working with environment variables
  • Working with remote providers
    • Use redis
    • Use etcd
  • Iterating configuration
  • Serialization
  • Storing configuration to file
• Advanced Usage
  • Use config builder
  • Write a config adapter
  • Write a remote provider
• Q & A
  • How to debug?
• Help and Discussion
• Donate
• How to Contribute
• You may also like
• License

Installation

Add this to your application's shard.yml:

dependencies:
  totem:
    github: icyleaf/totem

Quick Start

require "totem"

Operating configuration

totem = Totem.new
totem.set_default("name", "foo")
totem.set_defaults({
  "age"    => 18,
  "gender" => "male",
  "hobbies" => [
    "skateboarding",
    "snowboarding",
    "go"
  ]
})

totem.get("name").as_s # => "foo"
totem.get("age").as_i # => 18
totem.set("name", "bar")
totem.alias(alias_key: "key", key: "name")
totem.get("name").as_s # => "bar"
totem.get("key").as_s # => "bar"

Loading configuration

Support JSON, YAML and dotenv data from raw string and file.

From raw string

Load yaml string

raw = <<-EOF
Hacker: true
name: steve
hobbies:
- skateboarding
- snowboarding
- go
clothing:
  jacket: leather
  trousers: denim
  pants:
    size: large
age: 35
eyes : brown
EOF

totem = Totem.from_yaml raw
totem.get("Hacker").as_bool                           # => true
totem.get("age").as_i                                 # => 35
totem.get("clothing").as_h["pants"].as_h["size"].as_s # => "large"

Load json string

raw = <<-EOF
{
  "id": "0001",
  "type": "donut",
  "name": "Cake",
  "ppu": 0.55,
  "batters": {
    "batter": [
      {
        "type": "Regular"
      },
      {
        "type": "Chocolate"
      },
      {
        "type": "Blueberry"
      },
      {
        "type": "Devil's Food"
      }
    ]
  }
}
EOF

totem = Totem.from_json raw
totem.get("name")                                         # => "Cake"
totem.get("ppu")                                          # => 0.55
totem.get("batters").as_h["batter"].as_a[0].as_h["type"]  # => "Regular"

Load dotenv string

Add poncho to shards.yml and require the adapter.

require "totem"
require "totem/config_types/env"    # Make sure you require

raw = <<-EOF
# COMMENTS=work
STR='foo'
STR_WITH_COMMENTS=bar         # str with comment
STR_WITH_HASH_SYMBOL="abc#123"#stick comment
INT=33
EOF

totem = Totem.from_env raw
totem.get("str")                    # => "foo"
totem.get("str_with_comments")      # => bar
totem.get("str_with_hash_symbol")   # => "abc#123"
totem.get("int")                    # => "33"

From file

Add poncho to shards.yml and require the adapter if you need load dotenv file.

# Load yaml file from file with path
totem = Totem.from_file "./spec/fixtures/config.yaml"

# Load json file from file with multi-paths
totem = Totem.from_file "config.yaml", ["/etc", ".", "./spec/fixtures"]

# Load dotenv file
totem = Totem.from_file "config.env"

Usage

Load configuration with multiple paths

Totem can search multiple paths, but currently a single Totem instance only supports a single configuration file.

totem = Totem.new("config", "/etc/totem/")  # => New a instance with name and path of config file
totem.config_paths << "~/.totem"            # => path to look for the config file in
totem.config_paths << "./config"            # => optionally look for config in the working directory
begin
  totem.load!                               # => Find and read the config file (order by yaml/yml/json/env)
rescue e
  puts "Fatal error config file: #{e.message}"
end

Set Alias and using alias

Aliases permit a single value to be referenced by multiple keys

totem.alias("nickname", "Name")

totem.set("name", "foo")
totem.set("nickname", "bar")

totem.get("name")       # => "foo"
totem.get("nickname")   # => "foo"

Working with nested key

All accessor methods accept nested key:

totem.set_default("profile.user.name", "foo")
totem.set("profile.user.age", 13)
totem.alias("username", "profile.user.name")
totem.bind_env("profile.user.nickname", "PROFILE_USER_NICKNAME")
totem.get("profile.user.age")

Working with environment variables

Totem has full support for environment variables, example:

ENV["ID"] = "123"
ENV["FOOD"] = "Pinapple"
ENV["NAME"] = "Polly"

totem = Totem.new

totem.bind_env("ID")
totem.get("id").as_i        # => 123

totem.bind_env("f", "FOOD")
totem.get("f").as_s         # => "Pinapple"

totem.automative_env
totem.get("name").as_s      # => "Polly"

Working with environment prefix:

totem.automative_env(prefix: "totem")
# Same as
# totem.env_prefix = "totem"
# totem.automative_env = true

totem.get("id").as_i    # => 123
totem.get("food").as_s  # => "Pinapple"
totem.get("name").as_s  # => "Polly"

Working with remote providers

Totem retrieve configuration from Key-Value store, which means that you can get your configuration values on the air. Avaliable providers is redis and etcd.

Use redis

It dependency crystal-redis shard. Install it before use.

require "totem"
require "totem/remote_providers/redis"

totem = Totem.new
totem.add_remote(provider: "redis", endpoint: "redis://localhost:6379/0")

totem.get("user:name")      # => "foo"
totem.get("user:id").as_i   # => 123

You can also get raw data from one key with path:

totem.config_type = "json"  # There is no file extension in a stream data, supported extensions are all registed config types in Totem.
totem.add_remote(provider: "redis", endpoint: "redis://localhost:6379/0", path: "config:development")

totem.get("user:name")      # => "foo"
totem.get("user:id").as_i   # => 123

Use etcd

It dependency etcd-crystal shard and ONLY works etcd v2 API. Install it before use.

require "totem"
require "totem/remote_providers/etcd"

totem = Totem.new
totem.add_remote(provider: "etcd", endpoint: "http://localhost:2379")

totem.get("user:name")      # => "foo"
totem.get("user:id").as_i   # => 123

You can also get raw data from one key with path:

totem.config_type = "yaml"  # There is no file extension in a stream data, supported extensions are all registed config types in Totem.
totem.add_remote(provider: "etcd", endpoint: "http://localhost:2379", path: "/config/development.yaml")

totem.get("user:name")      # => "foo"
totem.get("user:id").as_i   # => 123

Iterating configuration

Iterate in Totem is very easy, you can get #keys, #flat_keys, #settings (a.k.a #to_h) even iterating it directly with #each:

totem.settings    # => {"id" => 123, "user" => {"name" => "foobar", "age" => 20}}
totem.keys        # => ["id", "user"]
totem.flat_keys   # => ["id", "user.name", "user.age"]

totem.each do |key, value|
  # do something
end

Serialization

Serialize configuration to Struct, at current stage you can pass a JSON::Serializable/YAML::Serializable struct to mapping.

struct Profile
  include JSON::Serializable

  property name : String
  property hobbies : Array(String)
  property age : Int32
  property eyes : String
end

totem = Totem.from_file "spec/fixtures/config.yaml"
profile = totem.mapping(Profile)
profile.name      # => "steve"
profile.age       # => 35
profile.eyes      # => "brown"
profile.hobbies   # => ["skateboarding", "snowboarding", "go"]

Serialize configuration with part of key:

struct Clothes
  include JSON::Serializable

  property jacket : String
  property trousers : String
  property pants : Hash(String, String)
end

totem = Totem.from_file "spec/fixtures/config.yaml"
clothes = profile.mapping(Clothes, "clothing")
# => Clothes(@jacket="leather", @pants={"size" => "large"}, @trousers="denim")

Storing configuration to file

Simple to use #store! method.

raw = <<-EOF
Hacker: true
name: steve
hobbies:
- skateboarding
- snowboarding
- go
clothing:
  jacket: leather
  trousers: denim
  pants:
    size: large
age: 35
eyes : brown
EOF

totem = Totem.from_yaml raw
totem.set("nickname", "Freda")
totem.set("eyes", "blue")
totem.store!("profile.json")

Advanced Usage

Use config builder

You can generate a configuration with Totem builder with any Object.

struct Configuration
  include Totem::ConfigBuilder

  build do
    config_type "json"
    config_paths ["/etc/totem", "~/.config/totem", "config/"]
  end
end

config = Configuration.configure do |c|
  c.set_default "name", "foobar"
end

config["name"] # => "foobar"

The builder also could mapping config to struct.

struct Profile
  include Totem::ConfigBuilder

  property name : String
  property hobbies : Array(String)
  property age : Int32
  property eyes : String

  build do
    config_type "yaml"
    config_paths ["/etc/totem", "~/.config/totem", "config/"]
  end
end

profile = Profile.configure
profile.name          # => "steve"
profile["nested.key"] # => "foo"

Write a config adapter

Creating the custom adapter by integration Totem::ConfigTypes::Adapter abstract class. Here has two methods must be implement: read and write. For example, let us write a INI adapter:

require "ini"

class INIAdapter < Totem::ConfigTypes::Adapter
  def read(raw)
    INI.parse(raw)
  end

  def write(io, config)
    config.settings.each do |key, items|
      next unless data = items.as_h?
      io << "[" << key << "]\n"
      data.each do |name, value|
        io << name << " = " << value << "\n"
      end
    end
  end
end

# Do not forget register it
Totem::ConfigTypes.register_adapter("ini", INIAdapter.new)
# Also you can set aliases
Totem::ConfigTypes.register_alias("cnf", "ini")

More examples to review built-in adapters.

Write a remote provider

Creating the custom remote provider by integration Totem::RemoteProviders::Adapter abstract class. Here has two methods must be implement: read and get, please reivew the built-in remote providers.

Q & A

How to debug?

You can use Crystal built-in #pp or #pp! method to prints a series of instance variables:

#<Totem::Config
 @config_paths=["/etc/totem", "~/.totem"],
 @config_name="config",
 @config_type="json",
 @key_delimiter=".",
 @automatic_env=false,
 @env_prefix=nil,
 @aliases={"user" => "profile.user.name"},
 @overrides={"profile" => {"user" => {"gender" => "male"}}, "name" => "foo"},
 @config={"profile" => {"user" => {"gender" => "unkown"}}, "name" => "bar"}},
 @env={"name" => "TOTEM_NAME"},
 @defaults={"name" => "alana"}>

Help and Discussion

You can browse the API documents:

https://icyleaf.github.io/totem/

You can browse the Changelog:

https://github.com/icyleaf/totem/blob/master/CHANGELOG.md

If you have found a bug, please create a issue here:

https://github.com/icyleaf/totem/issues/new

Donate

Totem is a open source, collaboratively funded project. If you run a business and are using Totem in a revenue-generating product, it would make business sense to sponsor Totem development. Individual users are also welcome to make a one time donation if Totem has helped you in your work or personal projects.

You can donate via Paypal.

How to Contribute

Your contributions are always welcome! Please submit a pull request or create an issue to add a new question, bug or feature to the list.

All Contributors are on the wall.

You may also like

• halite - HTTP Requests Client with a chainable REST API, built-in sessions and middlewares.
• markd - Yet another markdown parser built for speed, Compliant to CommonMark specification.
• poncho - A .env parser/loader improved for performance.
• popcorn - Easy and Safe casting from one type to another.
• fast-crystal - 💨 Writing Fast Crystal 😍 -- Collect Common Crystal idioms.

License

MIT License © icyleaf