stacker v1.1.1
Stacker - A lightweight file-based CMDB
Stacker is Salt PillarStack in Crystal.
It is implemented using crinja which is Jinja2 in Crystal :)
Installation
Manual installation
Grab the latest binary from the releases page and run it with the sample configuration project.
If you use asdf you can also install Stacker with asdf-stacker.
Docker
docker pull nicoladmin/stacker:latest
Note : The Docker mode comes with a wrapper script to ease interactions with the container
curl -sSL -o stacker.sh https://raw.githubusercontent.com/jbox-web/stacker/master/stacker.sh
chmod +x stacker.sh
Usage : stacker.sh {start|stop|restart|status|kill|clean|fetch|info|logs}
The 10 seconds test in Web mode
First we need to start Stacker with sample data :
Manual installation
stacker server --config example/stacker.yml
Docker
./stacker.sh start
Then fetch pillars with Curl :
curl --no-progress-meter -X POST -H "Content-Type: application/json" -d @example/grains/server1.curl.json http://127.0.0.1:3000/server1.example.net | jq
You can also navigate to http://127.0.0.1:3000/server1.example.net to see the generated pillars.
The 10 seconds test in CLI mode
In this mode you don't need the webserver to be running :
Manual installation
stacker fetch server1.example.net --config example/stacker.yml --grains example/grains/server1.json --pillar example/ext_pillar/server1.json | jq
Docker
./stacker.sh fetch server1.example.net --grains grains/server1.json --pillar ext_pillar/server1.json | jq
Usage
Usage:
stacker [flags...] [arg...]
Stacker is Salt PillarStack in Crystal
Flags:
--help # Displays help for the current command.
--version # Displays the version of the current application.
Subcommands:
fetch # Fetch host pillars
info # Show Stacker information
server # Run Stacker webserver
Configuration
By default Stacker looks for it's configuration file in the current directory (stacker.yml
).
You can pass an alternative file by using --config
flag.
The configuration file is a YAML file looking like this :
---
doc_root: example/doc_root
entrypoint: server-pillars
log_file: ./log/stacker.log
stacks:
default:
- example/doc_root/server-pillars/stack1.cfg
dev:
- example/doc_root/server-pillars/stack1.cfg
- example/doc_root/pillars/stack1.cfg
prod:
- example/doc_root/server-pillars/stack1.cfg
- example/doc_root/pillars/stack1.cfg
- example/doc_root/server-pillars/stack2.cfg
server_host: 127.0.0.1
server_port: 3000
server_environment: development
Note : You can use relative or absolute file path.
Config | Description |
---|---|
doc_root |
the webserver document root (must be specified). Since pillar are also crinja templates, it means where are the template files? |
entrypoint |
the webserver entrypoint (must be specified). The directory in the doc_root where we shoud look for <minion_d>.yml file |
log_file |
the path to the log file |
stacks |
a hash of namespaced stack configuration files (must be specified) |
server_host |
ip address to bind to (default 127.0.0.1 ) |
server_port |
port to bind to (default 3000 ) |
server_environment |
development or production (default production ) |
Salt integration
To integrate Stacker with Salt you first need to add the stacker pillar module in Salt :
- Declare extension modules directory in Salt (
/etc/salt/master.conf
or/etc/salt/master.d/f_defaults.conf
)
extension_modules: /data/salt/modules
- Create
/data/salt/modules/pillar
directory and puts stacker module in it
mkdir -p /data/salt/modules/pillar
wget -O /data/salt/modules/pillar/stacker.py https://raw.githubusercontent.com/jbox-web/stacker/master/salt/stacker.py
- Declare the new ext_pillar module in Salt
ext_pillar:
- stacker: http://127.0.0.1:3000
You can also pass parameters to Stacker module :
ext_pillar:
- stacker:
host: http://127.0.0.1:3000
namespace: 'production'
log_level: 'debug'
- Restart Salt, you're done :)
Namespaces
Use namespaces by using optional n=
query parameter :
curl http://127.0.0.1:3000/server1.example.net?n=dev
curl http://127.0.0.1:3000/server1.example.net?n=prod
Or with --namespace
flag when using Stacker CLI.
The default namespace when query parameter or CLI flag is omited is default
.
Output format
Set output format by using optional f=
query parameter :
curl http://127.0.0.1:3000/server1.example.net?f=json
curl http://127.0.0.1:3000/server1.example.net?f=yaml
Or with --output-format
flag when using Stacker CLI.
The default output format when query parameter or CLI flag is omited is json
.
Only json
and yaml
are supported.
Logs
- log level
The log level is dynamic. No need to restart the web server :)
Set log level by using optional l=
query parameter :
curl http://127.0.0.1:3000/server1.example.net?l=debug
curl http://127.0.0.1:3000/server1.example.net?l=trace
Or with --log-level
flag when using Stacker CLI.
The default log level when query parameter or CLI flag is omited is info
.
Log levels other than debug
or trace
are meaningless.
trace
level is very verbose as it dumps data before and after merge operations. In this case you might need some filtering, see below...
debug level will render something like this :
2020-10-02T23:18:27.149678Z INFO - processor: Building stack for: server2.example.net (namespace: prod)
2020-10-02T23:18:27.149897Z DEBUG - renderer: Compiled: example/doc_root/server-pillars/stack1.cfg
2020-10-02T23:18:27.149927Z DEBUG - processor: Loading: example/doc_root/server-pillars/01-base.yml
2020-10-02T23:18:27.149936Z DEBUG - processor: Compiling: example/doc_root/server-pillars/01-base.yml
2020-10-02T23:18:27.150030Z DEBUG - renderer: Compiled: example/doc_root/server-pillars/01-base.yml
2020-10-02T23:18:27.150078Z DEBUG - processor: Merging: example/doc_root/server-pillars/01-base.yml
2020-10-02T23:18:27.150099Z DEBUG - processor: Loading: example/doc_root/server-pillars/server2.example.net.yml
2020-10-02T23:18:27.150109Z DEBUG - processor: Compiling: example/doc_root/server-pillars/server2.example.net.yml
2020-10-02T23:18:27.150190Z DEBUG - renderer: Compiled: example/doc_root/server-pillars/server2.example.net.yml
2020-10-02T23:18:27.150263Z DEBUG - processor: Merging: example/doc_root/server-pillars/server2.example.net.yml
2020-10-02T23:18:27.150705Z DEBUG - renderer: Compiled: example/doc_root/pillars/stack1.cfg
2020-10-02T23:18:27.150775Z DEBUG - processor: Loading: example/doc_root/pillars/01-base/01-base.yml
2020-10-02T23:18:27.150784Z DEBUG - processor: Compiling: example/doc_root/pillars/01-base/01-base.yml
2020-10-02T23:18:27.150873Z DEBUG - renderer: Compiled: example/doc_root/pillars/01-base/01-base.yml
2020-10-02T23:18:27.150932Z DEBUG - processor: Merging: example/doc_root/pillars/01-base/01-base.yml
2020-10-02T23:18:27.150942Z DEBUG - processor: Loading: example/doc_root/pillars/01-base/app1.yml
2020-10-02T23:18:27.150949Z DEBUG - processor: Compiling: example/doc_root/pillars/01-base/app1.yml
2020-10-02T23:18:27.151042Z DEBUG - renderer: Compiled: example/doc_root/pillars/01-base/app1.yml
2020-10-02T23:18:27.151091Z DEBUG - processor: Merging: example/doc_root/pillars/01-base/app1.yml
2020-10-02T23:18:27.151108Z DEBUG - processor: Loading: example/doc_root/pillars/01-base/zdump.yml
2020-10-02T23:18:27.151115Z DEBUG - processor: Compiling: example/doc_root/pillars/01-base/zdump.yml
2020-10-02T23:18:27.151208Z DEBUG - renderer: Compiled: example/doc_root/pillars/01-base/zdump.yml
2020-10-02T23:18:27.151238Z DEBUG - processor: Merging: example/doc_root/pillars/01-base/zdump.yml
2020-10-02T23:18:27.151267Z DEBUG - processor: Loading: example/doc_root/pillars/02-roles/common/locale.yml
2020-10-02T23:18:27.151286Z DEBUG - processor: Compiling: example/doc_root/pillars/02-roles/common/locale.yml
2020-10-02T23:18:27.151392Z DEBUG - renderer: Compiled: example/doc_root/pillars/02-roles/common/locale.yml
2020-10-02T23:18:27.151426Z DEBUG - processor: Merging: example/doc_root/pillars/02-roles/common/locale.yml
2020-10-02T23:18:27.151452Z DEBUG - processor: Loading: example/doc_root/pillars/02-roles/common/timezone.yml
2020-10-02T23:18:27.151462Z DEBUG - processor: Compiling: example/doc_root/pillars/02-roles/common/timezone.yml
2020-10-02T23:18:27.151572Z DEBUG - renderer: Compiled: example/doc_root/pillars/02-roles/common/timezone.yml
2020-10-02T23:18:27.151597Z DEBUG - processor: Merging: example/doc_root/pillars/02-roles/common/timezone.yml
2020-10-02T23:18:27.151619Z DEBUG - processor: Loading: example/doc_root/pillars/02-roles/common/users.yml
2020-10-02T23:18:27.151627Z DEBUG - processor: Compiling: example/doc_root/pillars/02-roles/common/users.yml
2020-10-02T23:18:27.151795Z DEBUG - renderer: Compiled: example/doc_root/pillars/02-roles/common/users.yml
2020-10-02T23:18:27.151858Z DEBUG - processor: Merging: example/doc_root/pillars/02-roles/common/users.yml
2020-10-02T23:18:27.151879Z DEBUG - processor: Loading: example/doc_root/pillars/02-roles/client/salt.yml
2020-10-02T23:18:27.151887Z DEBUG - processor: Compiling: example/doc_root/pillars/02-roles/client/salt.yml
2020-10-02T23:18:27.152009Z DEBUG - renderer: Compiled: example/doc_root/pillars/02-roles/client/salt.yml
2020-10-02T23:18:27.152039Z DEBUG - processor: Merging: example/doc_root/pillars/02-roles/client/salt.yml
2020-10-02T23:18:27.152060Z DEBUG - processor: Loading: example/doc_root/pillars/02-roles/server/openssh.yml
2020-10-02T23:18:27.152069Z DEBUG - processor: Compiling: example/doc_root/pillars/02-roles/server/openssh.yml
2020-10-02T23:18:27.152231Z DEBUG - renderer: Compiled: example/doc_root/pillars/02-roles/server/openssh.yml
2020-10-02T23:18:27.152331Z DEBUG - processor: Merging: example/doc_root/pillars/02-roles/server/openssh.yml
2020-10-02T23:18:27.152351Z DEBUG - processor: Loading: example/doc_root/pillars/02-roles/server/docker.yml
2020-10-02T23:18:27.152360Z DEBUG - processor: Compiling: example/doc_root/pillars/02-roles/server/docker.yml
2020-10-02T23:18:27.152478Z DEBUG - renderer: Compiled: example/doc_root/pillars/02-roles/server/docker.yml
2020-10-02T23:18:27.152528Z DEBUG - processor: Merging: example/doc_root/pillars/02-roles/server/docker.yml
2020-10-02T23:18:27.152547Z DEBUG - processor: Loading: example/doc_root/pillars/02-roles/server/nodejs.yml
2020-10-02T23:18:27.152554Z DEBUG - processor: Compiling: example/doc_root/pillars/02-roles/server/nodejs.yml
2020-10-02T23:18:27.152688Z DEBUG - renderer: Compiled: example/doc_root/pillars/02-roles/server/nodejs.yml
2020-10-02T23:18:27.152729Z DEBUG - processor: Merging: example/doc_root/pillars/02-roles/server/nodejs.yml
2020-10-02T23:18:27.152748Z DEBUG - processor: Loading: example/doc_root/pillars/02-roles/server/php.yml
2020-10-02T23:18:27.152756Z DEBUG - processor: Compiling: example/doc_root/pillars/02-roles/server/php.yml
2020-10-02T23:18:27.152931Z DEBUG - renderer: Compiled: example/doc_root/pillars/02-roles/server/php.yml
2020-10-02T23:18:27.153030Z DEBUG - processor: Merging: example/doc_root/pillars/02-roles/server/php.yml
2020-10-02T23:18:27.153050Z DEBUG - processor: Loading: example/doc_root/pillars/02-roles/server/redis.yml
2020-10-02T23:18:27.153059Z DEBUG - processor: Compiling: example/doc_root/pillars/02-roles/server/redis.yml
2020-10-02T23:18:27.153534Z DEBUG - renderer: Compiled: example/doc_root/pillars/02-roles/server/redis.yml
2020-10-02T23:18:27.153604Z DEBUG - processor: Merging: example/doc_root/pillars/02-roles/server/redis.yml
2020-10-02T23:18:27.153631Z DEBUG - processor: Loading: example/doc_root/pillars/02-roles/custom/php-app-server-common.yml
2020-10-02T23:18:27.153639Z DEBUG - processor: Compiling: example/doc_root/pillars/02-roles/custom/php-app-server-common.yml
2020-10-02T23:18:27.153759Z DEBUG - renderer: Compiled: example/doc_root/pillars/02-roles/custom/php-app-server-common.yml
2020-10-02T23:18:27.153835Z DEBUG - processor: Merging: example/doc_root/pillars/02-roles/custom/php-app-server-common.yml
2020-10-02T23:18:27.153856Z DEBUG - processor: Loading: example/doc_root/pillars/02-roles/custom/php-app-server-development.yml
2020-10-02T23:18:27.153864Z DEBUG - processor: Compiling: example/doc_root/pillars/02-roles/custom/php-app-server-development.yml
2020-10-02T23:18:27.154676Z DEBUG - renderer: Compiled: example/doc_root/pillars/02-roles/custom/php-app-server-development.yml
2020-10-02T23:18:27.154824Z DEBUG - processor: Merging: example/doc_root/pillars/02-roles/custom/php-app-server-development.yml
2020-10-02T23:18:27.154931Z DEBUG - renderer: Compiled: example/doc_root/server-pillars/stack2.cfg
2020-10-02T23:18:27.154970Z DEBUG - processor: Loading: example/doc_root/server-pillars/server2.example.net/clean.yml
2020-10-02T23:18:27.154978Z DEBUG - processor: Compiling: example/doc_root/server-pillars/server2.example.net/clean.yml
2020-10-02T23:18:27.155054Z DEBUG - renderer: Compiled: example/doc_root/server-pillars/server2.example.net/clean.yml
2020-10-02T23:18:27.155076Z DEBUG - processor: Merging: example/doc_root/server-pillars/server2.example.net/clean.yml
2020-10-02T23:18:27.155086Z DEBUG - processor: Loading: example/doc_root/server-pillars/server2.example.net/stacker.yml
2020-10-02T23:18:27.155090Z DEBUG - processor: Compiling: example/doc_root/server-pillars/server2.example.net/stacker.yml
2020-10-02T23:18:27.155162Z DEBUG - renderer: Compiled: example/doc_root/server-pillars/server2.example.net/stacker.yml
2020-10-02T23:18:27.155181Z DEBUG - processor: Merging: example/doc_root/server-pillars/server2.example.net/stacker.yml
2020-10-02T23:18:27.155191Z DEBUG - processor: Loading: example/doc_root/server-pillars/server2.example.net/users.yml
2020-10-02T23:18:27.155195Z DEBUG - processor: Compiling: example/doc_root/server-pillars/server2.example.net/users.yml
2020-10-02T23:18:27.155451Z DEBUG - renderer: Compiled: example/doc_root/server-pillars/server2.example.net/users.yml
2020-10-02T23:18:27.155475Z DEBUG - processor: Merging: example/doc_root/server-pillars/server2.example.net/users.yml
2020-10-02T23:18:27.155486Z DEBUG - processor: Loading: example/doc_root/server-pillars/server2.example.net/zdump.yml
2020-10-02T23:18:27.155490Z DEBUG - processor: Compiling: example/doc_root/server-pillars/server2.example.net/zdump.yml
2020-10-02T23:18:27.155896Z DEBUG - renderer: Compiled: example/doc_root/server-pillars/server2.example.net/zdump.yml
2020-10-02T23:18:27.156648Z DEBUG - processor: Merging: example/doc_root/server-pillars/server2.example.net/zdump.yml
2020-10-02T23:18:27.156664Z INFO - processor: End of stack build for: server2.example.net (namespace: prod)
- filter logs by template path
Set the file path to debug by using optional p=
query parameter :
curl http://127.0.0.1:3000/server1.example.net?l=trace&p=doc_root/pillars/02-roles/server/openssh.yml
Or with --path
flag when using Stacker CLI.
- filter logs by steps
Set the step to debug by using optional s=
query parameter :
curl http://127.0.0.1:3000/server1.example.net?l=trace&p=doc_root/pillars/02-roles/server/openssh.yml&s=compile,yaml-load
Or with --step
flag when using Stacker CLI.
Valid options for the step
param are : compile
| yaml-load
| before-merge
| after-merge
| final
Scaling
The Stacker's web design leads to great possibilities :
- You can move Stacker and the pillar rendering process out of Salt server :
ext_pillar:
- stacker: http://stacker.example.corp:3000
- You can run multiple instances of Stacker and call them sequentially :
ext_pillar:
- stacker: http://127.0.0.1:3000?n=foo
- stacker: http://127.0.0.1:4000?n=bar
- stacker: http://127.0.0.1:5000?n=baz
With each instance having it's own stack configuration :)
Deployment
You can use the provided systemd unit to manage the Stacker daemon.
Merging strategies
Stacker implements merging strategies like PillarStack so you can use them in Stacker too :)
It works the same way and it's tested.
Template syntax
The template syntax is almost the same than Jinja2.
Like PillarStack you have access to these variables :
stack
pillar
minion_id
grains
(instead of__grains__
)
Stacker adds a bunch of filters and functions :
Filters :
- json filter (dump json without character escaping) (like the Salt one)
- traverse filter (like the Salt one)
- unique filter (like the Salt one)
Functions :
- log function (like the Salt one)
- dump function (it dumps objects to YAML in log file)
- array_push function
- merge_dict function
You can see examples of code in the documentation.
The following filters/tests/functions/tags/operators are supported :
filters:
abs()
append(string=)
attr(name=)
batch(linecount=2, fill_with=none)
capitalize()
center(width=80)
date(format=)
default(default_value='', boolean=false)
dictsort(case_sensitive=false, by='key')
escape()
filesizeformat(binary=false)
first()
float(default=0.0)
forceescape()
format()
groupby(attribute=)
indent(width=4, indentfirst=false)
int(default=0, base=10)
join(separator='', attribute=none)
json(indent=none)
last()
length()
list()
lower()
map()
pprint(verbose=false)
prepend(string=)
random()
reject()
rejectattr()
replace(old=, new=, count=none)
reverse()
round(precision=0, method='common', base=10)
safe()
select()
selectattr()
slice(slices=2, fill_with=none)
sort(reverse=false, case_sensitive=false, attribute=none)
string()
striptags()
sum(attribute=none, start=0)
title()
tojson(indent=none)
traverse(attribute=none, default=none)
trim()
truncate(length=255, killwords=false, end='...', leeway=none)
unique()
upper()
urlencode()
urlize(trim_url_limit=none, nofollow=false, target=none, rel=none)
wordcount()
wordwrap(width=79, break_long_words=true, wrapstring=none)
xmlattr(autoescape=true)
tests:
callable()
defined()
divisibleby(num=)
equalto(other=)
escaped()
even()
greaterthan(other=0)
in(seq=[])
iterable()
lessthan(other=0)
lower()
mapping()
nil()
none()
number()
odd()
sameas(other=)
sequence()
string()
undefined()
upper()
functions:
array_push(array=[], item=none)
cycler()
debug()
dict()
dump(object=none)
joiner(sep=', ')
log(object=none)
merge_dict(hash=none, other=none)
range(start=0, stop=0, step=1)
super()
tags:
autoescape~endautoescape
block~endblock
call~endcall
do
elif
else
endautoescape
endblock
endcall
endfilter
endfor
endif
endmacro
endraw
endset
endwith
extends
filter~endfilter
for~endfor
from
if~endif
import
include
macro~endmacro
raw~endraw
set~endset
with~endwith
operators:
operator[!=]
operator[%]
operator[*]
operator[**]
operator[+]
operator[-]
operator[/]
operator[//]
operator[<]
operator[<=]
operator[==]
operator[>]
operator[>=]
operator[and]
operator[not]
operator[or]
operator[~]
Development
To compile Stacker you will need Crystal compiler.
You can easily setup your development environment with asdf :
git clone https://github.com/jbox-web/stacker
make setup # Install asdf Crystal plugin and install Crystal compiler
make deps # Install Stacker dependencies
make build # Build Stacker in development mode
make relase # Build Stacker in production mode
Extend Stacker
If you need to add filters (or functions) just drop a new class with a few lines of Crystal code in /src/runtime and recompile Stacker with make build
(dev mode) or make release
(release mode).
Your custom filters (or functions) should be available in Jinja templates. To be sure run stacker info
and check the Crinja environment info.
Then feel free to submit a PR if you think it will be useful for people.
Roadmap
- implement Select Stacker config through grains|pillar|opts matching
- add documentation about "Stacker - Sample Project"
stacker
- 10
- 0
- 0
- 0
- 6
- 11 months ago
- May 5, 2020
MIT License
Wed, 06 Nov 2024 23:27:43 GMT