This library generates a systemd unit file to manage an Elixir release. It supports releases generated by Elixir 1.9+ mix release or Distillery.
At its heart, it's a mix task which reads information about the project from
mix.exs and config/config.exs then generates systemd unit files using Eex
templates. The goal is that the project defaults will generate a good systemd
unit file, and standard options support more specialized use cases.
It uses standard systemd functions and conventions to make your app a more
"native" OS citizen and takes advantage of systemd features to improve
security and reliability. While it can be used standalone, more advanced use
cases use scripts from e.g., mix_deploy.
This complete example app puts the pieces together.
Add the package to your list of dependencies in mix.exs:
def deps do
[
{:mix_systemd, "~> 0.8"},
]
endThe library tries to choose reasonable defaults, so you may not need to
configure anything. It reads the app name from mix.exs and calculates default
values for its configuration parameters. For example, if your app is named
foo_bar, it will create a service named foo-bar, deployed to
/srv/foo-bar, running under the user foo-bar.
You can override these parameters using settings in config/config.exs, e.g.:
config :mix_systemd,
app_user: "app",
app_group: "app",
base_dir: "/opt",
env_vars: [
"PORT=8080",
]There are four different kinds of things that we may want to configure:
-
Static information about application layout, e.g., file paths. This is the same for all machines in an environment, e.g., staging or prod.
-
Information specific to the environment, e.g., the hostname of the db server.
-
Secrets such as db passwords, API keys, or encryption keys.
-
Dynamic information such as the IP address of the server or other machines in the cluster.
Elixir has a couple of mechanisms for storing configuration. When you compile
the release, it converts Elixir-format config files like config/config.exs
into an initial application environment that is read by Application.get_env/3.
That's good for simple, relatively static apps. It's not ideal to store
passwords in the release file, though.
Elixir 1.9+ releases support dynamic configuration at runtime. You can configure
via the Elixir file config/runtime.exs which is loaded when the VM boots or
use the shell script rel/env.sh.eex to set environment vars.
With these you can theoretically do anything. In practice, however, it can be
more convenient and secure to process the config outside of the app. That's
where mix_systemd and mix_deploy come in.
The simplest thing is to set environment variables. Add individual vars to
env_vars, and they will be set in the systemd unit file. Add files to
env_files and systemd will load them before starting your app.
Your application then calls System.get_env/1 in config/runtime.exs or
application startup. Note that these environment vars are read at runtime,
not when building your app.
env_vars: [
# Set a variable, good for things that are not sensitive and don't change
"PORT=8080",
],
dirs: [
# create /etc/foo
:configuration,
],
env_files: [
# Read environment vars from the file /etc/foo/environment
["-", :configuration_dir, "/environment"],
]/etc/foo/environment looks like:
DATABASE_URL="ecto://foo_prod:[email protected]/foo_prod"
SECRET_KEY_BASE="EOdJB1T39E5Cdeebyc8naNrOO4HBoyfdzkDy2I8Cxiq4mLvIQ/0tK12AK1ahrV4y"
HOST="www.example.com"
ASSETS_HOST="assets.example.com"
RELEASE_COOKIE="LmCMGNz04yEJ4MQc6jt3cS7QjAppYOw_bQa7NE5hPZJGqL3Yry1jUg=="
config/runtime.exs does something like the following (the default files are good):
config :foo, Foo.Repo,
url: System.get_env("DATABASE_URL")
config :foo, FooWeb.Endpoint,
http: [:inet6, port: System.get_env("PORT") || 4000],
url: [host: System.get_env("HOST"), port: 443],
static_url: [host: System.get_env("ASSETS_HOST"), port: 443],
secret_key_base: System.get_env("SECRET_KEY_BASE"),
cache_static_manifest: "priv/static/cache_manifest.json"The question is how to get the environment files onto the server. For simple server deployments, we can copy the config to the server when doing the initial setup.
In cloud environments, we may run from a read-only image, e.g., an Amazon AMI, which gets configured at start up based on the environment by copying the config from an S3 bucket, e.g.:
umask 077
aws s3 sync --exact-timestamps --no-progress "s3://${CONFIG_BUCKET}/" "/etc/foo/"
chown -R "$DEPLOY_USER:$APP_GROUP" /etc/foo
find /etc/foo -type f -exec chmod 640 {} \;
find /etc/foo -type d -exec chmod 750 {} \;The following example runs the script /srv/foo/bin/deploy-sync-config-s3 from
mix_deploy. It uses an environment file in /srv/foo/etc/environment
to bootstrap the sync, e.g., setting the S3 bucket name. That file
is placed there by CodeDeploy at deploy time.
config :mix_systemd,
exec_start_pre: [
# Run before starting the app
# The `!` means the script is run as root, not as the app user
["!", :deploy_dir, "/bin/deploy-sync-config-s3"]
],
dirs: [
:configuration, # /etc/foo, app configuration, e.g. db passwords
:runtime, # /run/foo, temp files which may be deleted between runs
],
env_files: [
["-", :deploy_dir, "/etc/environment"], # /srv/foo/etc/environment
]
env_vars: [
# Tell release to use /run/foo for temp files
["RELEASE_TMP=", :runtime_dir],
]At a certain point, making everything into an environment var becomes annoying. It's verbose and vars are simple strings, so you have to encode values safely and convert them back to lists, integers or atoms.
Config providers let you load files in standard formats like TOML.
[foo."Foo.Repo"]
url = "ecto://foo_prod:[email protected]/foo_prod"
pool_size = 15
[foo."FooWeb.Endpoint"]
secret_key_base = "EOdJB1T39E5Cdeebyc8naNrOO4HBoyfdzkDy2I8Cxiq4mLvIQ/0tK12AK1ahrV4y"The app reads these config files on startup and merges them into the app config.
defp releases do
[
prod: [
include_executables_for: [:unix],
config_providers: [
{TomlConfigProvider, path: "/etc/foo/config.toml"}
],
steps: [:assemble, :tar]
]
]
endThe startup scripts read the initial application environment compiled into the
release, parse the config file, merge the values, write it to a temp file, then
start the VM. Because of that, they need a writable directory. That is
configured using the RELEASE_TMP environment var, normally set to the app's
runtime_dir.
dirs: [
:configuration,
:runtime,
],
env_vars: [
["RELEASE_TMP=", :runtime_dir],
],You can also store config params in an external configuration system and read them at runtime. An example is AWS Systems Manager Parameter Store.
Set a parameter using the AWS CLI:
aws ssm put-parameter --name '/foo/prod/db/password' --type ‘SecureString’ --value 'Sekrit!"While it's possible to read params in config/runtime.exs, it's tedious.
Better is to grab all of them at once and write them to a file, then read it in
with a Config Provider like aws_ssm_provider.
aws --region us-east-1 ssm get-parameters-by-path --path "/foo/prod/" --recursive --with-decryption --query "Parameters[]" > /etc/foo/ssm.jsondefp releases do
[
prod: [
include_executables_for: [:unix],
config_providers: [
{AwsSsmProvider, path: "/etc/foo/ssm.json"}
],
steps: [:assemble, :tar]
]
]
endYou can write code to do things like query the system for the primary IP
address, but cloud-init
already does it.
You just have to read the JSON file.
The most common use for this is setting the VM node name. In env.sh:
CLOUD_NAME=$(jq -r '.v1.cloud_name' < /run/cloud-init/instance-data.json)
if [ "$CLOUD_NAME" = "digitalocean" ]; then
IP_ADDR=$(jq -r '.ds.meta_data.interfaces.public[0].anchor_ipv4.ip_address' < /run/cloud-init/instance-data.json)
DEFAULT_IPV4="$IP_ADDR"
elif [ "$CLOUD_NAME" = "aws" ]; then
IP_ADDR=$(jq -r '.ds.meta_data."local-ipv4"' < /run/cloud-init/instance-data.json)
# IP_ADDR=$(jq -r '.ds.meta_data."public-ipv4"' < /run/cloud-init/instance-data.json)
AWS_REGION=$(jq -r '.v1.region' < /run/cloud-init/instance-data.json)
fi
RELEASE_DISTRIBUTION="name"
RELEASE_NODE="${RELEASE_NAME}@${IP_ADDR}"An important security principle is "least privilege". If an attacker manages to compromise the app, then they can do whatever it has permissions to do, not just what you expect. Because of that, I prefer that the account that the app runs under cannot write files, and having a writable config file that is also executed is the worst case scenario.
First, use the systemd.init task to copy the default template files from mix_systemd
to the rel/templates/systemd directory in your project. Copying them allows you to make local customizations
if necessary.
mix systemd.initNext, generate output files in the build directory under
_build/#{mix_env}/systemd/lib/systemd/system.
MIX_ENV=prod mix systemd.generateThe following sections describe common configuration options.
See lib/mix/tasks/systemd.ex for the details of more obscure options.
If you need to make changes not supported by the config options,
then you can check the templates in rel/templates/systemd
into source control and make your own changes. Contributions are welcome!
Configuration is done by setting options in config/config.exs, e.g.,
config :mix_systemd,
app_user: "app",
app_group: "app",
base_dir: "/opt",
env_vars: [
"PORT=8080",
]-
app_name: Elixir application name, an atom, from theapporapp_namefield in themix.exsproject. For umbrella apps, setapp_name. -
module_name: Elixir camel case module name version ofapp_name, e.g.,FooBar. -
release_name: Name of release, defaultapp_name. -
ext_name: External name, used for files and directories, defaultapp_namewith underscores converted to "-", e.g.,foo-bar. -
service_name: Name of the systemd service, defaultext_name. -
release_system::mix | :distillery, default:mix. Identifies the system used to generate the releases, Mix or Distillery.
-
app_user: OS user account that the app runs under, defaultext_name. -
app_group: OS group account, defaultext_name.
-
base_dir: Base directory for app files on target, default/srv. -
deploy_dir: Directory for app files on target, default#{base_dir}/#{ext_name}.
We use the
standard app directories,
for modern Linux systems. App files are under /srv, configuration under
/etc, transient files under /run, data under /var/lib.
Directories are named based on the app name, e.g., /etc/#{ext_name}.
The dirs variable specifies which directories the app uses.
By default, it doesn't create any directories. To enable directories, configure the dirs
param, e.g.:
dirs: [
:runtime, # Runtime files which may be deleted between runs, /run/#{ext_name}
:configuration, # Configuration, e.g., db passwords, /etc/#{ext_name}
:state, # Data or state persisted between runs, /var/lib/#{ext_name}
:cache, # Cache files which can be deleted, /var/cache/#{ext_name}
:logs, # External log files, not via journald, /var/log/#{ext_name}
:tmp, # Temp files, /var/tmp/#{ext_name}
],Recent versions of systemd (since 235) will create these directories at
start time based on the settings in the unit file. With earlier systemd
versions, you need to create them beforehand using installation scripts, e.g.,
mix_deploy.
For security, we set permissions to 750, more restrictive than the systemd
defaults of 755. You can configure permissions with variables like
configuration_directory_mode. See the defaults in
lib/mix/tasks/systemd.ex.
systemd_version: Sets the systemd version on the target system, default 235.
This determines which systemd features the library will enable. If you are
targeting an older OS release, you may need to change it. Here are the systemd
versions in common OS releases:
- CentOS 7: 219
- Ubuntu 16.04: 229
- Ubuntu 18.04: 237
The library uses a directory structure under deploy_dir which supports
unpacking multiple releases at the same time, similar to Capistrano.
scripts_dir: deployment scripts which, e.g., start and stop the unit, defaultbin, e.g.,/srv/foo/bin.current_dir: where the current Erlang release is unpacked or referenced by symlink, defaultcurrent, e.g.,/srv/foo/current.releases_dir: where versioned releases are unpacked, defaultreleases, e.g.,/srv/foo/releases.flags_dir: dir for flag files to trigger restart, e.g., whenrestart_methodis:systemd_flag, defaultflags, e.g.,/srv/foo/flags.
When using multiple releases and symlinks, the deployment process works as follows:
-
Create a new directory for the release with a timestamp like
/srv/foo/releases/20181114T072116. -
Upload the new release tarball to the server and unpack it to the
releasesdir. -
Make a symlink from
/srv/#{ext_name}/currentto the new release dir. -
Restart the app.
If you are only keeping a single version, then deploy it to the directory
/srv/#{ext_name}/current.
expand_keys lists variables that support variable expansion. By default it includes:
expand_keys: [
:env_files,
:env_vars,
:runtime_environment_service_script,
:exec_start,
:exec_stop,
:exec_start_pre,
:exec_start_wrap,
:read_write_paths,
:read_only_paths,
:inaccessible_paths,
]These variables are then availble when configuring the app. You can specify
values as a list of terms, and it will look up atoms as keys in the config.
This lets you reference, e.g., the deploy dir or configuration dir without
having to specify the full path, e.g., ["!", :deploy_dir, "/bin/myscript"] gets
converted to "!/srv/foo/bin/myscript".
The library sets OS environment vars in the unit file, allowing them to used by
the app or scripts like env.sh:
MIX_ENV:mix_env, defaultMix.env()LANG:env_lang, defaulten_US.utf8RUNTIME_DIR:runtime_dir, if:runtimeindirsCONFIGURATION_DIR:configuration_dir, if:configurationindirsLOGS_DIR:logs_dir, if:logsindirsCACHE_DIR:cache_dir, if:cacheindirsSTATE_DIR:state_dir, if:stateindirsTMP_DIR:tmp_dir, if:tmpindirs
You can set additional vars using env_vars, e.g.:
env_vars: [
"PORT=8080",
]You can also reference expand_keys variables, e.g.:
env_vars: [
["RELEASE_TMP=", :runtime_dir],
]You can read environment vars from files with env_files, e.g.:
env_files: [
["-", :deploy_dir, "/etc/environment"], # /srv/foo/etc/environment
["-", :configuration_dir, "environment"], # /etc/foo/environment
["-", :runtime_dir, "environment"], # /run/foo/environment
],The "-" at the beginning makes the file optional; the system will start without it. Later values override earlier values, so you can set defaults which get overridden in the local or runtime environment.
The release scripts may need to write temp files and log files, e.g., when
generating the application config files. By default, they do this under
the release dir, e.g., /srv/foo/current/tmp.
For security, it's better to deploy the app using a different user account from the one that the app runs under, with the source files read only. This makes it harder for an attacker to make changes to the source and then have the app run them.
In that case, we need to set an environment var which tells the release
startup scripts where they can write files. For Mix releases, that is
RELEASE_TMP, and for Distillery it is RELEASE_MUTABLE_DIR, e.g.:
env_vars: [
["RELEASE_TMP=", :runtime_dir],
]By default systemd will delete the runtime directory when restarting the app.
This can be annoying when debugging startup issues. You can set
runtime_directory_preserve to restart or yes to prevent this. See
RuntimeDirectoryPreserve.
The following variables set systemd variables:
service_type::simple | :exec | :notify | :forking.systemdType, default:simple.
Modern applications don't fork, they run in the foreground and rely on the
supervisor to manage them as a daemon. This is done by setting service_type
to :simple or :exec. Note that in simple mode, systemd doesn't actually
check if the app started successfully, it just continues starting other units.
If something depends on your app being up, :exec may be better.
Set service_type to :forking, and the library sets pid_file to
#{runtime_directory}/#{app_name}.pid and sets the PIDFILE OS env var to tell
the boot scripts where it is.
The Erlang VM runs pretty well in foreground mode, but traditionally runs as
as a standard Unix-style daemon, so forking might be better. systemd
expects foregrounded apps to die when their pipe closes. See
https://elixirforum.com/t/systemd-cant-shutdown-my-foreground-app-cleanly/14581/2
restart_method::systemctl | :systemd_flag | :touch, default:systemctl
Set this to :systemd_flag, and the library will generate an additional
unit file which watches for changes to a flag file and restarts the
main unit. This allows updates to be pushed to the target machine by an
unprivileged user account which does not have permissions to restart
processes. Touch the file #{flags_dir}/restart.flag, and systemd will
restart the unit.
-
working_dir: Current working dir for app. systemd WorkingDirectory, defaultcurrent_dir. -
limit_nofile: Limit on open files, systemd LimitNOFILE, default 65535 (obsolete, set tonilto use the currentsystemddefault of 524288). -
umask: Process umask, systemd UMask, default "0027". Note that this is octal, so it needs to be a string. -
restart_sec: Time in seconds to wait between restarts, systemd RestartSec, default 100ms. -
syslog_identifier: Logging name, systemd SyslogIdentifier, defaultservice_name -
limit_core: Limit on core size, systemd LimitCORE, defaultnil. Set toinfinityor a numeric value to enable core files for the app.
By default, the library generates commands which run release commands to start and stop the app:
ExecStart=<%= "#{exec_start_wrap}#{current_dir}/bin/#{release_name} #{start_command}" %>
ExecStop=<%= "#{current_dir}/bin/#{release_name} stop" %>
You can override these by setting exec_start and exec_stop explicitly, e.g.,
exec_start: [:deploy_dir, "bin/start-migrate"]This would run the script /srv/foo/bin/start-migrate to run database migrations before starting the app, e.g.:
#!/bin/sh
set -e
CURDIR=$PWD
BINDIR=$(dirname -- "$0")
cd "$BINDIR"; BINDIR=$PWD; cd "$CURDIR"
"$BINDIR/prod" eval 'Foo.Release.create_repos()'
"$BINDIR/prod" eval 'Foo.Release.migrate()'
PHX_SERVER=true exec "$BINDIR/prod" startScripts specified in exec_start_pre (systemd
ExecStartPre])
run before the main ExecStart script runs.
For example, to run database migrations:
exec_start_pre: [
[:current_dir, "/bin/", :release_name, " eval ", :module_name, ".Release.create_repos()"],
[:current_dir, "/bin/", :release_name, " eval ", :module_name, ".Release.migrate()"]
],
```
Another example is copying config files from an S3 bucket into `/etc/foo` to configure the app
using the `deploy-sync-config-s3` script from `mix_deploy`:
```elixir
exec_start_pre: [
# Run before starting the app
# `!` means the script is run as root, not as the app user
["!", :deploy_dir, "/bin/deploy-sync-config-s3"]
]
```
By default, scripts run as the same user and group as the main script.
Putting `!` in front makes the script run with
[elevated privileges](https://www.freedesktop.org/software/systemd/man/systemd.service.html#ExecStart=),
allowing it to write config to `/etc/foo` even if the main user account cannot for security reasons.
#### ExecStart wrapper script
Instead of running the main `ExecStart` script directly, you can run a shell script
which sets up the environment, then runs the main script with `exec`.
Set `exec_start_wrap` to the name of the script, e.g.
`deploy-runtime-environment-wrap` from `mix_deploy`.
In Elixir 1.9+ releases you can use `env.sh` to do similar things, but the
`exec_start_wrap` script runs earlier, potentially with elevated permissions,
so it may still be useful.
#### Runtime environment service
You can run your own separate `systemd` service to configure the runtime environment
before the app runs. Set `runtime_environment_service_script` to a script such
as `deploy-runtime-environment-file` from `mix_deploy`. This library will
create a `#{service_name}-runtime-environment.service` unit and make it a
systemd runtime dependency of the app.
### Runtime dependencies
Systemd starts units in parallel when possible. To enforce ordering, set
`unit_after_targets` to the names of systemd units that this unit depends on.
For example, if this unit should run after `cloud-init` to get [runtime network
information](https://cloudinit.readthedocs.io/en/latest/topics/network-config.html#network-configuration-outputs),
set:
```elixir
unit_after_targets: [
"cloud-init.target"
]
```
## Security
* `paranoia`: Enable systemd security options, default `false`.
NoNewPrivileges=yes
PrivateDevices=yes
PrivateTmp=yes
ProtectSystem=full
ProtectHome=yes
PrivateUsers=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
ProtectControlGroups=yes
MountAPIVFS=yes
│
* `chroot`: Enable `systemd` [chroot](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#RootDirectory=), default `false`.
Sets `systemd` `RootDirectory` to `current_dir`. You can also set [ReadWritePaths=, ReadOnlyPaths=,
InaccessiblePaths=](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#ReadWritePaths=)
with `read_write_paths`, `read_only_paths`, and `inaccessible_paths`, respectively.
## Distillery
Distillery has been replaced by Elixir native releases. This library works fine
with it, though. `exec_start_pre` scripts are particularly useful in the
absence of `env.sh`.
Configure the library by setting `release_system: :distillery`, e.g..
```elixir
config :mix_systemd,
release_system: :distillery,
exec_start_pre: [
# Run script as root before starting
["!", :deploy_dir, "/bin/deploy-sync-config-s3"]
],
dirs: [
:configuration,
:runtime,
],
runtime_directory_preserve: "yes",
env_vars: [
# Use /run/foo for temp files
["RELEASE_MUTABLE_DIR=", :runtime_dir],
# expand $CONFIGURATION_DIR in config files
REPLACE_OS_VARS=true,
]
```
Set up Distillery
[config providers](https://hexdocs.pm/distillery/Mix.Releases.Config.Providers.Elixir.html)
in `rel/config.exs`:
```elixir
environment :prod do
set config_providers: [
{Mix.Releases.Config.Providers.Elixir, ["${CONFIGURATION_DIR}/config.exs"]}
]
end
```
This reads files in Elixir config format. Instead of including your
`prod.secret.exs` file in `prod.exs`, you can copy it to the server separately,
and it will be read at startup.
The [TOML configuration provider](https://github.com/bitwalker/toml-elixir) works similarly:
```elixir
environment :prod do
set config_providers: [
{Toml.Provider, [path: "${CONFIGURATION_DIR}/config.toml"]},
]
end
```
Add the TOML config provider to `mix.exs`:
```elixir
{:toml_config_provider, "~> 0.2.0"}
```
You can generate a file under the release with an overlay in
`rel/config.exs`, e.g.:
```elixir
environment :prod do
set overlays: [
{:mkdir, "etc"},
{:copy, "rel/etc/environment", "etc/environment"},
# {:template, "rel/etc/environment", "etc/environment"}
]
end
```
That results in a file that would be read by:
```elixir
env_files: [
["-", :current_dir, "/etc/environment"],
],
```
Documentation is here: https://hexdocs.pm/mix_systemd
# Links
* https://www.freedesktop.org/software/systemd/man/systemd.exec.html
* https://www.freedesktop.org/software/systemd/man/systemd.service.html
* https://github.com/systemd/systemd/blob/master/NEWS
# Contacts
I am `jakemorrison` on on the Elixir Slack and Discord, `reachfh` on Freenode
`#elixir-lang` IRC channel. Happy to chat or help with your projects.
# Congributing
This project uses the Contributor Covenant version 2.1. Check [CODE_OF_CONDUCT.md](/CODE_OF_CONDUCT.md) for more information.