Exploring a redesign and discussing how a modern Rust configuration library should behave

[mehcode]

This library has gotten quite stale over the last year and I've been loathe to come back to it mostly because I don't like it. It's complex to use, complex to develop, etc. I apologize for letting it atrophy.

I'd like to thank you everyone who has contributed or submitted a PR ( especially those who've done so in the last several months and have just sat there ). It's not that I don't want to merge in your work, it's that if I come back to this I'll want to do too much and I haven't had the time available to do that.

---

With that out of the way I want to explore how a configuration library could behave in Rust, optimally, using some example use cases as a jumping off point:

// load configuration only from envrionment
let settings: Settings = config::env::vars()
    // filter to only keys that case-insentively start with s
    .prefix("app_")
    // explictly transform key names to allow traversing nested configuration
    // replaces .separator("__") from config v0.9
    .map(|key| key.replace("__", "."))
    .load()?;

// find a single configuration file from example.* in
let settings: Settings = config::file::find("example")
    // look in the following paths (by default cwd)
    .paths(&[xdg_config_home, home])
    // expect format in TOML only
    // default allows all formats compiled in
    .format(Format::Toml)
    // and load into Settings
    .load()?;

// find multiple configuration files and merge together
let settings: Settings = config::file::find_all(&[
    "default",
    mode // development, production, or test, etc.
]).load()?;

// load configuration from arguments
let settings: Settings = config::env::args(clap_matches).load();

// manually create a configuration stack
let settings: Settings = config::Stack::new()
    // start with an embedded JSON file 
    .add(config::file::from_bytes(Format::Json, include_bytes!("default.json")))
    // override from environment
    .add(config::env::vars())
    // load into settings
    .load()?;

// create a dynamic configuration (ala config v0.9)
// any _source_ could be built into a Config struct
let settings: config::Config = config::file::find("example").build();

// a dyn configuration can listen for changes on supported sources
let _handle = settings.watch(|| do_something_fun_when_config_changes);

// use path access
let host: &str = settings.get("db.host");

---

Thoughts? How would you want config to work?

Are there any pain points in the current design you'd like to discuss?

[mockersf]

Just a comment describing how I tend to setup configuration:

• Start with default values for config
• Then try to read a config file in XDG config folder and override
• Then, for every folder in the current path, starting from /, try to read a config file and override
• Allow user to specify / override some options using command line arguments, defined through structopt
• Finally, allow user to specify / override some options using environment variables

My pain points:

• Some configuration options are only possible through configuration files, other are only possible through command line arguments.
• I want my CLI to report errors for missing parameters in a consistent ways for missing parameters that can only be specified as arguments or for parameters that could also be present in a configuration file (if value is found in a file, do not report an error)
• Clap can show a default value for an argument in the --help, I would like it to show the value found from a configuration file if there is one rather than a default that would be override by said configuration file

[mwillsey]

I love how config lets you write hierarchical configuration settings, but that does raise an interesting question as to how overrides/merging works. Example:

struct Settings {
    foo: String,
    bar: Option<BarSettings>,
    bar: Option<BarSettings>,
}

struct BarSettings {...}
struct BazSettings {...}

And then my config looks like this:

foo = "hello world"

[bar]
item1 = 1234
item2 = "asdf"

If I create Settings from the config file, I'll end up with Settings.baz being None, which makes sense. My question is how would I override that (or merge it in) from the environment variables?

If you try to merge in something like APP_BAR= or something like that, you end up merging in a (potentially empty) string in for the value of a table! Then, serde will fail to deserialize into Settings. This is surprising to me because it's merging in a string for the value of (what was) a table.

I see a couple potential ways to address this:

1. Special case merging the empty string with a table, making the result and empty table
2. Somehow generalize 1 into some kind of "typed" merging
3. Manually walk the dynamic config, essentially replacing empty strings with empty tables by hand (where appropriate).

I suppose a similar situation could occur when getting config values from the command line. I think the larger questions is what does it mean to get config values from a non-hierarchical source, and then later merge them with a hierarchical source?

[mehcode]

@mwillsey

If I create Settings from the config file, I'll end up with Settings.bar being None, which makes sense. My question is how would I override that (or merge it in) from the environment variables?

The idea is that you would override using the key app.baz.one which would match a field one in BazSettings. As that's not possible to write in an env var there is a .separator("_") function you can call on config::Environment which does a string replace of _ for ..

---

@mockersf

1. Do you merge all of those configuration files together or is it last one wins?

My pain points: [...]

It seems we need to think hard on how CLI arguments tie into this. I have a fairly good idea on how to tie clap in normally but I hadn't thought of how to tie config into clap as a "default values provider".

[mwillsey]

@mehcode Sorry, edited parent comment. If I create a config from the above config file, I should get Settings.baz (not bar, as previously stated). The question is, how do I "cancel out" Settings.bar, which was defined as a table in the config file.

I understand the separator trick, but I AFAIK it's not possible to use that to un-define a table (in this case, Settings.bar.

[mehcode]

@mwillsey Hmm.. Un-define in general is a hard concept. APP_BAR= is understood as "undefined" not "null" (by default) as its a very common unix idiom to unset an environment variable like that instead of actually unsetting it. So it would do nothing to the previous properties.

We could probably allow a JSON string to be merged in for a table or array. Viper (go config lib) does that.. but that only can get you an empty map. Not a null map. I suppose that could still work if you go one property up.. APP='{"bar": null}'.

Un-setting from a string is probably a lot easier as you could just merge in a string and tell config to treat it as JSON or TOML or something.