Use xdg basedir spec on linux

[jleclanche]

Terraform Version

0.9.8

https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html

TLDR: Instead of ~/.terraform.d, configuration should be in the $XDG_CONFIG_HOME/terraform/ and cache (safely-deletable files) should be in $XDG_CACHE_HOME/terraform. If not defined, $XDG_CONFIG_HOME should be defaulted to $HOME/.config and $XDG_CACHE_HOME should be defaulted to $HOME/.cache.
I believe everything currently in .terraform.d is considered cache, therefore there's just $XDG_CACHE_HOME to worry about.

This is pretty standard and helps keep clutter out of the homedir :)

[apparentlymart]

Thanks for pointing this out, @jleclanche.

This seems like a reasonable idea as long as we can find a way to get there without breaking things for people already using the current paths, though should think about whether it makes sense to adopt this just for Linux (and thus have to document another platform-specific difference) or across all UNIX-flavored platforms (where it might seem more alien).

I'd like to let this one soak for a little while before we take any action. Thanks again for pointing it out!

[jleclanche]

👍
Out of curiosity is there anything that would actually break things if not given an upgrade path?
All I see in my dir is .terraform.d/checkpoint_cache .terraform.d/checkpoint_signature; both of which look to be cache for update checks.

[jleclanche]

Ah sorry, i see in config_unix.go that there's a .terraformrc file as well, I'm just not using it. Disregard.

[apparentlymart]

Yeah, indeed... seems worth thinking about splitting the cache-ish things from the settings-ish things too, like you said.

[jcrben]

Is there an environment variable to customize the location of this directory? I switch between OSX and Linux and I like to use environment variables to keep the location of config files consistent.

[jleclanche]

@apparentlymart I see you've been working on the config lately, any chance this is going in too?

[apparentlymart]

Hi again @jleclanche! Sorry for leaving this here so long.

We've been adding some more stuff into this dir in the mean time, such as more config files (.tfrc files) and a search location for plugins, so I think we do need to make sure to have a good upgrade path here, and indeed not everything in there is cache anymore. 😖

I do like the idea of following the relevant standards and of distinguishing config from cache, but I want to go into it with a stronger idea of how this changes all of our supported platforms, rather than just changing it for Linux, or indeed forcing Linux-specific conventions on other platforms.

I think to properly support this we need to alter our model to separately account for a config and cache dir on each platform, and then use the appropriate directory for each case. AFAICT on Mac OS X the appropriate directories are:

• Config: ~/Library/Preferences/
• Cache: ~/Library/Caches/

To get the migration path we need, and also to meet @jcrben's reasonable request at being able to use common directories across both OS X and Linux, I think we'd need to continue looking for config-ish things in ~/.terraform.d in addition to $XDG_CONFIG_HOME/terraform for pre-existing things (provider plugins and .tfrc files), but that doesn't seem like a big hardship.

AFAICT the BSDs also don't really use XDG and so continuing to use ~/.terraform.d alone is probably the most ergonomic thing there.

I wasn't able to get a read on whether Solaris users expect XDG support.

While we're doing this, we should also check if there's a config vs. cache dir distinction to be made for Windows systems as well. We currently use the same prefix for both on Windows.

I can't promise working on this soon since there are many other things competing for attention, but if we can get some consensus on what the most natural thing is on each platform then at least when someone has some time to look at this we'll have a reasonably-clear path forward.

[jleclanche]

If you want to look at how to implement it cross-platform, I recommend looking at the QStandardPaths implementation from Qt:

https://doc.qt.io/qt-5/qstandardpaths.html

This follows XDG standards on Linux and other platform-specific quirks for macOS and Windows.

[apparentlymart]

Awesome... thanks for that pointer, @jleclanche!

[jcrben]

In the case of a fair number of my tools, such as git, if I define the XDG variables, they are used on OSX as well. See https://git-scm.com/docs/git-config#git-config---global

A simpler place to start is to just let me place the entire directory into a different place.

[0az]

As a Mac user, I am against ~/Library/Preferences. According to rust-lang/rfcs#1615 (comment), the folder is for configuration plists for the "user defaults" subsystem, though I couldn't find an official source for this. Instead, there seems to be consensus in following XDG or providing an environment variable, both of which allow for neater, cleaner $HOMEs.

The actual change does not appear difficult, relative to the docs and migration. I might try submitting a PR for this once I'm more proficient at Go.

`rg 'terraform\.d' terraform/ -c` on 69c647c

terraform/plugins.go:1
terraform/config_windows.go:1
terraform/CHANGELOG.md:1
terraform/config_unix.go:1
terraform/command/command.go:1
terraform/website/upgrade-guides/0-7.html.markdown:1
terraform/website/guides/running-terraform-in-automation.html.md:1
terraform/website/docs/plugins/basics.html.md:2
terraform/website/docs/configuration/providers.html.md:4
terraform/website/docs/commands/cli-config.html.markdown:1
terraform/website/docs/commands/init.html.markdown:2

`rg 'terraform\.d' terraform/` on 69c647c

terraform/plugins.go
18:	// Look in ~/.terraform.d/plugins/ , or its equivalent on non-UNIX
terraform/CHANGELOG.md

488:* Load plugins from ~/.terraform.d/plugins/OS_ARCH/ and .terraformrc (#15769)
terraform/config_windows.go

33:	return filepath.Join(dir, "terraform.d"), nil
terraform/config_unix.go

27:	return filepath.Join(dir, ".terraform.d"), nil
terraform/command/command.go

28:const DefaultPluginVendorDir = "terraform.d/plugins/" + pluginMachineName
terraform/website/upgrade-guides/0-7.html.markdown

25:These binaries needed to all be extracted to somewhere in your $PATH or in the ~/.terraform.d directory for Terraform to work.
terraform/website/guides/running-terraform-in-automation.html.md

308:terraform.d/plugins/OS_ARCH directory, which will be searched before
terraform/website/docs/plugins/basics.html.md

42:directory, located at %APPDATA%\terraform.d\plugins on Windows and

43:~/.terraform.d/plugins on other systems.
terraform/website/docs/configuration/providers.html.md

214:Windows           | %APPDATA%\terraform.d\plugins

215:All other systems | ~/.terraform.d/plugins

286:plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"

302:export TF_PLUGIN_CACHE_DIR="$HOME/.terraform.d/plugin-cache"
terraform/website/docs/commands/cli-config.html.markdown

43:plugin_cache_dir   = "$HOME/.terraform.d/plugin-cache"
terraform/website/docs/commands/init.html.markdown

125:~/.terraform.d/plugins on most operating systems and

126:%APPDATA%\terraform.d\plugins on Windows.

[apparentlymart]

Hi all,

We're intentionally not making any changes related to this right now because we're trying to wrap up development of the v0.12.0 release, which already has a large scope and so we'd prefer not to make changes to how Terraform interacts with the host system at the same time. However, I'd love to dive into this some more after that and see what makes sense here.

Given that there are a few different options, I think the best way to proceed here is to come to some consensus on what behavior is desirable first and only then move forward to implementation. I am open to using XDG on OS X if that is emerging as a de-facto standard in other tools, but I'd like to go into that with a fully-detailed idea of exactly what paths that will end up using on each of the different platforms Terraform supports, including figuring out what makes sense for all of the non-Linux and non-MacOS unix flavors, and seeing also if any changes are warranted on Windows at the same time so that we can make all necessary changes of this kind in a single release.

In the mean time it'd be helpful to collect here links to existing precedent for how other software is handling these concerns across platforms, since I think consistency with common behavior will be the primary deciding factor in what path we take here.

[rouing]

@apparentlymart

Check this out:

https://wiki.archlinux.org/index.php/XDG_Base_Directory

This is a list of Software Conforming to the XDG Base Directory Standard. If the Source was avail at the time in a git/svn repo, the pr/merge/commit is linked there.

Ex:

https://cgit.freedesktop.org/libreoffice/ure/commit/?id=a6f56f7
https://cgit.freedesktop.org/libreoffice/bootstrap/commit/?id=25bd2ee

Those have OSX Examples

https://src.chromium.org/viewvc/chrome/trunk/src/chrome/common/chrome_paths.cc?r1=23057&r2=23056&pathrev=23057
This one checks for linux.

https://gitlab.gnome.org/GNOME/gimp/commit/60e0cfe
In this case, they check for the .config directory.

I think that is the best approach here.