How should I store my environment variables?

Question

This is a very broad question about methods and advice regarding environment variables / structure. But ultimately I'm looking for answers for the very specific question of 'How should I store my environment variables?'

Firstly some clarifications:

• An environment to me can be from 3 to 10 servers and is a way of containing a specific customer's infrastructure.
• Inside each environment there are a few variables which are mostly automatically generated from a few key inputs (name, size etc).

As it stands right now we're storing all of our environment variables in a structure like so:

<playbook>.yml                   # Various playbooks for deployment
roles/windows                    # Ansible role for Ubuntu
roles/ubuntu                     # Ansible role for Ubuntu
config/hosts/<name>.yml          # Ansible inventory
config/hosts/vars/<name>.json    # Environment specific variables 

Right now the config is initialised as a submodule in the git repository above. As the variables file changes rather frequently this has caused issues with the data changing, once, twice or even three times between commits making changes increasingly difficult to trace.

As I personally see it going forward we should be looking to store all our customer variables in a centralised / scalable way and then hook into it with a dynamic inventory with ansible.

I understand there are a few technologies which seem to do a part of what could be required such as Consul but they seem to work best in an environment which serves one big application rather than lots of smaller slightly differing ones.

I essentially see us having to write an inventory script and then just shove all of our data into some not-for-purpose built database and then keep going on as if nothing changed. I see this conceivably as a way to potentially trim down on a lot of the data we currently store and perhaps look into different ways of storing the data rather than just scaling up what serves it again.

I'm hoping somebody has some kind of experience in implementing infrastructure as code when having to deal with lots of smaller environments as opposed to one, two or three huge ones.

Any suggestions?

Answer 1

I've had two runs at doing environment variables in a scalable way and neither has ended up perfect because, as I've discovered, is a very tricky thing to get right. I'll give a summary of both of my experiences below:

Common Factors

• Environment variables are stored in a separate repository from the original source code (they are submoduled together but are still based on separate repos)
• There is a separate "build" process for the artifact and it's variables.
• There is not a separate release process for the environment variables. If you want to change environment variables you need to go through the same change review boards and usual

Using Consul KV Pairs

The environment variables are loaded from an artifact repository (never the original git repo) and loaded into a namespaced KV pair tree, for example

/env/dev1/my/application/v1.1.1

Where the preceding dev1 is the name of the environment, the my/application is the application namespace, and the v1.1.1 is the version of the environment variables to use.

To developers all these things are invisible. At runtime the platform checks that the environment exists in the current consul cluster (if it doesn't there's a problem and it's errors out), it then checks the subtree for the applications namespace (this way there can be no cross contamination where one app references another apps vars) then the version number of the configuration is taken from label connected to the deployable artifact. Updating this label is the key thing here because it means if we lost both production data centers we could stand up the environment again by simple reading the meta data from our deployable artifacts and loading all the environment variables into the KV store.

Problems With This Approach Developers always, and I mean every single time, found a way to slip configuration changes into the environment that had significant impacts to how the application would run. Because it always ended up being easier to get configuration changes approved than code changes.

Storing a "Deployment" Artifact with variables embedded

This tightly couples the exact version of the artifact to the version of configuration. If you changed configuration then you had to rebuild this deployment artifact.

The deployment artifact itself was essentially a yaml file which contained the URL to the releasable binary and all the configuration attached to it.

The platform contains componentry to read variables and then out them into the process tree of the application when it starts up.

This has so far been a lot more successful because there is an artifact we can trace the history of and which we can hold up to any review board and say "this is the only artifact we care about, we don't need t look at any other changes, only changes to this thing" (ie. Version of application to deploy, environment variables included etc.

This makes it just a little bit harder for developers to try and build logic into their application that will change its behavior based on variables so they can slip in changes without going through appropriate testing cycles.

Bonus Points

Consider application secrets. Our solution to this has so far been to provide a public RSA key which development teams use to encrypt an extended Java key store (almost every language has a library somewhere that can read Java keys stores) this is then treated like a third type of artifact and is pulled onto the server, decrypted with our platform private key, and provided to the application at run time.

Admittedly secrets management is its own can of worms. But it's probably worth considering.

Answer 2

If your environments are per customer, I would suggest in your specific case to have a repository per customer. (In general it is repository per environment.) This repository would have a standard directory structure for environment variables, ansible variables and inventories, strongly encrypted secrets (account access tokens, private keys, etc.). You would git submodule the code into those repositories. I would probably do it in multiple repositories. One for ansible roles and modules, one for maintenance and deployment scripts, one for each major application running in the environments.

Now you can optionally actually fork the code or otherwise pin the submodule at specific tag for release, making sure that the code managing the customer's environment would not change unless tested and released.

If you are using an artifact repository, make sure the artifacts are properly versioned and those versions are specified in the environment variables properly.

Automation is important because the environment variables should not be updated by humans where possible, but generated by scripts. Make sure that there are nearly no manual updates in the per customer inventory and developers only update the code repositories. If they want to make configuration change, it should be done to one of the generating scripts, which is then run to generate the variables and the diff is committed into the customer repository. It pays to setup continuous integration for this process. Without this at some point there will be too many repositories to maintain.