Note: The first few sections here are significantly more narrative-ish than any technical blog post has a right to be. Skip to the “S3crets” header for the real meat.

Also Note: I moved *.normmaclennan.com to AWS over the weekend, which has resulted in lots of things to write posts about. This is one of them.

I like git. I enjoy keeping things in git repositories. But some things just should not be kept in git repositories.

One of the things I like to keep in git repositories is servers. That’s really a topic for another post. But, in short, it involves putting Terraform, Vagrant, and Puppet together in a repository and seeing that happens. But when I go to provision such a server, I need a few things: AWS credentials, SSH keys, SSL certificates, encryption keys, cloud-init configuration…the list goes on.

So how do I balance my impulse to put every scrap of code I write up on github and the need to keep secrets secret?

Somebody Set Up Us The Secrets

Of course, the simplest - well ‘simplest’ - solution to this is to keep your own local copies of the settings in the directory, but .gitignore them. Then the real config file is planted on the server manually (or via some other/unknown external force).

This introduces a barrier to entry. Now when I clone the repository I’m left with some random sprinkling of settings.example.changeme in random places throughout the repo and I can’t do anything with the code until I’ve fixed everything up. Then, I need to ensure that I don’t commit my custom settings.

It also introduces some unnecessary complexity. Now, when setting up a new server, I need to make sure the ‘real’ values have been populated by $unknown_force. If this is a personal project, I am $unknown_force. I need to gather all of the non-versioned secret files, whereever they are, and put them on the server manually. If $unknown_force is not me, I just need to trust that it was done correctly. And if my server is not functioning correctly, I need to figure out which item-that-is-not-automated has failed.

Encryption and Inconvenience

The traditional advice is to encrypt all of your secret settings. Which is probably a good idea in many cases. I’ve been known to encrypt Hiera data here and there with hiera-gosecret.

But it is very much a tradeoff. As you can probably see in the image above, encrypted settings are terrible to work with. What even is that setting? Whenever you need to view or change the settins, you need to decrypt and re-encrypt. Just hope you have the key close at hand.

This requires a little bit less thought (e.g. what can I commit, what needs to be ignored?), and requires a minimum of planting secrets - you just need the encryption key for the settings. Not the worst thing in the world.

Incremental Improvement

Unfortunately, I don’t really have any revolutionary ideas. But it’s not all bad news.

We were discussing this at work a little while ago. We use Secret Server to store our passwords and secret files. Some of the READMEs in our repos had sections to the effect of “download the following files from Secret Server and put them in the following places.” And that got super-tedious.

One day we were discussing Secret Server’s actually-pretty-good API and somebody had the great idea to make a Bundler/Librarian/Berkshelf-alike for secrets.

Zanzibar

And, thus, Zanzibar was born. And there has been a proliferation of repositories with Zanzifiles that will download all requisite secrets into a central gitignored directory. Now, those READMEs just say “run bundle install to get Zanzibar and zanzibar install to download all necessary secrets.” Some will even fetch you different secrets depending on which branch you have checked out (e.g. dev vs prd secrets).

nb. it was named after the GI Joe character, but we later learned that the “zanzibar gem” is actually a type of flower. So choose whichever etymology you like.

And I saw that it was good. And I totally wanted to use it for my personal projects. Problem being, I don’t have access to Secret Server. I looked into doing something similar with LastPass or 1Password or anything like that. But nothing already existed and it looked to difficult to make.

Then I realized that I keep many of my secret files in an S3 bucket.

S3crets

So I set about making a gem called s3crets. And when I went to publish it, I found such a gem already exists. Oops. A quick rename to aws-s3crets later and we were good to go.

Just like Zanzibar, S3crets is a gem that puts a Bundler-like wrapper around the gathering of secret files. But S3crets, as the name implies, gets them from an S3 bucket.

Just s3crets init in your repository to get a template Secretfile. Or you can:

1
$ s3crets init -b [S3 Bucket Name] -r [AWS region] -f [AWS credentials file]

If you don’t give S3crets a credentials file path, it will default to ~/.aws/credentials. It uses the aws-sdk gem under the hood, so you can provide credentials to S3crets in the same way as you can for the aws-sdk gem.

After you have a Secretfile, open it up and add some secrets. It’s a simple yaml file. Just add a list underneath the secrets hash:

1
---
settings:
  bucket: 'secrets_bucket'
  region: 'us-west-2'
  secret_dir: secrets/dev
secrets:
  aws_key: 'AWS/Keys/ec2-myteam-write'
  ssh_key: 'SSH/myteam/myserver/server-priv'
  cloud_config: 'AWS/cloudinit/myserver.yaml'

The key is just a semantically-meaningful name to give users a hint as to why they are downloading these files. The value is, of course, the key/path to an object inside the S3 bucket.

Then, when you s3crets install, it will download the files to secret_dir (if secret_dir is not provided, it downloads to the current directory). Make sure those files are gitignored.

S3crets also provides a default task that will enumerate all Secretfiles in your repository and generate a bunch of rake secrets:foldername tasks to run s3crets install in each directory.

How I’ve Been Using It

As I’ve been transitioning this blog’s server to live in a git repo (to be open sourced along with a post or two about it), I have made heavy use of S3crets. As a matter of fact, this is what I wrote it for.

I have a coule of Secretfiles. One for my local/dev secrets and one for my production secrets. All I have on my system is an AWS credentials file with read-only access to the Secrets bucket.

The Secretfiles describe the following secrets:

  • An AWS credentials file with write access to S3/EC2/Route53 for Terraform to use.
  • An SSH private key to match the public key on the server Terraform will provision for me.
  • The cloud-init config for said server.
  • The SSL certificate and key for the webserver.
  • The gosecret encryption key for the values that I still need to encrypt into hiera-gosecret.

This way, I can make use easy use of these secret files without having to worry about accidentally sending any of the up to github when I push the rest of the repository.