Managing Terraform and Ansible secrets with Make
We all know that managing secrets when developing infrastructure as code can be really tricky, today we start to see some good solutions as Hasicorp Vault and probably others that are being developed.
This weekend I decided to take a more simple approach, using tools that everyone who is reading this article most probably have already installed on his or her computer. I’m currently using Ubuntu nevertheless any other distribution, WSL or Mac will most probably work.
The problem
On my daily work I need to share code not only with my teammates but also with other departments, such as the development department. We all know that sharing a production password or any other kind of secret can have a very negative impact, not only it would be a security breach but also it can end on the hands of one developer which may have the temptation to access some service that he otherwise would not be able to access it.
On the other hand we also want to make sure that we do not need to input this passwords manually every time we wish to deploy something, our goal is to automate as much as possible, the more we automate the less we may encounter errors.
So the question that was puzzling me was “How can I protect my secrets within my code, keep it simple, and at the same time allows me run it through an automated pipeline using Jenkins, Gitlab or any other tool?”
The tools
As I said earlier I will be using tools that we already have installed on our computer, so the tools I’ve use were:
- Make tool
- GnuPG
- Ansible Vault
- Git Hooks
- And of course shell scripting
This project is related to infrastructure as code, so the technologies I’ll be showing here will be Terraform and Ansible, but the concept may be extended to other kind of technologies.
This article does not intent to explain how to write makefiles not even shell scripting, there are already a lot of information on the internet, so if you are not use to read or write this kind of files we may feel somehow lost, but don’t worry try to understand the logic behind hit.
The source code is also available here.
The Vault and the Vault Password
On this approach we will use 2 passwords
- A master password called VAULT_PASSWORD
- A encryption password used by Ansible Vault stored on a file called VAULT_PASSWORD_FILE
This approach makes Ansible Vault able to read a password from a file instead of passing as an argument of the command itself.
As you may now have guess the VAULT_PASSWORD is the password we use to lock/unlock the vault password file, this password is passed to the make command by a environmental variable with that name.
As you can see the vault part is quite simple in concept, having a master password that opens the vault and allows Ansible Vault to encrypt/decrypt secrets allow us to automate the process, we just make sure that for instance we have an EC2 instance running on AWS very well restricted/protected with this environment variable set and the deployment will run smoothly.
Locking/Unlocking the vault
Let’s now see how it’s done.
Two make targets had been created for this actions, make lock and make unlock, remember that for this to work must have the environment variable set.
VAULT_PASSWORD=my_very_strong_master_password make lock
Let’s see the code now, for now ignore the targets related to encrypt files, we will get into that later:
VAULT_PASSWORD_FILE=security/vault-password
VAULT_UNECRYPTED := $(shell [ -e ${VAULT_PASSWORD_FILE} ] && echo 1 || echo 0 )
# IF VAULT ALREADY OPENED
ifeq ($(VAULT_UNECRYPTED), 1)
open-vault:
@echo "$@: vault already opened, ignoring"
close-vault:
@echo "$@: closing vault"
@gpg --quiet -c --batch --armor --passphrase "${VAULT_PASSWORD}" -o ${VAULT_PASSWORD_FILE}.lock ${VAULT_PASSWORD_FILE}
@rm ${VAULT_PASSWORD_FILE}
lock: vault-set-env encrypt-ssh-keys encrypt-terraform-files close-vault
unlock: vault-set-env decrypt-ssh-keys decrypt-terraform-files
# IF VAULT CLOSED
else
open-vault:
@echo "$@: opening vault"
@gpg --quiet -d --batch --armor --passphrase "${VAULT_PASSWORD}" -o ${VAULT_PASSWORD_FILE} ${VAULT_PASSWORD_FILE}.lock
@rm ${VAULT_PASSWORD_FILE}.lock
close-vault:
@echo "$@: vault already closed, ignoring"
lock: vault-set-env open-vault encrypt-ssh-keys encrypt-terraform-files
@echo "$@: closing vault"
@gpg --quiet -c --batch --armor --passphrase "${VAULT_PASSWORD}" -o ${VAULT_PASSWORD_FILE}.lock ${VAULT_PASSWORD_FILE}
@rm ${VAULT_PASSWORD_FILE}
unlock: vault-set-env open-vault decrypt-ssh-keys decrypt-terraform-files
endif
As you can see we from the code we first check if the VAULT_PASSWORD_FILE exists, this may look a little awkward, but it’s very easy to understand, every time we lock the vault we remove the unencrypted version.
@echo "$@: closing vault"
@gpg --quiet -c --batch --armor --passphrase "${VAULT_PASSWORD}" -o ${VAULT_PASSWORD_FILE}.lock ${VAULT_PASSWORD_FILE}
@rm ${VAULT_PASSWORD_FILE}
This means that if a version of the unencrypted file exists, it most probably we have the vault already unlocked, so that is why we make this check on the beginning, depending on if the vault is already unlocked the lock/unlock targets will have a different behavior.
As we could see the vault is a very simple approach, every time we lock it we use the master password to encrypt it using GnuPG, to unlock we revert the process.
Having the vault opened will allow Ansible Vault to easily pick up the password from the file and encrypt/decrypt the secrets.
Encrypting / Decrypting secrets
Now that we already now how to open and close the vault let’s now focus on how we will protect our secrets. In this particular case I’ll be protecting private SSH keys and other terraform var-files.
My project structure looks like this:
Project
|----ansible
| |----inventory
| | |----non-prod
| | |----prod
| | |----rnd
| |----roles
| | | < my roles here >
| |----project.yml
|----security
| |----non-prod
| | |----ssh-keys
| | | | < my private ssh keys here >
| | |----terraform
| | | | < my private terraform variables here >
| |----prod
| | |----ssh-keys
| | | | < my private ssh keys here >
| | |----terraform
| | | | < my private terraform variables here >
| |----rnd
| | |----ssh-keys
| | | | < my private ssh keys here >
| | |----terraform
| | | | < my private terraform variables here >
| |----vault-password
|----terraform
| |----backend
| | |----non-prod
| | | |----config.yml
| | |----prod
| | | |----config.yml
| | |----rnd
| | | |----config.yml
| |----vars
| | |----non-prod
| | | |---- < my var files >
| | |----prod
| | | |---- < my var files >
| | |----rnd
| | | |---- < my var files >
Probably not the best, but again this is just a proof of concept. The security folder is were I want to have my secrets encrypted and protected so don’t worry about the rest of the folders for now.
Let’s again look into the code:
SSH_PRIVATE_KEYS=security/${ENV}/ssh-keys
TERRAFORM_SECRETS=security/${ENV}/terraform
VAULT_PASSWORD_FILE=security/vault-password
SSH_KEYS := $(foreach ENV,$(ENVS),$(wildcard ${SSH_PRIVATE_KEYS}/*.key))
TERRAFORM_SECRET_VARS := $(foreach ENV,$(ENVS),$(wildcard ${TERRAFORM_SECRETS}/*.tfvars))
ifeq ($(strip ${SSH_KEYS}),)
encrypt-ssh-keys:
@echo "$@: no private ssh keys, skipping"
decrypt-ssh-keys:
@echo "$@: no private ssh keys, skipping"
else
encrypt-ssh-keys:
@ansible-vault encrypt --vault-password-file=${VAULT_PASSWORD_FILE} ${SSH_KEYS} 2>/dev/null && [ $$? -eq 0 ] && echo "$@ : ${SSH_KEYS}encrypted" || echo "$@ : ${SSH_KEYS}already encrypted, skipping"
decrypt-ssh-keys:
@ansible-vault decrypt --vault-password-file=${VAULT_PASSWORD_FILE} ${SSH_KEYS} 2>/dev/null && [ $$? -eq 0 ] && echo "$@ : ${SSH_KEYS}decrypted" || echo "$@ : ${SSH_KEYS}already decrypted, skipping"
endif
ifeq ($(strip ${TERRAFORM_SECRET_VARS}),)
encrypt-terraform-files:
@echo "$@: no terraform secrets, skipping"
decrypt-terraform-files:
@echo "$@: no terraform secrets, skipping"
else
encrypt-terraform-files:
@ansible-vault encrypt --vault-password-file=${VAULT_PASSWORD_FILE} ${TERRAFORM_SECRET_VARS} 2>/dev/null && [ $$? -eq 0 ] && echo "$@ : ${TERRAFORM_SECRET_VARS}encrypted" || echo "$@ : ${TERRAFORM_SECRET_VARS}already encrypted, skipping"
decrypt-terraform-files:
@ansible-vault decrypt --vault-password-file=${VAULT_PASSWORD_FILE} ${TERRAFORM_SECRET_VARS} 2>/dev/null && [ $$? -eq 0 ] && echo "$@ : ${TERRAFORM_SECRET_VARS}decrypted" || echo "$@ : ${TERRAFORM_SECRET_VARS}already decrypted, skipping"
endif
On this targets we take care of encrypting/decrypting the secrets using ansible-vault and asking it to use our password file. As long as we open the vault ansible-vault will be able to encrypt/decrypt the files.
The ifeq sentence allow us to make sure that there are files in the folder to be encrypted/decrypted.
Protecting Git from committing unprotected passwords
As I explained in the beginning of this article, our main goal is to assure that we keep our secrets protected, one risk we might have is if some authorized team member commits to the remote repository unencrypted secrets.
Here is where git hooks come into play, git allows for shell scripts run before committing the code to the remote repository, we just need to make sure we check for unencrypted secrets that are staged and take action before committing them.
Again let’s look into the code:
#!/bin/bash
TERRAFORM_SECRETS="security/*/terraform"
SSH_PRIVATE_KEYS="security/*/ssh-keys"
VAULT_PASSWORD_FILE="security/vault-password"
if [ -z "${VAULT_PASSWORD}" ]; then
printf "\n${BOLD}${RED}***** CRITICAL ERROR *****${RESET}\n"
printf "${YELLOW}Please set ${BOLD}VAULT_PASSWORD${RESET}${YELLOW} environment variable before working${RESET}\n"
printf "${YELLOW}with this project, for more information check README.md!${RESET}\n\n"
exit 1
fi
exec 1>&2
if [ -e ${VAULT_PASSWORD_FILE}.lock ]; then
if gpg --batch --passphrase "${VAULT_PASSWORD}" --trust-model always --quiet --decrypt ${VAULT_PASSWORD_FILE}.lock 2>/dev/null >/dev/null; then
printf "${BOLD}${GREEN}Right vault password!${RESET}\n"
else
printf "${BOLD}${RED}Wrong encrypted vault! Security alert! Please verify your VAULT_PASSWORD!${RESET}"
exit 1
fi
else
VAULT_UNLOCKED=1
if git diff --name-only --cached ${VAULT_PASSWORD_FILE}; then
printf "${BOLD}${YELLOW}Found unlocked unlocked vault on commit, removing it!${RESET}\n"
git rm --cached ${VAULT_PASSWORD_FILE} 2>/dev/null
fi
fi
for SSH_PRIV_KEY in $(find ${SSH_PRIVATE_KEYS} -type f -name "*.key"); do
# check if file is about to be commited
if git diff --name-only --cached ${SSH_PRIV_KEY} 2>/dev/null >/dev/null; then
if ssh-keygen -y -e -f ${SSH_PRIV_KEY} 2>/dev/null >/dev/null; then
printf "${BOLD}${YELLOW}SSH private key not encrypted, removing from current commit (%s)${RESET}\n" "${SSH_PRIV_KEY}"
git rm --cached ${SSH_PRIV_KEY} 2>/dev/null
UNENCRYPTED_SSH_KEYS+=("${SSH_PRIV_KEY}")
ERROR=1
else
printf "${BOLD}${GREEN}SSH private key encrypted (%s)${RESET}\n" "${SSH_PRIV_KEY}"
fi
fi
done
for TF_SECRET_FILE in $(find ${TERRAFORM_SECRETS} -type f -name "*.tfvars"); do
# check if file is about to be commited
if git diff --name-only --cached ${TF_SECRET_FILE} 2>/dev/null >/dev/null; then
# Check if the file is an ansible encrypted file
#
# 1st step: check for the signature according to ansible format
# ( https://docs.ansible.com/ansible/latest/user_guide/vault.html#vault-format )
#
if egrep '\$ANSIBLE_VAULT;[0-9]+(\.[0-9]+)+(;([a-z;A-Z,1-9,\.]*))*' --quiet ${TF_SECRET_FILE}; then
# 2nd step: check if the lines between header and last line have
# 80 characters ( 81 if we count the line break )
# Again according to valut format all of this lines should have
# 80 characters with the exception of the last line.
# Get the number of lines and subtract by two, first is header
# second is last line and also get the number of characters
NUM_LINES=$(wc -l < ${TF_SECRET_FILE})
NUM_LINES=$(( ${NUM_LINES} - 2 ))
NUM_CHARACTERS=$(egrep '\$ANSIBLE_VAULT;[0-9]+(\.[0-9]+)+(;([a-z;A-Z,1-9,\.]*))*' -v ${TF_SECRET_FILE} | egrep -v "$(tail -n 1 ${TF_SECRET_FILE})" | wc -c )
# if the file is correct the NUM_CHARACTERS should be divisible by 81 and it
# should also be equal to the NUM_LINES;
CHECK_DIVISION=$(printf "%.3f" $(( 1000 * ${NUM_CHARACTERS} / 81 ))e-3)
if printf "%s" "${CHECK_DIVISION}" | egrep --quiet "[0-9]+.000"; then
if [ $(( ${NUM_CHARACTERS} / 81 )) -ne ${NUM_LINES} ]; then
printf "${BOLD}${YELLOW}Terraform secret file is not encrypted, removing from current commit (%s)${RESET}\n" "${TF_SECRET_FILE}"
git rm --cached ${TF_SECRET_FILE} 2>/dev/null
UNENCRYPTED_TF_FILES+=("${TF_SECRET_FILE}")
ERROR=1
else
printf "${BOLD}${YELLOW}Terraform secret is encrypted (%s)${RESET}\n" "${TF_SECRET_FILE}"
fi
else
printf "${BOLD}${YELLOW}Terraform secret file is not encrypted, removing from current commit (%s)${RESET}\n" "${TF_SECRET_FILE}"
git rm --cached ${TF_SECRET_FILE} 2>/dev/null
UNENCRYPTED_TF_FILES+=("${TF_SECRET_FILE}")
ERROR=1
fi
else
printf "${BOLD}${YELLOW}Terraform secret file is not encrypted, removing from current commit (%s)${RESET}\n" "${TF_SECRET_FILE}"
git rm --cached ${TF_SECRET_FILE} 2>/dev/null
UNENCRYPTED_TF_FILES+=("${TF_SECRET_FILE}")
ERROR=1
fi
fi
done
if [ ${ERROR} -eq 1 ]; then
# Make sure that we lock all secrets
printf "${BOLD}${YELLOW}Locking secrets${RESET}\n"
make lock
# Add unprotected files to the commit again
printf "${BOLD}${YELLOW}Adding files again to the commit${RESET}\n"
for SSH_PRIV_KEY in ${UNENCRYPTED_SSH_KEYS}; do
printf "${BOLD}${Green}Addind private ssh-key: %s${RESET}\n" "${SSH_PRIV_KEY}"
git add ${SSH_PRIV_KEY} 2>/dev/null
done
for TF_SECRET_FILE in ${UNENCRYPTED_TF_FILES}; do
printf "${BOLD}${GREEN}Addind terraform secret: %s${RESET}\n" "${TF_SECRET_FILE}"
git add ${TF_SECRET_FILE} 2>/dev/null
done
printf "${BOLD}${GREEN}Addind locked vault password%s${RESET}\n"
git add ${VAULT_PASSWORD_FILE}.lock 2>/dev/null
fi
printf "${BOLD}${YELLOW}Commit will now be done${RESET}\n"
The above script is called every time we try to commit something to git, it will look for secret files being committed which are unencrypted, if it finds some unprotected file it will unstage it and in the end will lock again all files and stage again those files which were unprotected.
The only thing we need to assure is that every team member that uses this repo and have commit rights have this hook installed on is machine. What I’ve come up, thanks to this post, was to create one make target that each team member have to call as soon as they start working on the code.
Conclusion
As I said earlier this is one of the multiple approaches one can take to protect secrets. I just wanted to show that sometimes we can tackle some of more complex issues with simpler solutions then adopting a new technology. The import is that in the end of the day
On the next post I’ll show how I’ve used make to call terraform in a easier way and without the need to add –var-file argument whenever I add a new file, and also how I’ve integrated it with this vault system.