Blog

We all know that managing secrets when developing infrastructure as code can be really tricky, today we start to see some good solutions as Hasicorp Vault and probably others that are being developed.

This weekend I decided to take a more simple approach, using tools that everyone who is reading this article most probably have already installed on his or her computer. I’m currently using Ubuntu nevertheless any other distribution, WSL or Mac will most probably work.

The problem

On my daily work I need to share code not only with my teammates but also with other departments, such as the development department. We all know that sharing a production password or any other kind of secret can have a very negative impact, not only it would be a security breach but also it can end on the hands of one developer which may have the temptation to access some service that he otherwise would not be able to access it.

On the other hand we also want to make sure that we do not need to input this passwords manually every time we wish to deploy something, our goal is to automate as much as possible, the more we automate the less we may encounter errors.

So the question that was puzzling me was “How can I protect my secrets within my code, keep it simple, and at the same time allows me run it through an automated pipeline using Jenkins, Gitlab or any other tool?”

The tools

As I said earlier I will be using tools that we already have installed on our computer, so the tools I’ve use were:

• Make tool
  
• GnuPG
• Ansible Vault
• Git Hooks
• And of course shell scripting

This project is related to infrastructure as code, so the technologies I’ll be showing here will be Terraform and Ansible, but the concept may be extended to other kind of technologies.

This article does not intent to explain how to write makefiles not even shell scripting, there are already a lot of information on the internet, so if you are not use to read or write this kind of files we may feel somehow lost, but don’t worry try to understand the logic behind hit.
The source code is also available here.

The Vault and the Vault Password

On this approach we will use 2 passwords

• A master password called VAULT_PASSWORD
• A encryption password used  by Ansible Vault  stored on a file called VAULT_PASSWORD_FILE

This approach makes Ansible Vault able to read a password from a file instead of passing as an argument of the command itself.

As you may now have guess the VAULT_PASSWORD is the password we use to lock/unlock the vault password file, this password is passed to the make command by a environmental variable with that name.

As you can see the vault part is quite simple in concept, having a master password that opens the vault and allows Ansible Vault to encrypt/decrypt secrets allow us to automate the process, we just make sure that for instance we have an EC2 instance running on AWS very well restricted/protected with this environment variable set and the deployment will run smoothly.

Locking/Unlocking the vault

Let’s now see how it’s done.

Two make targets had been created for this actions, make lock and make unlock, remember that for this to work must have the environment variable set.

VAULT_PASSWORD=my_very_strong_master_password make lock

Let’s see the code now, for now ignore the targets related to encrypt files, we will get into that later:

VAULT_PASSWORD_FILE=security/vault-password
VAULT_UNECRYPTED := $(shell [ -e ${VAULT_PASSWORD_FILE} ] && echo 1 || echo 0 )

# IF VAULT ALREADY OPENED
ifeq ($(VAULT_UNECRYPTED), 1)  
open-vault: 
    @echo "$@: vault already opened, ignoring"
close-vault: 
    @echo "$@: closing vault"
    @gpg --quiet -c --batch --armor --passphrase "${VAULT_PASSWORD}" -o ${VAULT_PASSWORD_FILE}.lock ${VAULT_PASSWORD_FILE}
    @rm ${VAULT_PASSWORD_FILE}
lock: vault-set-env encrypt-ssh-keys encrypt-terraform-files close-vault
unlock: vault-set-env decrypt-ssh-keys decrypt-terraform-files
# IF VAULT CLOSED
else    
open-vault:
    @echo "$@: opening vault"
    @gpg --quiet -d --batch --armor --passphrase "${VAULT_PASSWORD}" -o ${VAULT_PASSWORD_FILE} ${VAULT_PASSWORD_FILE}.lock 
    @rm ${VAULT_PASSWORD_FILE}.lock 
close-vault: 
    @echo "$@: vault already closed, ignoring"
lock: vault-set-env open-vault encrypt-ssh-keys encrypt-terraform-files
    @echo "$@: closing vault"
    @gpg --quiet -c --batch --armor --passphrase "${VAULT_PASSWORD}" -o ${VAULT_PASSWORD_FILE}.lock ${VAULT_PASSWORD_FILE}
    @rm ${VAULT_PASSWORD_FILE} 
unlock: vault-set-env open-vault decrypt-ssh-keys decrypt-terraform-files 
endif

As you can see we from the code we first check if the VAULT_PASSWORD_FILE exists, this may look a little awkward, but it’s very easy to understand, every time we lock the vault we remove the unencrypted version.

@echo "$@: closing vault"
    @gpg --quiet -c --batch --armor --passphrase "${VAULT_PASSWORD}" -o ${VAULT_PASSWORD_FILE}.lock ${VAULT_PASSWORD_FILE}
    @rm ${VAULT_PASSWORD_FILE} 

This means that if a version of the unencrypted file exists, it most probably we have the vault already unlocked, so that is why we make this check on the beginning, depending on if the vault is already unlocked the lock/unlock targets will have a different behavior.

As we could see the vault is a very simple approach, every time we lock it we use the master password to encrypt it using GnuPG, to unlock we revert the process.

Having the vault opened will allow Ansible Vault to easily pick up the password from the file and encrypt/decrypt the secrets.

Encrypting / Decrypting secrets

Now that we already now how to open and close the vault let’s now focus on how we will protect our secrets. In this particular case I’ll be protecting private SSH keys and other terraform var-files.

My project structure looks like this:

Project
|----ansible
|    |----inventory
|    |    |----non-prod
|    |    |----prod
|    |    |----rnd
|    |----roles
|    |    | < my roles here >
|    |----project.yml
|----security
|    |----non-prod
|    |    |----ssh-keys
|    |    |    | < my private ssh keys here >
|    |    |----terraform
|    |    |    | < my private terraform variables here >
|    |----prod
|    |    |----ssh-keys
|    |    |    | < my private ssh keys here >
|    |    |----terraform
|    |    |    | < my private terraform variables here >
|    |----rnd
|    |    |----ssh-keys
|    |    |    | < my private ssh keys here >
|    |    |----terraform
|    |    |    | < my private terraform variables here >
|    |----vault-password
|----terraform
|    |----backend
|    |    |----non-prod
|    |    |    |----config.yml
|    |    |----prod
|    |    |    |----config.yml
|    |    |----rnd
|    |    |    |----config.yml
|    |----vars
|    |    |----non-prod
|    |    |    |---- < my var files >
|    |    |----prod
|    |    |    |---- < my var files >
|    |    |----rnd
|    |    |    |---- < my var files >

Probably not the best, but again this is just a proof of concept. The security folder is were I want to have my secrets encrypted and protected so don’t worry about the rest of the folders for now.

Let’s again look into the code:

SSH_PRIVATE_KEYS=security/${ENV}/ssh-keys
TERRAFORM_SECRETS=security/${ENV}/terraform
VAULT_PASSWORD_FILE=security/vault-password
SSH_KEYS := $(foreach ENV,$(ENVS),$(wildcard ${SSH_PRIVATE_KEYS}/*.key))
TERRAFORM_SECRET_VARS := $(foreach ENV,$(ENVS),$(wildcard ${TERRAFORM_SECRETS}/*.tfvars))

ifeq ($(strip ${SSH_KEYS}),)
encrypt-ssh-keys:
    @echo "$@: no private ssh keys, skipping"
decrypt-ssh-keys:
    @echo "$@: no private ssh keys, skipping"
else
encrypt-ssh-keys:
    @ansible-vault encrypt --vault-password-file=${VAULT_PASSWORD_FILE} ${SSH_KEYS} 2>/dev/null && [ $$? -eq 0 ] && echo "$@ : ${SSH_KEYS}encrypted" || echo "$@ : ${SSH_KEYS}already encrypted, skipping"
decrypt-ssh-keys:
    @ansible-vault decrypt --vault-password-file=${VAULT_PASSWORD_FILE} ${SSH_KEYS} 2>/dev/null && [ $$? -eq 0 ] && echo "$@ : ${SSH_KEYS}decrypted" || echo "$@ : ${SSH_KEYS}already decrypted, skipping"
endif

ifeq ($(strip ${TERRAFORM_SECRET_VARS}),)
encrypt-terraform-files:
    @echo "$@: no terraform secrets, skipping"
decrypt-terraform-files:
    @echo "$@: no terraform secrets, skipping"
else
encrypt-terraform-files:
    @ansible-vault encrypt --vault-password-file=${VAULT_PASSWORD_FILE} ${TERRAFORM_SECRET_VARS} 2>/dev/null && [ $$? -eq 0 ] && echo "$@ : ${TERRAFORM_SECRET_VARS}encrypted" || echo "$@ : ${TERRAFORM_SECRET_VARS}already encrypted, skipping"
decrypt-terraform-files:
    @ansible-vault decrypt --vault-password-file=${VAULT_PASSWORD_FILE} ${TERRAFORM_SECRET_VARS} 2>/dev/null && [ $$? -eq 0 ] && echo "$@ : ${TERRAFORM_SECRET_VARS}decrypted" || echo "$@ : ${TERRAFORM_SECRET_VARS}already decrypted, skipping"
endif

On this targets we take care of encrypting/decrypting the secrets using ansible-vault and asking it to use our password file. As long as we open the vault ansible-vault will be able to encrypt/decrypt the files.

The ifeq sentence allow us to make sure that there are files in the folder to be encrypted/decrypted.

Protecting Git from committing unprotected passwords

As I explained in the beginning of this article, our main goal is to assure that we keep our secrets protected, one risk we might have is if some authorized team member commits to the remote repository unencrypted secrets.

Here is where git hooks come into play, git allows for shell scripts run before committing the code to the remote repository, we just need to make sure we check for unencrypted secrets that are staged and take action before committing them.

Again let’s look into the code:

#!/bin/bash
TERRAFORM_SECRETS="security/*/terraform"
SSH_PRIVATE_KEYS="security/*/ssh-keys"
VAULT_PASSWORD_FILE="security/vault-password"

if [ -z "${VAULT_PASSWORD}" ]; then
    printf "\n${BOLD}${RED}***** CRITICAL ERROR *****${RESET}\n"
    printf "${YELLOW}Please set ${BOLD}VAULT_PASSWORD${RESET}${YELLOW} environment variable before working${RESET}\n"
    printf "${YELLOW}with this project, for more information check README.md!${RESET}\n\n"
    exit 1
fi

exec 1>&2

if [ -e ${VAULT_PASSWORD_FILE}.lock ]; then 

    if gpg --batch --passphrase "${VAULT_PASSWORD}" --trust-model always --quiet --decrypt ${VAULT_PASSWORD_FILE}.lock 2>/dev/null >/dev/null; then
        printf "${BOLD}${GREEN}Right vault password!${RESET}\n"
    else
        printf "${BOLD}${RED}Wrong encrypted vault! Security alert! Please verify your VAULT_PASSWORD!${RESET}"
        exit 1
    fi
else
    VAULT_UNLOCKED=1
    if git diff --name-only --cached ${VAULT_PASSWORD_FILE}; then
        printf "${BOLD}${YELLOW}Found unlocked unlocked vault on commit, removing it!${RESET}\n"
        git rm --cached ${VAULT_PASSWORD_FILE} 2>/dev/null
    fi
fi

for SSH_PRIV_KEY in $(find ${SSH_PRIVATE_KEYS} -type f -name "*.key"); do 

    # check if file is about to be commited

    if git diff --name-only --cached ${SSH_PRIV_KEY} 2>/dev/null >/dev/null; then

        if ssh-keygen -y -e -f ${SSH_PRIV_KEY} 2>/dev/null >/dev/null; then
            printf "${BOLD}${YELLOW}SSH private key not encrypted, removing from current commit (%s)${RESET}\n" "${SSH_PRIV_KEY}"
            git rm --cached ${SSH_PRIV_KEY} 2>/dev/null
            UNENCRYPTED_SSH_KEYS+=("${SSH_PRIV_KEY}")
            ERROR=1
        else
            printf "${BOLD}${GREEN}SSH private key encrypted (%s)${RESET}\n" "${SSH_PRIV_KEY}"
        fi
    fi

done

for TF_SECRET_FILE in $(find ${TERRAFORM_SECRETS} -type f -name "*.tfvars"); do 

    # check if file is about to be commited

    if git diff --name-only --cached ${TF_SECRET_FILE} 2>/dev/null >/dev/null; then

        # Check if the file is an ansible encrypted file
        #
        # 1st step: check for the signature according to ansible format
        # ( https://docs.ansible.com/ansible/latest/user_guide/vault.html#vault-format )
        # 
        if egrep '\$ANSIBLE_VAULT;[0-9]+(\.[0-9]+)+(;([a-z;A-Z,1-9,\.]*))*' --quiet ${TF_SECRET_FILE}; then 

            # 2nd step: check if the lines between header and last line have
            # 80 characters ( 81 if we count the line break )
            # Again according to valut format all of this lines should have
            # 80 characters with the exception of the last line.

            # Get the number of lines and subtract by two, first is header
            # second is last line and also get the number of characters

            NUM_LINES=$(wc -l < ${TF_SECRET_FILE})
            NUM_LINES=$(( ${NUM_LINES} - 2 ))
            NUM_CHARACTERS=$(egrep '\$ANSIBLE_VAULT;[0-9]+(\.[0-9]+)+(;([a-z;A-Z,1-9,\.]*))*' -v ${TF_SECRET_FILE} | egrep -v "$(tail -n 1 ${TF_SECRET_FILE})" | wc -c )
            
            # if the file is correct the NUM_CHARACTERS should be divisible by 81 and it
            # should also be equal to the NUM_LINES;

            CHECK_DIVISION=$(printf "%.3f" $(( 1000 * ${NUM_CHARACTERS} / 81 ))e-3)

            if printf "%s" "${CHECK_DIVISION}" | egrep --quiet "[0-9]+.000"; then

                if [ $(( ${NUM_CHARACTERS} / 81 )) -ne ${NUM_LINES} ]; then
                    printf "${BOLD}${YELLOW}Terraform secret file is not encrypted, removing from current commit (%s)${RESET}\n" "${TF_SECRET_FILE}"
                    git rm --cached ${TF_SECRET_FILE} 2>/dev/null
                    UNENCRYPTED_TF_FILES+=("${TF_SECRET_FILE}")
                    ERROR=1
                else
                    printf "${BOLD}${YELLOW}Terraform secret is encrypted (%s)${RESET}\n" "${TF_SECRET_FILE}"
                fi
            
            else
                printf "${BOLD}${YELLOW}Terraform secret file is not encrypted, removing from current commit (%s)${RESET}\n" "${TF_SECRET_FILE}"
                git rm --cached ${TF_SECRET_FILE} 2>/dev/null
                UNENCRYPTED_TF_FILES+=("${TF_SECRET_FILE}")
                ERROR=1
            fi
        else
            printf "${BOLD}${YELLOW}Terraform secret file is not encrypted, removing from current commit (%s)${RESET}\n" "${TF_SECRET_FILE}"
            git rm --cached ${TF_SECRET_FILE} 2>/dev/null
            UNENCRYPTED_TF_FILES+=("${TF_SECRET_FILE}")
            ERROR=1
        fi

    fi

done

if [ ${ERROR} -eq 1 ]; then 

    # Make sure that we lock all secrets
    printf "${BOLD}${YELLOW}Locking secrets${RESET}\n"
    make lock

    # Add unprotected files to the commit again
    printf "${BOLD}${YELLOW}Adding files again to the commit${RESET}\n"
    for SSH_PRIV_KEY in ${UNENCRYPTED_SSH_KEYS}; do
        printf "${BOLD}${Green}Addind private ssh-key: %s${RESET}\n" "${SSH_PRIV_KEY}"
        git add ${SSH_PRIV_KEY} 2>/dev/null
    done

    for TF_SECRET_FILE in ${UNENCRYPTED_TF_FILES}; do 
        printf "${BOLD}${GREEN}Addind terraform secret: %s${RESET}\n" "${TF_SECRET_FILE}"
        git add ${TF_SECRET_FILE} 2>/dev/null
    done

    printf "${BOLD}${GREEN}Addind locked vault password%s${RESET}\n"
    git add ${VAULT_PASSWORD_FILE}.lock 2>/dev/null

fi

printf "${BOLD}${YELLOW}Commit will now be done${RESET}\n"

The above script is called every time we try to commit something to git, it will look for secret files being committed  which are unencrypted, if it finds some unprotected file it will unstage it and in the end will lock again all files and stage again those files which were unprotected.

The only thing we need to assure is that every team member that uses this repo and have commit rights have this hook installed on is machine. What I’ve come up, thanks to this post, was to create one make target that each team member have to call as soon as they start working on the code.

Conclusion

As I said earlier this is one of the multiple approaches one can take to protect secrets. I just wanted to show that sometimes we can tackle some of more complex issues with simpler solutions then adopting a new technology. The import is that in the end of the day

On the next post I’ll show how I’ve used make to call terraform in a easier way and without the need to add –var-file argument whenever I add a new file, and also how I’ve integrated it with this vault system.