Vault webhook - complete secret support with consul-template

The following is a guest blog post from Jürgen Weber, Bank-Vaults user and contributor extraordinaire.

Here at hipages, we have a legacy approach to how we keep and maintain our ‘secrets’. The login details for some of our primary application resources are easy to obtain and with this carries great risk.. So to solve this we decided to embark on a ‘secrets’ project and implement Hashicorps Vault.

As a part of this project, we looked at a variety of solutions. Being a small operations team handling 5+ product teams we are always looking for ways to make the big ‘feel’ in terms of time and resources small. This includes realising that you can not be an expert in every technology, all the way to ensuring that what you make available to your teams is quick and easy to understand and implement.

The Kubernetes Operator method is one big win in this area. Companies and owners of products can turn their products (like Vault or Prometheus) into a Kubernetes resource that an administrator can just submit a Resource Definition and voila it exists.

Enter the Banzai Cloud vault-operator, bank-vaults.

Once we had our Vault installation up and running with the help of the Banzai Cloud team we then needed to make our applications ‘talk’ to it somehow. As a guide, we had been using various ​Howtos​ and YouTube videos we found by Seth Vargo as our frame of reference on how to achieve this. His examples use the Vault Kubernetes Auth plugin and consul-template to maintain a configuration file for your application on a memory resident disk inside a Kubernetes Pod. Consul-template will then maintain the credentials and the file, it also has the ability to run an arbitrary command to HUP your application or ask it to read the template file after changes are made to it by consult-template.

I was able to get this working relatively easily but the problem we were seeing with the approach is that it is incredibly manual. I started the Kubernetes implementation here at hipages many a year ago and this was before helm was the norm. This means our suite of applications that are running in Kubernetes are applied using generic Kubernetes YAML manifest files, with each manifest for each application residing in that applications code base. It would mean running around to our 20+ applications/Git repositories and adding all the bits and pieces to every app. How droll!

How can we avoid doing this? We found the answer in the form of a ‘mutating-webhook’ that is part of the bank-vaults project.

A mutating WebHook is the production of a Kubernetes API resource​ where you can ask the Kubernetes API to submit resources to a web server or piece of code of your choosing. In this case, when the Kubernetes API wants to ‘CREATE’ a pod, send it to the bank-vaults mutating webhook first.

At the time of the investigation, the webhook only supported the KV store and PKI backends (using the bank-vaults vault-env CLI) but for our use case, we would need support for the AWS and database secrets backends. I then had a very close look at the code for the webhook and vault-env and considered how consul-template could fit into this ecosystem.

The general workflow for vault-env:

• It looks for pods with the right EnVars/ConfigMaps/Secrets with the correct prefix vault: or >>vault:.
• It then injects an init container running vault that gets a vault token and copies it into the /vault/ volume.
• It then injects another init container that copies vault-env into the /vault/ volume.
• It then inspects every container in the pod, adds the required env vars for vault-env.
• It then concatenates the container command and args and turns that into the args and make /vault/vault-env the command.
• It then runs /vault/vault-env.
• vault-env parses the env/configmaps/secrets updates the env/configmaps and secrets per the correct strings (value: vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY).
• vault-env then executes the binary.

The general workflow for consul-template:

• It requires an init container running vault agent that gets a vault token.
• It has a sidecar container running consul-template in the Pod.
• consul-template takes its configuration and writes a template to a Pod resident filesystem.
• consul-template will renew its vault token when 50% of its lifetime has completed.
• consul-template will renew the credentials in your template based on the configured grace period.
• consul-template can run a command to tell your application that the configuration has changed.
• consul-template itself can be HUP’d to tell it the credentials have been revoked and to get new ones.

A break down of the two approaches:

vault-env:

• Has this really cool env/secret/config map updating mechanism.
• Limited support for secret backends.
• It does not support TTL.
• Takes over the live container.
• It does not deal with files.

consul-template:

• Supports all backends out of the box with its templating.
• Runs as a sidecar daemon and handles TTL on secrets.
• Writes config to a file system (preferably a memory resident one so it disappears on tear down) for the app to pick up.
• Can HUP/run a command to tell the process to take the updated config.
• Does not deal with env/secrets/config maps.

I now have this super neat way to implement what I needed… but it did not support what I needed. So the solution? Add consul-template support to the bank-vaults mutating webhook and have the best of both worlds. So with the help of the BanzaiCloud team, I got down to adding consul-template support in the webhook, with some very efficient code reviews it was all accepted and merged within a week or so.

Here is how you use this new feature (note; this assumes you have a working installation of Vault):

Install the webhook:

helm init -c 
helm repo add banzaicloud-stable http://kubernetes-charts.banzaicloud.com/branch/master 
helm upgrade --namespace my-namespace --install my-vault 
banzaicloud-stable/vault-secrets-webhook 

Setup your consule-template configuration via a ConfigMap:

 1apiVersion: v1 
 2kind: ConfigMap 
 3metadata: 
 4  labels: 
 5    app: my-app 
 6  name: my-app-consul-template 
 7data: 
 8  config.hcl: | 
 9    vault { 
10      ssl { 
11        ca_cert = "/vault/tls/ca.crt" 
12      } 
13      retry { 
14        backoff = "1s" 
15      } 
16    } 
17    template { 
18      contents = <<EOH 
19        {{- with secret "database/creds/readonly" }} 
20        username: {{ .Data.username }} 
21        password: {{ .Data.password }} 
22        {{ end }} 
23      EOH 
24      destination = "/vault/secrets/config.yaml" 
25      command     = "/bin/sh -c \"kill -HUP $(pidof vault-demo-app) || true\"" 
26    } 

Add annotations to your Deployment or PodSpec:

 1  template: 
 2    metadata: 
 3      annotations: 
 4        vault.security.banzaicloud.io/vault-ct-configmap: "my-app-consul-template" 
 5        vault.security.banzaicloud.io/vault-role: my-app 
 6      labels: 
 7        app: my-app 
 8      name: my-app 
 9      namespace: my-namespace 
10    spec:
11      volumes: 
12        - name: app-volume 
13          emptyDir: {} 
14…

You can now see our injected pods:

$ kubectl get pod mypod-svhjh -n my-namespace -o json | jq '.spec.initContainers[].name'
"vault-agent"

This pod starts up and logs into Vault and authenticates against it using the PodSpec’s defined serviceAccountName. It will write out a token into a memory resident shared file system.

$ kubectl get pod mypod-svhjh -n my-namespace -o json | jq '.spec.containers[].name'
"consul-template"
"..."

Kubernetes Batch/Jobs: 🔗︎

If you decide to add the consul-template webhook annotations to a Job Spec, do be aware the default will maintain consul-template as a daemon. This is not desired as the job will never terminate. I recently added another annotation vault.security.banzaicloud.io/vault-ct-once. You can set this to true and consul-template will run once write the template and shutdown. Meaning your Job can run to completion and the pod will die.

Caveats: 🔗︎

You are now editing Pods on the fly, so maybe some users may find it confusing or odd when they install a deployment/pod/helm chart in Kubernetes and all of a sudden it has all of these ‘extra containers’.

Bio: 🔗︎

Jürgen Weber is a technology enthusiast and a lover of all things shiny and new. Having extensive experience in AWS and automation technologies like Puppet. Now the proud maintainer of a successful greenfield Kubernetes implementation at hipages.com.au.

Learn more about Bank-Vaults:

• Secret injection webhook improvements
• Backing up Vault with Velero
• Vault replication across multiple datacenters
• Vault secret injection webhook and Istio
• Mutate any kind of k8s resources
• HSM support
• Injecting dynamic configuration with templates
• OIDC issuer discovery for Kubernetes service accounts
• Show all posts related to Bank-Vaults