Inject secrets directly into Pods from Vault revisited

A key part of the Banzai Cloud Pipeline platform, has always been our strong focus on security. We incorporated Vault into our architecture early on in the design process, and we have developed a number of support components to be easily used with Kubernetes. We love what Vault enables us to do, but, as with many things security-related, strengthening one part of our system exposed a weakness elsewhere. For us, that weakness was K8s secrets, which is the standard way in which applications consume secrets and credentials on Kubernetes. Any secret that is securely stored in Vault and then unsealed for consumption will eventually end up as a K8s secret, and with much less protection and security than we’d like. K8s secrets use base64 encoding that, while better than nothing, does not satisfy our standards, and likely fails to satisfy the standards of most enterprise clients as well. As a result, we’ve developed a solution wherein we can bypass the K8s secrets mechanism and inject secrets directly into Pods from Vault.

Vault -> Kubernetes secrets -> Pod 🔗︎

If you are familiar with Kubernetes secrets, you know that these secrets are stored in etcd. When we say that we intend to bypass K8s security, we mean that we don’t intend to touch etcd at all; the problem with etcd is that when data is encrypted at rest, it is encrypted with a global key (see the relevant documentation). That’s less than ideal in a multi-tenant cluster, where independent and unrelated users might potentially gain access to the secrets of others. Also, if you already have a security team that’s operating a certified Vault installation, they’re probably not going to be happy about placing an unencrypted secret in an intermediary location (Kubernetes secrets, i.e. etcd).

Banzai Cloud’s Pipeline platform already used Kubernetes webhooks to provide a range of advanced features (security scans, spot instance scheduling, annotating webhooks, etc.), and it occured to us that using a webhook to inject secrets directly into Kubernetes containers from Vault would be a good way of bypassing etcd.

Let’s dive into how it works.

Kubernetes mutating webhook for injecting secrets 🔗︎

Our mutating admission webhook injects an executable into containers (in a non-intrusive way) inside Pods, which then request secrets from Vault through special environment variable definitions. This project was inspired by a number of other projects (e.g. channable/vaultenv, hashicorp/envconsul), but one thing that makes it unique is that it is a daemonless solution.

First, the Kubernetes webhook checks if a container has environment variables with values that correspond to a specific schema. Then it reads the values for those variables directly from Vault at start-up:

1	env:
2	- name: AWS_SECRET_ACCESS_KEY
3		value: "vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY"

After that, the init-container is injected into the Pod, and a small binary called vault-env is attached to it as an in-memory volume. That volume is mounted to all containers with the appropriate environment variable definitions.

The init-container also changes the command of the container to run vault-env, instead of running the application directly. vault-env starts up, connects to Vault (using the Kubernetes Auth method), checks that the environment variables have a reference to a value stored in Vault (vault:secret/....) and replaces that with a corresponding value from Vault’s secrets backend. Afterward, vault-env executes the original process (with syscall.Exec()), which uses the secret that was originally stored in Vault.

Using this solution prevents secrets stored in Vault from landing in Kubernetes secrets (and in etcd).

vault-env was designed to work on Kubernetes, but there’s nothing stopping it from being used outside of Kubernetes as well. It can be configured with the standard Vault client’s environment variables, since there’s a standard Go Vault client underneath.

Currently, the Kubernetes Service Account-based Vault authentication mechanism is used by vault-env, which requests a Vault token in return for the Service Account of the container it’s being injected into. But our implementation is going to change in order to allow the use of the Vault Agent’s Auto-Auth feature very soon. This will allow users to request tokens in init-containers with all the authentication mechanisms supported by Vault Agent, so they won’t be handcuffed to the Kubernetes Service Account-based method.

Why is this more secure than using Kubernetes secrets or using any other custom sidecar container? 🔗︎

Our solution is particularly lightweight and uses only existing Kubernetes constructs like annotations and environment variables. No confidential data ever persists on the disk - not even temporarily - or in etcd. All secrets are stored in memory, and only visible to the process that requests them. If you want to make this solution even more robust, you can disable kubectl exec-ing in running containers. If you do so, no one will be able to hijack injected environment variables from a process.

Additionally, there is no persistent connection with Vault, and any Vault token used to read environment variables is flushed from memory before the application starts, in order to minimize attack surface.

A complete example 🔗︎

This example will guide you through setting up a fully functional Vault installation with the Banzai Cloud Vault operator, and help you to create an example deployment that will be mutated by the webhook so that environment variables can be injected into Pods:

# These examples require Helm 3 and kubectl:

# Add the Banzai Cloud Helm repository
helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com

# Create a namespace for the bank-vaults components called vault-infra
# Namespace labeling is required, because the webhook's mutation is based on label selectors
kubectl create namespace vault-infra
kubectl label namespace vault-infra name=vault-infra

# Install the vault-operator to the vault-infra namespace
helm upgrade --namespace vault-infra --install vault-operator banzaicloud-stable/vault-operator --wait

# Clone the bank-vaults project
git clone git@github.com:banzaicloud/bank-vaults.git
cd bank-vaults

# Create a Vault instance with the operator which has the Kubernetes auth method configured
kubectl apply -f operator/deploy/rbac.yaml
kubectl apply -f operator/deploy/cr.yaml

# Now you have a fully functional Vault installation on top of Kubernetes,
# orchestrated by the `banzaicloud/vault-operator` and `banzaicloud/bank-vaults`.

# Next, install the mutating webhook with Helm into its own namespace (to bypass the catch-22 situation of self mutation)
helm upgrade --namespace vault-infra --install vault-secrets-webhook banzaicloud-stable/vault-secrets-webhook --wait

# Set the Vault token from the Kubernetes secret
# (strictly for demonstrative purposes, we have K8s unsealing in cr.yaml)

export VAULT_TOKEN=$(kubectl get secrets vault-unseal-keys -o jsonpath={.data.vault-root} | base64 --decode)

# Tell the CLI that the Vault Cert is signed by a custom CA

kubectl get secret vault-tls -o jsonpath="{.data.ca\.crt}" | base64 --decode > $PWD/vault-ca.crt
export VAULT_CACERT=$PWD/vault-ca.crt

# Tell the CLI where Vault is listening (the certificate has 127.0.0.1 as well as alternate names)

export VAULT_ADDR=https://127.0.0.1:8200

# Forward the TCP connection from your Vault pod to localhost (in the background)

kubectl port-forward service/vault 8200 &

# Write a secret into Vault, which will be injected as an environment variable

vault kv put secret/accounts/aws AWS_SECRET_ACCESS_KEY=s3cr3t

# Apply the deployment with special environment variables
# It will be mutated by the webhook

kubectl apply -f deploy/test-deployment.yaml

The deployment will be mutated by the webhook, because it has at least one environment variable that has a value that is a reference to a path in Vault. Here’s what the original deployment looks like:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-secrets
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-secrets
  template:
    metadata:
      labels:
        app: hello-secrets
      annotations:
        vault.security.banzaicloud.io/vault-addr: "https://vault:8200"
        vault.security.banzaicloud.io/vault-tls-secret: "vault-tls"
    spec:
      serviceAccountName: default
      containers:
      - name: alpine
        image: alpine
        command: ["sh", "-c", "echo $AWS_SECRET_ACCESS_KEY && echo going to sleep... && sleep 10000"]
        env:
        - name: AWS_SECRET_ACCESS_KEY
          value: "vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY"

It produces Pods like so (only the relevant parts are shown here, Pods are mutated directly):

apiVersion: v1
kind: Pod
metadata:
  name: hello-secrets-575554499f-26894
  labels:
    app: hello-secrets
  annotations:
    vault.banzaicloud.io/vault-addr: "https://vault:8200"
    vault.security.banzaicloud.io/vault-tls-secret: "vault-tls"
spec:
  initContainers:
  - name: copy-vault-env
    command:
    - sh
    - -c
    - cp /usr/local/bin/vault-env /vault/
    image: banzaicloud/vault-env:latest
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - mountPath: /vault/
      name: vault-env
  containers:
  - name: alpine
    command:
    - /vault/vault-env
    args:
    - sh
    - -c
    - echo $AWS_SECRET_ACCESS_KEY $ && echo going to sleep... && sleep 10000
    image: alpine
    imagePullPolicy: Always
    env:
    - name: AWS_SECRET_ACCESS_KEY
      value: vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY
    - name: VAULT_ADDR
      value: https://vault:8200
    - name: VAULT_SKIP_VERIFY
      value: "false"
    - name: VAULT_PATH
      value: kubernetes
    - name: VAULT_ROLE
      value: default
    - name: VAULT_IGNORE_MISSING_SECRETS
      value: "false"
    - name: VAULT_CACERT
      value: /vault/tls/ca.crt
    volumeMounts:
    - mountPath: /vault/
      name: vault-env
    - mountPath: /vault/tls/ca.crt
      name: vault-tls
      subPath: ca.crt
  volumes:
  - emptyDir:
      medium: Memory
    name: vault-env
  - name: vault-tls
    secret:
      secretName: vault-tls

As you can see, none of the original environment variables in the definiition have been touched, and the sensitive value of the AWS_SECRET_ACCESS_KEY variable is only visible inside the alpine container.

Using charts without an explicit container.command and container.args 🔗︎

The Webhook is now capable of determining the container’s entry point and command with the help of image metadata queried from the image registry. This data is cached until the webhook Pod is restarted. If the registry is publicly accessible (without authentication), you don’t need to do anything, but, if the registry requires authentication, the necessary credentials have to be made available in the Pod’s imagePullSecrets section, or in the Pod’s ServiceAccount.

NOTE: Future improvement: on AWS and GKE and other cloud providers get a credential dynamically with the cloud-specific SDK

MySQL example: 🔗︎

# Put the MySQL passwords into Vault
vault kv put secret/mysql MYSQL_ROOT_PASSWORD=s3cr3t MYSQL_PASSWORD=3xtr3ms3cr3t

# Install the MySQL chart with root and a user password sourced from Vault
helm upgrade --install mysql stable/mysql \
  --set mysqlRootPassword=vault:secret/data/mysql#MYSQL_ROOT_PASSWORD \
  --set mysqlPassword=vault:secret/data/mysql#MYSQL_PASSWORD \
  --set "podAnnotations.vault\.security\.banzaicloud\.io/vault-addr"=https://vault:8200 \
  --set "podAnnotations.vault\.security\.banzaicloud\.io/vault-tls-secret"=vault-tls \
  --wait


# Open a connection towards the MySQL Service
kubectl port-forward service/mysql 3306 &

# Read the MySQL user password's secret from Vault
# Make sure you still have the port-forward from the previous example
vault read secret/data/mysql

Key         Value
---         -----
data        map[MYSQL_PASSWORD:3xtr3ms3cr3t MYSQL_ROOT_PASSWORD:s3cr3t]
metadata    map[created_time:2019-09-05T13:03:42.980780517Z deletion_time: destroyed:false version:1]

# Open up the MySQL shell with the root user and its corresponding password from Vault `s3cr3t`
mysql -h 127.0.0.1 -u root -p

Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 84
Server version: 5.7.14 MySQL Community Server (GPL)

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>


# And here's the magic: there are still no secrets in the env vars!
kubectl exec -it mysql-749cfddc67-5slcb bash
root@mysql-749cfddc67-5slcb:/\# env | grep MYSQL.*PASSWORD
MYSQL_PASSWORD=vault:secret/data/mysql#MYSQL_PASSWORD
MYSQL_ROOT_PASSWORD=vault:secret/data/mysql#MYSQL_ROOT_PASSWORD

Of course, if you exec into the Pod and rerun vault-env it will fork a new process which will have those environment variables correctly set:

/vault/vault-env env | grep MYSQL.*PASSWORD
2019/09/05 13:42:09 Received new Vault token
2019/09/05 13:42:09 Initial Vault token arrived
MYSQL_PASSWORD=3xtr3ms3cr3t
MYSQL_ROOT_PASSWORD=s3cr3t

Consequentially, it’s advised you disallow the "pods/exec" Kubernetes RBAC rule snippet for users you don’t want doing this:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  namespace: default
  name: pod-execer
rules:
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create"]

Getting secrets data from Vault and transplanting it into a Kubernetes secret 🔗︎

You can mutate secrets by setting annotations and defining the proper Vault path in the secret data:

apiVersion: v1
kind: Secret
metadata:
  name: sample-secret
  annotations:
    vault.security.banzaicloud.io/vault-addr: "https://vault.default.svc.cluster.local:8200"
    vault.security.banzaicloud.io/vault-role: "default" # In case of Secrets the webhook's ServiceAccount is used
    vault.security.banzaicloud.io/vault-skip-verify: "true"
    vault.security.banzaicloud.io/vault-path: "kubernetes"
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: eyJhdXRocyI6eyJodHRwczovL2RvY2tlci5pbyI6eyJ1c2VybmFtZSI6InZhdWx0OnNlY3JldC9kYXRhL2RvY

In the example above, the secret type is kubernetes.io/dockerconfigjson, and the webhook is capable of getting credentials from vault. The base64 encoded data contains vault paths for usernames and passwords for docker repositories. You can create it with the following commands:

kubectl create secret docker-registry dockerhub --docker-username="vault:secret/data/dockerrepo#DOCKER_REPO_USER" --docker-password="vault:secret/data/dockerrepo#DOCKER_REPO_PASSWORD"
kubectl annotate secret dockerhub vault.security.banzaicloud.io/vault-addr="https://vault.default.svc.cluster.local:8200"
kubectl annotate secret dockerhub vault.security.banzaicloud.io/vault-role="default"
kubectl annotate secret dockerhub vault.security.banzaicloud.io/vault-skip-verify="true"
kubectl annotate secret dockerhub vault.security.banzaicloud.io/vault-path="kubernetes"

Multiple (and dynamic) secret backends not just KV 🔗︎

Currently, vault-env supports reading Values from the KV backend, but we have added support for dynamic secrets as well - database URLs with temporary usernames and passwords for batch or scheduled jobs, for example. This feature is implemented with consul-template’s Vault component and is based on the work of Jürgen Weber. It deserved its own blog-post, and is described in detail in Vault webhook - complete secret support with consul-template.

Extensions in the works 🔗︎

Something we’re still working on is templating (transforming/combining secret values), an extension based on the Go and Sprig templates; this is a frequently requested freature.

The webhook is now available as an integrated service (we used to call them posthooks) of the Pipeline platform, which means that you can install it on a Kubernetes clusters via Pipeline (with the UI or with the CLI). It will then configure Vault with the authentications and policies necessary for it to work with our webhook - track this issue here.

For more information, or if you’re interested in contributing, check out the Bank-Vaults repo - the Vault Swiss army knife and operator for Kubernetes, and/or give us a GitHub star if you think the project deserves it!

Learn more about Bank-Vaults:

• Secret injection webhook improvements
• Backing up Vault with Velero
• Vault replication across multiple datacenters
• Vault secret injection webhook and Istio
• Mutate any kind of k8s resources
• HSM support
• Injecting dynamic configuration with templates
• OIDC issuer discovery for Kubernetes service accounts
• Show all posts related to Bank-Vaults
  

About Banzai Cloud Pipeline 🔗︎

Banzai Cloud’s Pipeline provides a platform for enterprises to develop, deploy, and scale container-based applications. It leverages best-of-breed cloud components, such as Kubernetes, to create a highly productive, yet flexible environment for developers and operations teams alike. Strong security measures — multiple authentication backends, fine-grained authorization, dynamic secret management, automated secure communications between components using TLS, vulnerability scans, static code analysis, CI/CD, and so on — are default features of the Pipeline platform.