Mutating Webhook

How the webhook works - overview 🔗︎

Kubernetes secrets are the standard way in which applications consume secrets and credentials on Kubernetes. Any secret that is securely stored in Vault and then unsealed for consumption eventually ends up as a Kubernetes secret. However, despite their name, Kubernetes secrets are not exactly secure, since they are only base64 encoded.

The mutating webhook of Bank-Vaults is a solution that bypasses the Kubernetes secrets mechanism and injects the secrets retrieved from Vault directly into the Pods. Specifically, the mutating admission webhook injects (in a very non-intrusive way) an executable into containers of Deployments and StatefulSets. This executable can request secrets from Vault through special environment variable definitions.

An important and unique aspect of the webhook is that it is a daemonless solution (although if you need it, you can deploy the webhook in daemon mode as well).

Why is this more secure than using Kubernetes secrets or any other custom sidecar container? 🔗︎

Our solution is particularly lightweight and uses only existing Kubernetes constructs like annotations and environment variables. No confidential data ever persists on the disk or in etcd - not even temporarily. All secrets are stored in memory, and are only visible to the process that requested them. Additionally, there is no persistent connection with Vault, and any Vault token used to read environment variables is flushed from memory before the application starts, in order to minimize attack surface.

If you want to make this solution even more robust, you can disable kubectl exec-ing in running containers. If you do so, no one will be able to hijack injected environment variables from a process.

The webhook checks if a container has environment variables defined in the following formats, and reads the values for those variables directly from Vault during startup time.

        env:
        - name: AWS_SECRET_ACCESS_KEY
          value: vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY
# or
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: aws-key-secret
              key: AWS_SECRET_ACCESS_KEY
# or
        - name: AWS_SECRET_ACCESS_KEY
            valueFrom:
              configMapKeyRef:
                name: aws-key-configmap
                key: AWS_SECRET_ACCESS_KEY

The webhook checks if a container has envFrom and parses the defined ConfigMaps and Secrets:

        envFrom:
          - secretRef:
              name: aws-key-secret
# or
          - configMapRef:
              name: aws-key-configmap

Secret and ConfigMap examples 🔗︎

Secrets require their payload to be base64 encoded, the API rejects manifests with plaintext in them. The secret value should contain a base64 encoded template string referencing the vault path you want to insert. Run echo -n "vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY" | base64 to get the correct string.

apiVersion: v1
kind: Secret
metadata:
  name: aws-key-secret
data:
  AWS_SECRET_ACCESS_KEY: dmF1bHQ6c2VjcmV0L2RhdGEvYWNjb3VudHMvYXdzI0FXU19TRUNSRVRfQUNDRVNTX0tFWQ==
type: Opaque

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-key-configmap
data:
  AWS_SECRET_ACCESSKEY: vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY

For further examples and use cases, see Configuration examples and scenarios.