CI/CD secret management on Kubernetes

At Banzai Cloud we’re building a managed Cloud Native application and devops platform, called Pipeline. Pipeline supercharges the development, deployment and scaling of container-based applications with native support for multi- and hybrid-cloud environments.

Pipeline’s built-in CI/CD solution is capable of creating Kubernetes clusters, running and testing builds, packaging and deploying applications as Helm charts, and lots more—all while its secrets are stored and managed by Vault.

Note: The Pipeline CI/CD module mentioned in this post is outdated and not available anymore. You can integrate Pipeline to your CI/CD solution using the Pipeline API. Contact us for details.

Secrets and CI 🔗︎

These days CI/CD systems do way more than just build code: they access SCM repositories, push artifacts, create clusters, and more—all in multi-tenant and (often) open source environments. Simultaneously, developers within a given organization may have different roles and authorizations, however, they all need to be able to start builds and check their outcomes and logs. It’s difficult to manage all that: requires lots of maintenance, and storage way in excess of environment variables, files, or being stuck in CI descriptors.

Also, secrets should be short lived, or volatile—they should be easily and often revoked and replaced (especially when they are compromised), all of which should be automated.

We quickly realized this while working on Pipeline and:

• we standardized and open sourced a Vault operator for Kubernetes
• our enterprise customers were unhappy with Kubernetes’ secrets (base64 encoding, tsk-tsk), so we allowed them to inject secrets directly into pods from Vault
• we run all CI builds wihtin Kubernetes in unprivileged mode
• secrets that are pulled from Vault are ephemeral and persistent
• in Pipeline CI/CD descriptor secrets are referenced only, and binding is automatically occurs only when necessary and for only as long as it’s required
• additionally, build artifacts and images can be security scanned as part of the CI pipeline

Secret management in CI, the Pipeline way 🔗︎

In Pipeline, users are members of organizations. Users may invite other users into organizations, and a user can be a member of more than one organization at a time. These organizations can have different types of associated resources: Kubernetes clusters created inside organizations, object storage buckets, spotguide catalogs, etc. One feature that all these have in common is that they require secrets to work properly. Usually, these secrets allow Pipeline to interact with your cloud provider account in order to provision and manage any underlying resources. Secrets may also be associated with applications running on your Kubernetes clusters, like a TLS certificate chain for MySQL, or a Kubernetes cluster’s own credentials. Secrets are stored in Vault with added layers of abstraction that allow them to be provided to applications and build pipelines.

Using secrets in build steps 🔗︎

At some point, you’ll probably be executing a build step that needs to use a credential to do its job. In the CI/CD flow, which runs natively on Kubernetes, Pipeline secrets can be injected directly into Pods that represent build steps. If you need one of your secrets for a specific build step, you can specify it in the step’s secretFrom parameter. This binds the value of a secret’s field to an environment variable supplied to your build step. For example, you can use a secret that’s holding Docker Hub credentials to build and push an image:

pipeline:
  # ...
  build_container:
    image: plugins/docker
    repo: foo/bar
    secretFrom:
      DOCKER_USERNAME:
        name: my-docker-secret
        keyRef: username
      DOCKER_PASSWORD:
        name: my-docker-secret
        keyRef: password

To use a Pipeline secret in the steps of a specific CI/CD project, you must first attach the secret. On the web interface, go to the CI/CD page, click on the project’s settings button, and attach the secret you want to use.

A plugin’s documentation will specify the name of the environment variables that the plugin expects the credentials to be in, but you have the freedom to choose how you store them in Pipeline. Generally speaking, it’s possible to create a Generic-type secret, and add fields to it with the same names as the variables. Then the keys in the secretFrom object will be repeated as keyRefs, and the name parameter will be the same for all variables. However, the example of a docker plugin above uses a Password type secret, which has username and password fields. These are mapped to anticipated variable names.

Installing secrets to clusters 🔗︎

If your application (your Helm charts) needs credentials in the form of Kubernetes Secrets, you can install them to the cluster in order to let the Pods of a given namespace consume them. The following code snippet is an example configuration for installing a secret:

pipeline:
  install_secret:
    image: banzaicloud/ci-pipeline-client:0.7
    action: InstallSecret
    clusterSecret:
      sourceSecretName: pipeline-secret-name
      name: installed-secret-name
      namespace: default

With the help of the Pipeline, users can define how these secrets should be injected into the Kubernetes cluster their build is running on.

Learn more about Banzai Cloud Pipeline platform 🔗︎

If you’re interested in learning more about using Pipeline with a CI system, contact us.

If you’d like to learn more about Banzai Cloud check out our other posts on this blog, the Pipeline, Hollowtrees and Bank-Vaults projects.