kubeadm intermediate CA certificates with Vault

As those of you who are following us here at Banzai Cloud may or may not be aware, we are in the middle of releasing/certifying our own Kubernetes distribution — Pipeline Kubernetes Engine (PKE). PKE will be orchestrated the same way as other providers already supported by Pipeline, and will benefit from/inherit those features of the Banzai Cloud Pipeline platform that you already know and love.

If you’re interested in learning more about PKE and our vision for buidling multi and hybrid cloud managed (application) environments, please read this post

PKE has a seamless and automated bootstrap procedure, which requires a PKI infrastructure. One of the jigsaw components we use in this process is kubeadm. Kubeadm can manage PKI certificates by itself but can also integrate with preexisting PKI infrastructure.

The Pipeline platform already contains a secret management engine. We support a lot of very different secret types with that engine — from simple username/password pairs to TLS certificates. It’s possible not only to store those secrets with the engine, but to generate them as well (so long as no static input has been given). The supporting code relies heavily on Pipeline and uses Bank-Vaults to work.

Banzai Cloud PKE Kubernetes distribution clusters use our own PKI infrastructure, which is backed by Vault. This makes it possible among other obvious benefits, to seamlessly renew certificates, and to rebuild a cluster during disaster recovery or during a migration.

In the cluster setup we follow the Single Root CA arrangement, for which we have created a new secret type called distribution that holds all the certificates necessary during the kubeadm bootstrapping process. In this setup you can create a single root CA for the cluster, controlled by an administrator (which is Pipeline + Vault, in our case). This root CA can then create multiple intermediate CAs and delegate all further certificate creation to Kubernetes itself.

The required certificates are:

For each cluster or distribution type, "banzaicloud" uses a separate Vault PKI secret engine mounted to a different path (there are four of these), which is something Vault’s architecture requires us to do — for more details check out this issue as well as the Vault documentation. One secret engine is necessary for the cluster’s root CA and three for the intermediate CAs.

Example of how to call the Secret API and to generate a kubeadm TLS bundle for PKE 🔗︎

This example assumes that you already have a fully operational Pipeline instance running, in this case, on your machine. If you need some help doing that, follow along with our developer documentation.

 1# Instruct Pipeline to create a TLS bundle secret for our "banzai" distribution cluster
 2
 3$ curl -H "Authorization: Bearer $TOKEN" \
 4  -H "Content-Type: application/json" \
 5  -d "{\"type\":\"distribution\", \"name\":\"banzai\", \"values\":{\"clusterUid\": \"12-345\"}}" \
 6  http://localhost:9090/pipeline/api/v1/orgs/1/secrets | jq
 7
 8{
 9  "name": "banzai",
10  "type": "distribution",
11  "id": "26a8dab0bff77fe1821477c0e0b7c09883419b136ae6a745e85a01af664d2db3",
12  "updatedAt": "2019-01-24T12:48:11.081042376Z",
13  "updatedBy": "bonifaido",
14  "version": 1
15}
16
17# Check that Vault has mounted all the specific PKI engines
18
19$ vault secrets list
20Path                                              Type         Accessor              Description
21----                                              ----         --------              -----------
22clusters/12-345/pki/ca/                           pki          pki_fb4f4c0a          root PKI engine for cluster 12-345
23clusters/12-345/pki/etcd-ca/                      pki          pki_041fb1d5          etcd-ca intermediate PKI engine for cluster 12-345
24clusters/12-345/pki/kubernetes-ca/                pki          pki_04aeb3fd          kubernetes-ca intermediate PKI engine for cluster 12-345
25clusters/12-345/pki/kubernetes-front-proxy-ca/    pki          pki_e0a64032          kubernetes-front-proxy-ca intermediate PKI engine for cluster 12-345
26cubbyhole/                                        cubbyhole    cubbyhole_1561fda0    per-token private secret storage
27database/                                         database     database_32607d87     MySQL Database secret engine.
28identity/                                         identity     identity_0f057f0d     identity store
29secret/                                           kv           kv_df66d275           key/value secret storage
30sys/                                              system       system_0e6bb73f       system endpoints used for control, policy and debugging

Kubeadm Singe Root CA setup 🔗︎

By default, kubeadm uses the /etc/kubernetes/pki directory to store its own, on-the-fly, self-signed certificates. Fortunately, kubeadm is intelligent enough to use preexisting CA certificates, and will skip the generation step if it does. All we need to do is to place our intermediate certificates into the given structure by calling Pipeline’s Secret API.

 1#### Gathering generated intermediate CA certificates from Pipeline
 2
 3$ mkdir -p /etc/kubernetes/pki/etcd/
 4$ cd /etc/kubernetes/pki/
 5$ certs=`curl -H "Authorization: Bearer $TOKEN" \
 6-H "Content-Type: application/json" \
 7http://localhost:9090/pipeline/api/v1/orgs/1/secrets/26a8dab0bff77fe1821477c0e0b7c09883419b136ae6a745e85a01af664d2db3`
 8$ echo "$certs" | jq -r .values.kubernetesCaCert > ca.crt
 9$ echo "$certs" | jq -r .values.kubernetesCaKey > ca.key
10$ echo "$certs" | jq -r .values.frontProxyCaCert > front-proxy-ca.crt
11$ echo "$certs" | jq -r .values.frontProxyCaKey > front-proxy-ca.key
12$ echo "$certs" | jq -r .values.etcdCaCert > etcd/ca.crt
13$ echo "$certs" | jq -r .values.etcdCaKey > etcd/ca.key
14$ unset certs
15
16#### This will result in the following directory structure:
17
18$ tree /etc/kubernetes/pki/
19/etc/kubernetes/pki/
20|-- ca.crt
21|-- ca.key
22|-- etcd
23|   |-- ca.crt
24|   `-- ca.key
25|-- front-proxy-ca.crt
26`-- front-proxy-ca.key

It is time to call kubeadm init and continue with the normal init workflow to setup a Kubernetes control plane on the first Kubernetes master node.

Future work 🔗︎

If you would like to have more control over your PKI infrastructure during the kubeadm bootstrap process, it’s advisable that you ‘bring your own’, which can be a difficult to generate and tedious maintain. However, with Pipeline this process is automated so that all of your kubeadm orchestrated clusters will have their own isolated PKI infrastructure, allowing you full control. In forthcoming blogposts we will describe how our PKE Kubernetes distribution is further configured during the provisioning process as managed by Pipeline.

About Banzai Cloud Pipeline 🔗︎

Banzai Cloud’s Pipeline provides a platform for enterprises to develop, deploy, and scale container-based applications. It leverages best-of-breed cloud components, such as Kubernetes, to create a highly productive, yet flexible environment for developers and operations teams alike. Strong security measures — multiple authentication backends, fine-grained authorization, dynamic secret management, automated secure communications between components using TLS, vulnerability scans, static code analysis, CI/CD, and so on — are default features of the Pipeline platform.