Using Minikube as part of your CI/CD flow

Suppose you’re working on a project which is running on Kubernetes - like we usually do - and you would like to test out this project on each and every pull request or commit. You can write many unit and integration tests, but at the end of the day, the proof of the pudding is in the eating. A real test would be to start up the application on the same platform where it will end up being deployed in production (in this case Kubernetes) and exercise it with some real workloads (aka end-to-end tests).

Creating and maintaining a Kubernetes cluster on cloud-providers for every single commit and pull request can be a costly and slow procedure, although Pipeline automates this process on six different cloud providers. Minikube comes to our rescue, a lightweight (and cheap) alternative of your production Kubernetes cluster. Minikube can run on your local machine allowing you to have a locally runnable acceptance test and can be used as part of your CI system as well.

Minikube on CircleCI 🔗︎

We are working on a lot of open-source projects where external contributors are sending PRs, for the sake example lets take one one of our popular projects, Bank-Vaults. We realized that we spent lots of time on testing PRs manually but this process could have been automated so we decided to setup Minikube in CircleCI and install our Vault-operator into it because the operator uses almost all components of the project. We have implemented our Minikube based testing for Bank-Vaults in two steps.

First we changed from the Docker based worker to a VM based worker and installed the required tools into it, see this pull-request for more details.

NOTE: The Docker based worker had to be replaced with a VM since CircleCI doesn’t provide a privileged Docker Daemon (which Minikube requires)

In the second pull-request we added the actual test case to the build, so it builds and installs the Vault operator and requests to provision two different flavors of Vault with it:

• one with a simple file based storage backend
• one with in collaboration with the etcd-operator using etcd as a storage backend

After this we use kurun to run a simple application which authenticates against Vault with the Kubernetes ServiceAccount it is running in, and queries the kv backend.

Here is an actual build running through all the steps: https://circleci.com/gh/banzaicloud/bank-vaults/796

Issues encountered and resolved 🔗︎

Issues that we found during implementing the tests:

• Operator SDK health check is broken

• Missing RBAC rules for the Operator

• kurun should compile static binaries

• Bank-Vaults client should wait with timeout

As you can see the investment has already paid off.

The interesting part from the configuration (extract) looks as follows:

 1version: 2
 2jobs:
 3  build:
 4    machine:
 5      image: circleci/classic:201808-01
 6      docker_layer_caching: true
 7
 8    environment:
 9      K8S_VERSION: v1.12.0
10      KUBECONFIG: /home/circleci/.kube/config
11      MINIKUBE_VERSION: v0.31.0
12      MINIKUBE_WANTUPDATENOTIFICATION: false
13      MINIKUBE_WANTREPORTERRORPROMPT: false
14      MINIKUBE_HOME: /home/circleci
15      CHANGE_MINIKUBE_NONE_USER: true
16      GOPATH: /home/circleci/go
17
18    working_directory: /home/circleci/go/src/github.com/banzaicloud/bank-vaults
19    steps:
20      - checkout
21      - run:
22          name: Build Docker images
23          command: |
24            make docker
25            make docker-operator
26            docker build -f Dockerfile.vault-env -t banzaicloud/vault-env:latest .
27            docker build -f Dockerfile.webhook -t banzaicloud/vault-secrets-webhook:latest .
28      - run:
29          name: Setup kubectl
30          command: |
31            curl -Lo kubectl https://storage.googleapis.com/kubernetes-release/release/${K8S_VERSION}/bin/linux/amd64/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/
32            mkdir -p ${HOME}/.kube
33            touch ${HOME}/.kube/config
34      - run:
35          name: Setup minikube
36          command: |
37            curl -Lo minikube https://github.com/kubernetes/minikube/releases/download/${MINIKUBE_VERSION}/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
38      - run:
39          name: Start minikube
40          command: |
41            sudo -E minikube start --vm-driver=none --cpus 2 --memory 4096 --kubernetes-version=${K8S_VERSION}
42      - run:
43          name: Operator acceptance test
44          command:
45            hack/acceptance-test.sh

To conclude, using Minikube as a lightweight alternative for real Kubernetes pays off quickly for testing. It needs some extra configuration in your preferred CI system but the integration of your application with Kubernetes will be tested continuously as well.

About Banzai Cloud Pipeline 🔗︎

Banzai Cloud’s Pipeline provides a platform for enterprises to develop, deploy, and scale container-based applications. It leverages best-of-breed cloud components, such as Kubernetes, to create a highly productive, yet flexible environment for developers and operations teams alike. Strong security measures — multiple authentication backends, fine-grained authorization, dynamic secret management, automated secure communications between components using TLS, vulnerability scans, static code analysis, CI/CD, and so on — are default features of the Pipeline platform.