Running TiDB on Kubernetes

At Banzai Cloud we provision different applications or frameworks to Pipeline, the PaaS we built on Kubernetes. We practice what we preach, and our PaaS’ control plane also runs on Kubernetes and requires a layer of data storage. It was therefore necessary that we explore two different use cases: how to deploy and to run a distributed, scalable and fully SQL compliant DB to cover our client’s, and our own, internal needs. Additionally, most of the legacy or Java Enterprise Edition applications we provision to the Pipeline platform require a database backend. While it may be true that we (currently) support only two wire protocols, this post focuses on how we run/deploy, operate, autoscale and monitor TiDB - a mysql wire protocol-based database.

TiDB 🔗︎

TiDB (pronounced: /‘taɪdiːbi:/ tai-D-B, etymology: titanium) is a Hybrid Transactional/Analytical Processing (HTAP) database. Inspired by the design of Google F1 and Google Spanner, TiDB features infinite horizontal scalability, strong consistency, and high availability. The goal of TiDB is to serve as a one-stop solution for online transactions and analysis.

tl;dr: 🔗︎

We deploy, run, scale and monitor TiDB on Kubernetes. We love TiDB’s architecture and the separation of concerns inherent in its building blocks, which perfectly suits k8s.

$ helm repo add banzaicloud-incubator http://kubernetes-charts-incubator.banzaicloud.com
$ helm repo update
$ helm install banzaicloud-incubator/tidb

Introduction 🔗︎

This chart bootstraps a TiDB deployment on a Kubernetes cluster using the Helm package manager.

Prerequisites 🔗︎

• Kubernetes 1.7+ with Beta APIs enabled
• PV provisioner support in the underlying infrastructure

Installing the Chart 🔗︎

To install the chart with the release name my-release:

$ helm install --name my-release banzaicloud-incubator/tidb

It deploys TiDB to the Kubernetes cluster with the its default configuration. The configuration section lists the parameters that can be configured during installation.

Uninstalling the Chart 🔗︎

To uninstall/delete the my-release deployment:

$ helm delete my-release

The above command removes all Kubernetes components associated with the chart and deletes the release.

Configuration 🔗︎

The following table lists the configurable parameters of the TiDB chart and their default values.

Specify each parameter using the --set key=value[,key=value] argument to helm install.

Alternatively, a .yaml file that specifies the values for these parameters may be provided during the chart’s installation. For example:

$ helm install --name my-release -f values.yaml banzaicloud-incubator/tidb

Tip: You can use the default values.yaml

Persistence 🔗︎

The chart mounts a Persistent Volume to a given location. By default, the volume is created using dynamic volume provisioning. An existing PersistentVolumeClaim can be defined thusly:

Existing PersistentVolumeClaims 🔗︎

1. Create the PersistentVolume
2. Create the PersistentVolumeClaim
3. Install the chart

$ helm install --set persistence.existingClaim=PVC_NAME banzaicloud-incubator/tidb

What’s next 🔗︎

This posts highlights how easy it is to use TiDB on Kubernetes through Helm. Obviously, on Pipeline we do things differently, and the cluster and the deployment is provisioned either through the REST API.

Remark 1: Currently we use our own Helm charts, but we’ve noticed that PingCAP is already working on a TiDB operator - once that’s released or the source code is made available we’ll reconsider this approach. We love Kubernetes operators and have written/use quite a few, so we look forward to getting our hands on a new one.

Remark 2: In the event of PD node/StatefulSet failure Pipeline auto-recovers, however, due to an issue with the PD’s internal etcd re-joining, that recovery may not be successful.