Banzai Cloud Supertubes (Supertubes) is the deployment tool for setting up and operating production-ready Kafka clusters on Kubernetes, leveraging a Cloud Native technology stack. Supertubes includes Zookeeper, the Banzai Cloud Kafka operator, Envoy, and many other components hosted in a managed service mesh. All components are automatically installed, configured, and managed in order to operate a production-ready Kafka cluster on Kubernetes.

Key features 🔗︎

Some of the key features of Supertubes are:

  • Fine-grained broker configuration support for heterogeneous cluster layouts.
  • Declarative topic and user management through custom resources (CRs).
  • Automatic, mTLS-based encrypted and authenticated communication between all Supertubes components.
  • Advanced Grafana dashboards to monitor all Supertubes components.
  • Automatic reaction and self-healing based on Prometheus alerts.
  • Alert-based reactions for graceful up and downscaling and adding volumes to brokers.
  • Disaster recovery using volume snapshots and cross-cluster replication using MirrorMaker2.
  • Rolling upgrades for continuous operations.

The Banzai Cloud Kafka operator is a core part of Banzai Cloud Supertubes that helps you create production-ready Kafka cluster on Kubernetes, with scaling, rebalancing, and alerts based self healing. While the Kafka operator itself is an open-source project, the Banzai Cloud Supertubes product extends the functionality of the Kafka operator with commercial features (for example, declarative ACL handling, built-in monitoring, and multiple ways of disaster recovery). Read a detailed comparison of Supertubes and the Kafka operator.

Architecture 🔗︎

Supertubes architecture

What makes Supertubes unique? 🔗︎

Banzai Cloud Supertubes is specifically built to run and manage Apache Kafka on Kubernetes. Other solutions use Kubernetes StatefulSets to run Kafka, but this approach is not really suitable for Kafka. Supertubes is based on simple Kubernetes resources (Pods, ConfigMaps, and PersistentVolumeClaims), allowing a much more flexible approach that makes possible to:

  • Modify the configuration of unique Brokers: you can configure every Broker individually.
  • Remove specific Brokers from clusters.
  • Use multiple Persistent Volumes for each Broker.
  • Do rolling upgrades without data loss or service disruption.

On-prem, multi-cloud, and hybrid-cloud support 🔗︎

Supertubes allows you to run large Kafka clusters not only in the cloud, but also on-premises, multi-cloud, and hybrid-cloud environments. Running Kafka over Istio, and to automate the creation of Kafka clusters across single-cloud multi AZ, multi-cloud and especially hybrid-cloud environments.

Kafka in on-prem, multi-cloud, and hybrid-cloud environments

Running Kafka over Istio 🔗︎

Supertubes makes running Kafka over Istio extremely easy, bringing additional security benefits, scalability and durability, locality based load balancing, and lots more makes it a perfect choice.

Kafka protocol support in Envoy 🔗︎

Envoy is a next generation network proxy, built for the cloud native era. It supports a wide variety of application protocols, including Kafka. The benefits of a network proxy understanding higher level protocol implementations are huge. In case of Kafka, the list of benefits include:

  • Out of the box tracing and monitoring within a Kafka mesh
  • Consumer group metrics
  • Information about apps and their version of the client libraries
  • Request validation
  • Protocol version translations
  • Automatic topic name conversions without having to modify the clients
  • Mirroring topics to another clusters (we run many hybrid Kubernetes clusters)
  • Functional parity across runtimes

External Access via LoadBalancer 🔗︎

Supertubes externalizes access to Kafka using a single LoadBalancer, so there’s no need for a LoadBalancer for each Broker.

Kafka External

Event-based scaling and self-healing 🔗︎

Supertubes exposes Cruise-Control and Kafka JMX metrics to Prometheus, and acts as a Prometheus Alert Manager. It receives alerts defined in Prometheus, and creates actions based on Prometheus alert annotations, so it can handle and react to alerts automatically, without having to involve human operators.

Graceful Kafka Cluster Scaling 🔗︎

To scale Kafka clusters both up and down gracefully, Supertubes integrates LinkedIn’s Cruise Control, and is configured to react to events. The three default actions are:

  • upscale cluster (add a new Broker)
  • downscale cluster (remove a Broker)
  • add additional disk to a Broker

You can also define custom actions.

Kafka-operator alerts

Vertical capacity scaling 🔗︎

There are many situations in which the horizontal scaling of a cluster is impossible. When only one Broker is throttling and needs more CPU or requires additional disks (because it handles the most partitions), a StatefulSet-based solution is useless, since it does not distinguishes between replicas’ specifications. The handling of such a case requires unique Broker configurations. If we need to add a new disk to a unique Broker, with a StatefulSet-based solution, we waste a lot of disk space (and money). Since it can’t add a disk to a specific Broker, the StatefulSet adds one to each replica.

With Supertubes, adding a new disk to any Broker is as easy as changing a CR configuration. Similarly, any Broker-specific configuration can be done on a Broker by Broker basis.

SSL-encrypted communication 🔗︎

Supertubes fully automates managed mutual TLS (mTLS) encryption and authentication. You don’t need to configure your brokers to use SSL, as Supertubes provides mTLS out-of-the box at the network layer (implemented through a light-weight, managed Istio mesh). All services deployed by Supertubes (Zookeeper, Kafka operator, the Kafka cluster, Cruise Control, MirrorMaker2, and so on) interact with each other using mTLS.

Kafka SSL

  • If the client application is included into the managed light-weight Istio mesh used by Supertubes than the client application will interact with Kafka over mTLS out of the box, no SSL configuration is needed at the client application.
  • If the client application runs outside the managed mesh, the client application needs to be configured to use a client certificate that was signed by the same CA as what the mesh is using. Such a client certificate can be issued by executing supertubes istio certificate generate-client-certificate. Note that some additional configuration is needed in this scenario which is use case specific.
  • If the client application runs outside the Kubernetes cluster that hosts the Kafka cluster, the scenario is essentially the same as when the client application runs outside the managed mesh. The only difference is that the client application must connect to the external endpoint of the Kafka cluster instead of the internal endpoint which is only reachable from within the Kubernetes cluster.

Disaster recovery 🔗︎

CSI 🔗︎

Kafka disaster recovery on Kubernetes with CSI allows you to backup and restore your clusters using Kubernetes volume snapshots. While this solution provides a good enough disaster recovery option (and a super quick recovery), it doesn’t help when the entire Kubernetes cluster hosting the Kafka cluster is lost.

MM2 🔗︎

MirrorMaker2 leverages the Connect framework to replicate topics between Kafka clusters, and has the following benefits:

  • Both topics and consumer groups are replicated.
  • Topic configuration and ACLs are replicated.
  • Cross-cluster offsets are synchronized.
  • Partitioning is preserved.

Supertubes is using MirrorMaker2 to set up cross-cluster replication between remote Kafka clusters and recover a lost Kafka cluster from a remote Kafka cluster. It deploys a MirrorMaker2 instance for each Kafka cluster into the same namespace where the Kafka cluster resides.

Cross-cluster replication

Read more about disaster recovery with Supertubes

Kafka operator, Supertubes Core, and Supertubes Pro 🔗︎

Our solution to run Apache Kafka on Kubernetes comes in three flavours:

  • The Banzai Cloud Kafka operator is an open source project that delivers the basic functionality of our solution.
  • Supertubes Core adds commercial support and optionally integration support to the Kafka operator.
  • Supertubes Pro is a commercial product that includes all the features mentioned in this guide, commercial support, and optionally integration support.

Read the detailed comparison.

Next steps 🔗︎