In this post, we’ll describe how to provision your Apache Kafka cluster using either Supertubes or the Koperator so that it is accessible by external client applications that run outside the Kubernetes cluster. There are two methods that allow you to externally publish your Kafka cluster: using LoadBalancer type services using NodePort type services The LoadBalancer method is a very convenient way to publish your Kafka cluster, as it allows you to forgo setting up a Load Balancer, provisioning public IPs, configuring routing rules, etc.

Ever since its inception, Supertubes has been about asking the question, “What if?” What if we built Kafka on the solid foundation of Istio and leveraged it and the Envoy proxy’s powerful networking capabilities to combine the two into something even more powerful? Today, along with releasing ksqlDB support for Supertubes, we’re engaging in even more speculation about what Istio and Kafka can do, together, to improve our lives. Over the years, Kafka has grown itself into a whole platform with an entire ecosystem built around it, consisting of components like Kafka Connect, Kafka Schema Registry, and Mirror Maker 2.

Hybrid and multi-cloud infrastructure is becoming more and more common in enterprise IT environments, since it provides a superior set of capabilities than what might otherwise be gained from a single cloud. Specifically, it allows us to use different cloud services for different use-cases within a single environment, helping to avoid vendor lock-in. Using both on-prem and cloud environments makes it possible to use the best of both worlds, letting users transition to the cloud at their own pace; it’s very rare for businesses of significant size, nowadays, to exist exclusively on either.

tl;dr On August 25th, one of our customers, who was using Supertubes to operate Kafka clusters on Kubernetes on Amazon, began to experience a service disruption in their Kubernetes cluster. Their previously configured Prometheus alerts notified the infra team what services were impacted. It turned out that Amazon was having some instance connectivity issues, in one of the London AZs. No data loss occurred thanks to the properly configured Kafka and Zookeeper clusters, which had both been distributed across multiple zones.

Leveraging a Cloud Native technology stack, Banzai Cloud Supertubes is the ultimate deployment tool for setting up and operating production-ready Kafka clusters on Kubernetes. While Supertubes installs and manages all the infrastructure components of a production-ready Kafka cluster on Kubernetes (like Zookeeper, Koperator , Envoy, etc) it also includes several convenience components, like Schema Registry, multiple disaster recovery options, Cruise Control and lots more. The list of Supertubes features is very long and covers everything you need to self host and run a production-ready Kafka cluster on Kubernetes.

tl;dr: The Supertubes approach to handling Kafka ACLs in Kubernetes provides a clearer way of seeing what’s actually happening by introducing a logical separation of ACL components under the names: KafkaACL, KafkaRole and KafkaResourceSelector. That way we get reusable parts that help maintain the system in the long term, allowing us to handle ACLs with a declarative approach, and overcoming the difficulties inherent in handling ACLs in a Kubernetes environment.

Running Kafka on Istio with mTLS is, in of itself, an interesting topic, but before we can talk about how Banzai Cloud’s Supertubes allows us to do that, let’s take a step back and look at how SSL works in Kafka. Maybe then we can answer the question, why do we need Kafka in Istio with mTLS at all? Supertubes is Banzai Cloud’s Kafka as a Service, which runs on Kubernetes inside an Istio service mesh.

It’s no news that for quite a while our Apache Kafka on Kubernetes take, Supertubes has been happily running inside an Istio-based service mesh, in both single or multi-cluster setups across hybrid clouds. While we have touched on several aspects of the advantages Istio gave us, this post’s aim is to collect some of the issues, cornerstones and benefits. We see the service mesh as a key component of every modern Cloud Native stack.

It’s been some time since we open sourced our Koperator , an operator designed from square one to take advantage of the full potential of Kafka on Kubernetes, and have built Supertubes on top to manage Kafka for our customers. One of the most requested enterprise feature has been the implementation of rolling upgrades. Built on the Banzai Cloud’s Koperator , Supertubes adds support and orchestrates these upgrades. Accordingly, in today’s blog we’re going to take a deep dive into the technical details of how the operator handles an update and how you can use Supertubes to seamlessly upgrade to the newest 2.

In one of the earlier Kafka disaster recovery on Kubernetes with CSI post we discussed how brokers of an Apache Kafka clusters can be backed up and restored leveraging Kubernetes volume snapshots. While this solution provides a good enough disaster recovery option (and a super quick recovery), it doesn’t help when the entire Kubernetes cluster hosting the Kafka cluster is lost. In this post, we show you how to: back up a Kafka cluster to a remote Kafka cluster running on a separate Kubernetes cluster, and also how to recover the lost Kafka cluster with Supertubes.

Apache Kafka was designed with a heavy emphasis on fault-tolerance and high-availability in mind, and thus provides different methods of ensuring enterprise-grade resiliency such as: replication factor - which defines how many partition replicas of a topic should be kept, each one being stored on a different broker. A replication factor of N allows for the loss of N-1 brokers. The new brokers will replace the lost brokers until the number of remaining broker(s) reaches the replication factor, or the number of copies of the partitions of a topic.

Apache Kafka is a distributed streaming platform used to build reliable, scalable and high-throughput real-time streaming systems. Its capabilities, while impressive, can be further improved through the addition of Kubernetes. Accordingly, we’ve built an open-source Koperator and Supertubes to run and seamlessly operate Apache Kafka on Kubernetes through its various features, like fine-grain broker configuration, metrics based scaling with rebalancing, rack awareness, and graceful rolling upgrades - just to name a few.