Updating the Kafka distribution with Supertubes

Updating the distribution in an Apache Kafka cluster is one of the most challenging actions a cluster operator has to face.

Since Supertubes can deploy and manage Kafka clusters that use custom Kafka distributions, the Supertubes CLI has built-in support for updating the distribution as well. This section guides you through the steps you need to take in order to update the images of the Kafka broker in your cluster.

There are two possible scenarios:

• upgrading the Kafka version to a new version (for example, 2.6.0 -> 2.7.0)
• the Kafka version does not change or is downgraded (for example, 2.6.0 -> 2.5.1)

Note: Follow to the upgrade section if the version of any broker will be increased during the update (in case of clusters where the versions are inhomogeneous).

In case there’s any ambiguity in the Kafka-related instructions of this manual, check the official Kafka documentation.

Upgrading steps 🔗︎

The upgrade of the Kafka brokers happens in two phases:

1. Rolling upgrade of brokers with the new image and pinned inter protocol version
2. Rolling upgrade of brokers removing the pinned inter protocol version

Finishing the first phase results a fully functional cluster with the desired Kafka version. However, the protocol version is still set to the previous version, therefore certain functions/features might not be available until the second phase is completed.

The CLI refers to the second step as the post-update step.

Update the distribution from the CLI 🔗︎

Both upgrade and downgrade works with the Supertubes CLI by running the supertubes cluster update CLI command. For details, see the examples below, or check the command line reference or check out the examples below.

You will be prompted by the CLI to provide the version of the new image if a custom image is used. This allows the CLI to determine if performing an upgrade is needed or not.

Upgrade 🔗︎

You can upgrade the Kafka cluster by providing the newer version of the Kafka broker image to the supertubes cluster update command.

1. Create a backup of your KafkaCluster CR (for example, into the kafkacluster.yaml file). You will need the CR for the cluster update command.

2. Start the upgrade by running the supertubes cluster update command and monitor your cluster during the operation.

   • If you specify the --wait option, the CLI will also monitor the operation.
   • If you specify both the --wait and --post-update options, the CLI automatically removes the pinned inter protocol version after the first rolling upgrade has been finished.

3. (Optional) Resolve post update steps either manually or using the CLI.

For example, to upgrade your kafka cluster in the kafka namespace to the official Banzai Cloud image with 2.7.0 Kafka brokers (and automatically resolve the post-update steps), you can use the following command:

$ supertubes cluster update --namespace kafka --kafka-cluster kafka --cluster-config-file kafkacluster.yaml --kafka-image ghcr.io/banzaicloud/kafka:2.13-2.7.0-bzc.2 --wait --post-update

Downgrade 🔗︎

In case of a downgrade, or when the Kafka versions of the brokers do not change, you don’t need post-update steps.

1. Create a backup of your KafkaCluster CR (for example, into the kafkacluster.yaml file). You will need the CR for the cluster update command.
2. Start the upgrade and monitor your cluster during the operation. If you specify the --wait option, the CLI will also monitor the operation.

For example, to update your cluster kafka cluster in the kafka namespace using the <custom-image> and the kafkacluster.yaml backup you created at Step 1, you can use the following command:

$ supertubes cluster update --namespace kafka --kafka-cluster kafka --cluster-config-file kafkacluster.yaml --kafka-image <custom-image>

Update the distribution manually 🔗︎

In some cases, you need to update the Kafka distribution without using the CLI. For example, if you want to upgrade a cluster that has log.message.format.version configured.

To manually update your Kafka distribution without using the Supertubes CLI, complete the following steps.

1. Create a backup of your KafkaCluster CR.

2. Choose a minimal Kafka protocol version that every broker understands. Choose the lowest of:

   • The lower Kafka version of the current and the new version (to be upgraded). Depending on whether this is a downgrade or an upgrade, it will be the new or the current one.
   • If there are multiple brokers with different Kafka versions, choose the lowest version.
   • If there are any overrides of the inter.broker.protocol.version configuration on the broker or broker config level, choose the lowest version.

3. Update the KafkaCluster custom resource with the following changes:

   • Set the new image as spec.clusterImage and remove any broker or broker config level overrides.
   • Set the inter.broker.protocol.version to the minimal protocol version, and remove any overrides. Use only major.minor version, for example, 2.6
   • If any broker has the log.message.format.version configured, set that to the lowest version. Use only major.minor version, for example, 2.6

     CAUTION:

     If the inter.broker.protocol.version is not the lowest of all possible Kafka broker versions, the cluster might become unusable, and the rolling upgrade process can stuck. Make sure to calculate it correctly.
   • If the new image is a custom image (not an official Banzai Cloud image), add annotations to the broker. For details, see Managing custom Kafka distribution with Supertubes.

4. Wait for the update to finish.

5. (Optional) If the update was an upgrade, remove the inter.broker.protocol.version (and log.message.format.version if overridden) configuration from the KafkaCluster custom resource. This will trigger another rolling upgrade - wait until it has finished.