Manage Kafka Connect clusters

Supertubes automates the deployment of Kafka Connect clusters and the creation of connectors through KafkaConnect custom resource instances and KafkaConnector custom resource instances, respectively.

You can manage these resources both imperatively and declaratively.

Imperative Kafka Connect cluster management 🔗︎

The Supertubes CLI provides commands that deploy Kafka Connect clusters, and connectors to these clusters. The easiest way to quickly set up a Kafka Connect cluster is to run the:

supertubes install -a

command which will set up a Kafka Connect cluster with default settings, linked to the provisioned demo Apache Kafka cluster and the deployed Schema Registry instance.

To create an additional Kafka Connect cluster, or update or delete an existing one, use the following command:

supertubes cluster kafka-connect [command]

These create and update sub-commands require KafkaConnect custom resource specifications as descriptors. For details on these commands, see supertubes cluster kafka-connect.

Declarative Kafka Connect cluster management 🔗︎

As we’ve already touched on briefly, a Kafka Connect cluster is represented by a KafkaConnect custom resource. Supertubes deploys the Kubernetes resources necessary to set up Kafka Connect clusters in accordance with the specifications of these custom resources. As a result, any Kafka Connect configuration changes made in a custom resource are automatically propagated to the corresponding Kafka Connect cluster.

apiVersion: kafka.banzaicloud.io/v1beta1
kind: KafkaConnect
metadata:
  name: my-kafka-connect
spec:
  clusterRef:
    # Name of the KafkaCluster custom resource that signifies the Kafka cluster this Kafka Connect instance will connect to
    name: kafka
  schemaRegistryRef:
    # (optional) Name of the SchemaRegistry custom resource that represents the Schema Registry to be made available for connectors
    name: kafka
  # The port Kafka Connect listens at for API requests (default: 8083)
  servicePort:
  # (optional) Annotations to be applied to Kafka Connect pods
  podAnnotations:
  # (optional) Annotations to be applied to the service that exposes Kafka Connect API on port `servicePort`
  serviceAnnotations:
  # (optional) Labels to be applied to Kafka Connect pods
  podLabels:
  # (optional) Labels to be applied to the service that exposes the Kafka Connect API on port `servicePort`
  serviceLabels:
  # (optional) Affinity settings for Kafka Connect pods https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity
  affinity:
  # Service account for Kafka Connect pods (default: default)
  serviceAccountName:
  # The compute resource requirements
  #  requests:
  #    cpu: 1
  #    memory: 2.2Gi
  #  limits:
  #    cpu: 2
  #    memory: 2.5Gi
  resources:
  # (optional) Node selector setting for Kafka Connect pods https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector
  nodeSelector:
  # (optional) Tolerations setting for Kafka Connect pods (https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/)
  tolerations:
  # (optional) Volume mounts for Kafka Connect pods (https://kubernetes.io/docs/concepts/storage/volumes/)
  volumeMounts:
  # (optional): Volumes for Kafka Connect pods (https://kubernetes.io/docs/concepts/storage/volumes/)
  volumes:
  # (optional): Jmx Exporter configuration
  jmxExporter:
  # Heap settings for Kafka Connect (default: -Xms512M -Xmx2048M)
  heapOpts:
  # Minimum number of replicas (default: 1)
  minReplicas:
  # Maximum number of replicas (default: 5)
  maxReplicas:
  # Controls whether mTLS is enforced between Kafka Connect and client applications (default: true)
  MTLS:
  # Defines the config values for Kafka Connect in key - value pairs format
  kafkaConnectConfig:
  # Additional environment variables to launch Kafka Connect with. The following list of environment variables CAN NOT be
  # set through this field as these are either set through a specific field (e.g. `HeapOpts`) or are dynamically computed: KAFKA_HEAP_OPTS, KAFKA_OPTS, CONNECT_REST_ADVERTISED_HOST_NAME,
  # CONNECT_BOOTSTRAP_SERVERS, CONNECT_GROUP_ID, CONNECT_CONFIG_STORAGE_TOPIC, CONNECT_OFFSET_STORAGE_TOPIC, CONNECT_STATUS_STORAGE_TOPIC,
  # CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL, CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL
  envVars:

Supertubes computes and maintains the following KafkaConnect configurations automatically. You cannot override these settings:

bootstrap.servers
group.id
config.storage.topic
offset.storage.topic
status.storage.topic
key.converter.schema.registry.url
value.converter.schema.registry.url

Further information 🔗︎

Kafka Connect and Connector examples