Supertubes automates the deployment of Kafka Connect clusters and the creation of connectors through KafkaConnect custom resource instances and KafkaConnector custom resource instances, respectively.
You can manage these resources both imperatively and declaratively.
Imperative Kafka Connect connector management 🔗︎
The Supertubes CLI provides commands that deploy Kafka Connect clusters, and connectors to these clusters. The easiest way to quickly set up a Kafka Connect cluster is to run the:
supertubes install -a
command which will set up a Kafka Connect cluster with default settings, linked to the provisioned demo Apache Kafka cluster and the deployed Schema Registry instance.
To create, update, or delete Kafka connectors, use the following command:
supertubes cluster kafka-connector [command]
These create and update sub-commands require KafkaConnector custom resource specifications as descriptors. For details on these commands, see supertubes cluster kafka-connector.
Declarative Kafka Connect connector management 🔗︎
The pattern for managing connectors is similar to that of Kafka Connect. There is a KafkaConnector custom resource that represents a connector type which is used by Supertubes to instantiate connectors on Kafka Connect clusters.
apiVersion: kafka.banzaicloud.io/v1beta1
kind: KafkaConnector
metadata:
name: my-connector
spec:
# The connector class name (e.g. io.confluent.connect.s3.S3SinkConnector)
class:
# The maximum number of tasks the Kafka Connector can create
tasksMax:
# References the Kafka Connect cluster where this connector is to run
clusterRef:
name: my-kafka-connect
# Defines the config values for Kafka Connector in key - value pairs format
config:
# Whether to pause or resume the connector execution
pause:
