Kafka Connect is a tool that allows for the scalable and reliable streaming of data between Apache Kafka and other systems. Supertubes deploys Kafka Connect with Confluent’s Community Connectors included, and supports the use of schemas via Schema Registry.

Supertubes automates the deployment of Kafka Connect clusters and the creation of connectors through KafkaConnect custom resource instances and KafkaConnector custom resource instances, respectively.

Supertubes Kafka connect structure Supertubes Kafka connect structure

Kafka Connect API endpoints 🔗︎

You can access the deployed Kafka Connect clusters at the kafka-connect-svc-<kafka-connect-name>.<namespace>.svc:<servicePort> endpoint from within the Kubernetes cluster.

Security 🔗︎

Users that connect to Kafka Connect are authenticated using mTLS by default. You can disable this using the MTLS field in the KafkaConnect custom resource.

Kafka Connect and connector ACLs 🔗︎

Supertubes takes care of setting up all the ACLs necessary for Kafka Connect itself. However, you must set up the ACLs for the connectors separately. Typically:

  • A connector of the sink type requires READ access to the topics it reads, and to the consumer group named connect-{connector-name}.
  • A connector of the source type requires WRITE access to the topics it writes to.

A Kafka Connect cluster and the connectors it hosts are authenticated by the service account the Kafka Connect deployment runs with, so you must configure the connector-specific ACLs with that in mind.

Further information 🔗︎