Client applications outside the Kubernetes cluster

This scenario covers using Kafka ACLs when your client applications are outside the Kubernetes cluster. As a result, the client applications must connect to the endpoint bound to the Kafka cluster’s external listener. In this scenario, the client applications must present a client certificate to authenticate themselves.

Using Kafka ACLs when your client applications are outside the Istio mesh

Prerequisites 🔗︎

To use Kafka ACLs with Istio mTLS, you need:

a Kubernetes cluster (version 1.15 and above), with
at least 8 vCPU and 12 GB of memory, and
with the capability to provision LoadBalancer Kubernetes services.
A Kafka cluster.

Steps 🔗︎

This procedure uses cert-manager to issue client certificates to represent the client application.

Enable ACLs and configure an external listener using Supertubes. Complete the following steps.

Verify that your deployed Kafka cluster is up and running:

supertubes cluster get --namespace <namespace-of-your-cluster> --kafka-cluster <name-of-your-kafka-cluster> --kubeconfig <path-to-kubeconfig-file>

Expected output:

Namespace  Name   State           Image                               Alerts  Cruise Control Topic Status  Rolling Upgrade Errors  Rolling Upgrade Last Success
kafka      kafka  ClusterRunning  banzaicloud/kafka:2.13-2.5.0-bzc.1  0       CruiseControlTopicReady      0

Enable ACLs and configure an external listener. The deployed Kafka cluster has no ACLs, and external access is disabled by default. Enable them by applying the following changes:

supertubes cluster update --namespace kafka --kafka-cluster kafka --kubeconfig <path-to-kubeconfig-file> -f -<<EOF
apiVersion: kafka.banzaicloud.io/v1beta1
kind: KafkaCluster
spec:
  ingressController: "istioingress"
  istioIngressConfig:
    gatewayConfig:
      mode: PASSTHROUGH
readOnlyConfig: |
    auto.create.topics.enable=false
    offsets.topic.replication.factor=2
    authorizer.class.name=kafka.security.authorizer.AclAuthorizer
    allow.everyone.if.no.acl.found=false
listenersConfig:
    externalListeners:
    - type: "plaintext"
        name: "external"
        externalStartingPort: 19090
        containerPort: 9094
EOF

The update in the previous step reconfigures the Kafka cluster to receive rolling updates. Verify that this is reflected in the state of the cluster.

supertubes cluster get --namespace kafka --kafka-cluster kafka --kubeconfig <path-to-kubeconfig-file>

Expected output:

Namespace  Name   State                    Image                               Alerts  Cruise Control Topic Status  Rolling Upgrade Errors  Rolling Upgrade Last Success
kafka      kafka  ClusterRollingUpgrading  banzaicloud/kafka:2.13-2.5.0-bzc.1  0       CruiseControlTopicReady      0

Wait until the reconfiguration is finished and the cluster is in the ClusterRunning state. This can take a while, as the rolling upgrade applies changes on a broker-by-broker basis.

Install and configure cert-manager for the client. Complete the following steps.

Connect to the cluster where your client application is running.

Install cert-manager on the cluster.

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v0.14.2/cert-manager.yaml

Specify a cluster issuer for cert-manager that has the same CA or root certificate as the Istio mesh, otherwise, the application’s client certificate won’t be valid for the mTLS enforced by Istio.

Get the CA certificate used by Istio:
```
kubectl get secrets -n istio-system istio-ca-secret -o yaml
```
This secret has different fields than what cert-manager expects.

Create a new secret from this in a format that works for cert-manager.

kubectl create -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: ca-key-pair
  namespace: cert-manager
data:
  tls.crt: <tls-crt-from-istio-ca-secret>
  tls.key: <your-tls-key-from-istio-ca-secret>
EOF

kubectl create -f - <<EOF
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: ca-issuer
  namespace: cert-manager
spec:
  ca:
    secretName: ca-key-pair
EOF

Create a Kafka user that the client application will use to identify itself. Grant this user access to the topics it needs.

kubectl create -f - <<EOF
apiVersion: kafka.banzaicloud.io/v1alpha1
kind: KafkaUser
metadata:
  name: external-kafkauser
  namespace: default
spec:
  clusterRef:
    name: kafka
    namespace: kafka
  secretName: external-kafkauser-secret
  pkiBackendSpec:
    pkiBackend: "cert-manager"
    issuerRef:
      name: "ca-issuer"
      kind: "ClusterIssuer"
  topicGrants:
    - topicName: example-topic
      accessType: read
    - topicName: example-topic
      accessType: write
EOF

Get the endpoint bound to the external listener of the Kafka cluster:

kubectl get svc -n kafka kafka-meshgateway

Example output:

NAME                TYPE           CLUSTER-IP     EXTERNAL-IP                                                                PORT(S)
kafka-meshgateway   LoadBalancer   10.10.44.209   aff4c6887766440238fb19c381779eae-1599690198.eu-north-1.elb.amazonaws.com   19090:32480/TCP,19091:30972/TCP,29092:30748/TCP

Export the client certificate stored in external-kafkauser-secret which represents Kafka user external-kafkauser, and the CA certificate of Istio:

kubectl get secret external-kafkauser-secret -o 'go-template={{index .data "ca.crt"}}' | base64 -D > /var/tmp/ca.crt
kubectl get secret external-kafkauser-secret -o 'go-template={{index .data "tls.crt"}}' | base64 -D > /var/tmp/tls.crt
kubectl get secret external-kafkauser-secret -o 'go-template={{index .data "tls.key"}}' | base64 -D > /var/tmp/tls.key

Use the exported client credentials and the CA certificate in your application to connect to the external listener of the Kafka cluster. (Otherwise, Istio automatically rejects the client application.) The following command is an example to connect with the kafkacat client application:

kafkacat -L -b aff4c6887766440238fb19c381779eae-1599690198.eu-north-1.elb.amazonaws.com:29092 -X security.protocol=SSL -X ssl.key.location=/var/tmp/tls.key -X ssl.certificate.location=/var/tmp/tls.crt -X ssl.ca.location=/var/tmp/ca.crt

Metadata for all topics (from broker -1: ssl://aff4c6887766440238fb19c381779eae-1599690198.eu-north-1.elb.amazonaws.com:29092/bootstrap):
 2 brokers:
  broker 0 at aff4c6887766440238fb19c381779eae-1599690198.eu-north-1.elb.amazonaws.com:19090 (controller)
  broker 1 at aff4c6887766440238fb19c381779eae-1599690198.eu-north-1.elb.amazonaws.com:19091
 1 topics:
  topic "example-topic" with 3 partitions:
    partition 0, leader 0, replicas: 0,1, isrs: 0,1
    partition 1, leader 1, replicas: 1,0, isrs: 0,1
    partition 2, leader 0, replicas: 0,1, isrs: 0,1