Declarative Kafka ACLs

Supertubes handles Kafka ACLs in Kubernetes by introducing a logical separation of ACL components:

KafkaResourceSelector,
KafkaRole and
KafkaACL.

These components describe:

What somebody can access,
When somebody has access to it, and
Who is receiving the permissions.

These reusable parts help maintain the system in the long term, and allow you to handle ACLs with a declarative approach - in line with the GitOps and the Configuration as Code trends of today - overcoming the difficulties inherent in handling ACLs in a Kubernetes environment.

Supertubes authenticates Kafka clients using their Kubernetes namespaces and service accounts.

The following figure shows how the components all come together.

KafkaACL structure

KafkaResourceSelectors - the What 🔗︎

KafkaResourceSelectors are filters for one or more Kafka Resources of the same type. These types can be any of the following:

topic for when you would like to apply them to Kafka topics
group for consumer groups
transactionalId when you want to select a single writer
and cluster when you want to impact the whole cluster

ResourceSelectors are a tool that help you to keep from repeating yourself every time you’d like to refer to a Kafka resource. They also provide a centralized place to modify and track all our resources handled by ACL. The following command shows which resources are covered by the ACLs.

kubectl get KafkaResourceSelector

KafkaResourceSelector example - topic 🔗︎

The following example selects the _schemas topic. Since the pattern is literal, there should be an exact match for the name field. You can also use pattern prefixed to suggest that the name field act as a prefix, creating even more versatile selectors by grouping together multiple topics. This is especially handy if you have a lot of smartly named topics, and you don’t want to create a selector for every single one of them.

apiVersion: kafka.banzaicloud.io/v1beta1
kind: KafkaResourceSelector
metadata:
  name: schemas-topic
  namespace: kafka
spec:
  type: topic
  name: _schemas
  pattern: literal

The following is identical to the previous example, but selects the __consumer_offsets topic.

apiVersion: kafka.banzaicloud.io/v1beta1
kind: KafkaResourceSelector
metadata:
  name: consumer-offsets-topic
  namespace: kafka
spec:
  type: topic
  name: __consumer_offsets
  pattern: literal

apiVersion: kafka.banzaicloud.io/v1beta1
kind: KafkaResourceSelector
metadata:
  name: schema-registry-group
  namespace: kafka
spec:
  type: group
  name: schema-registry
  pattern: literal

KafkaRoles - the When 🔗︎

Roles are standard part of access control systems. KafkaRoles provide a way to easily group multiple ACL permissions into one single reusable resource.

The following two KafkaRoles (consumer and producer) are deployed by default when you install Supertubes, meaning they’re ready to use, and you do not have to manually apply them to the cluster.

apiVersion: kafka.banzaicloud.io/v1beta1
kind: KafkaRole
metadata:
  name: consumer
spec:
  topic: # allow operations on topics
    allow:
    - read
    - describe
  group: # allow operations on consumer groups
    allow:
    - read

apiVersion: kafka.banzaicloud.io/v1beta1
kind: KafkaRole
metadata:
  name: producer
spec:
  topic:
    allow:
    - write
    - describe
    - create
  transactionalId:
    allow:
    - write
    - describe

You can also create your own custom groups of ACL operations. In the spec section you can specify your allow and deny permissions under the same four resource types that can be used in KafkaResourceSelectors:

topic
group
transactionalId
cluster

For example, to add a read operation to all producers, just add read under the producer role, and it will be automatically propagated to every KafkaACL CR that references it.

Note: Roles are not mandatory. They are a reusable tool to avoid copy-paste design patterns.

KafkaACL - the Who 🔗︎

The KafkaACL custom resource provides a binding between a subject, KafkaResourceSelector, and KafkaRole. Using this CR you can specify:

to which principal you want to apply the permissions defined in KafkaRole, and
which Kafka resource you’d like to grant access to through KafkaResourceSelector.

To use custom roles, you have to create them first.

Set the authorizer in your KafkaCluster CR:

readOnlyConfig: |
   authorizer.class.name=kafka.security.authorizer.AclAuthorizer
   allow.everyone.if.no.acl.found=false

Create KafkaResourceSelector CR.
Create the KafkaACL CR.
First, provide in the spec section the subject of the ACL.
```
kind: User
name: CN=<scheme-registry-namespace>-<schema-registry-service-account>
```
The value of kind can be User or anything else that the configured Kafka authorizer (see authorizer.class.name kafka config for details) supports.

The name should be the exact name of the service account you’d like to give the permissions to, in a format that depends on the location of the client:
- Kubernetes service accounts used by client applications running inside the mesh are represented as CN=<namespace>-<service-account-name>
- Kafka users created for client applications that are outside the mesh (regardless whether inside or outside of the Kubernetes cluster) based on the KafkaUser custom resource with certificates issued by cert-manager are represented as CN=<user-name-specified-in-the cr>,O=cert-manager
- If the certificate that identifies the Kafka user was issued by any other means, then use the value of the DN field of the certificate as the name.
Specify which KafkaCluster you’d like to bind the ACL to. Often there are multiple Kafka clusters on a single Kubernetes cluster, which helps you separate ACLs from each other.
```
clusterRef:
    name: kafka
    namespace: kafka
```

Define ACLs. The acls section lets you define inline permissions for your principal, the same way you can define them in KafkaRoles.

KafkaRoles are optional, and acls are the reason why:KafkaRoles are reusable sections of acls. acls are very useful, for example, in one-off operations, so you don’t need to create KafkaRoles with only one operation in them.

acls:
    - topic:
        operations:
          allow:
            - write
            - describe
            - describe_configs
        resourceSelectors:
          - name : schemas-topic
            namespace: kafka
    - topic:
        operations:
          allow:
            - describe
        resourceSelectors:
          - name : consumer-offsets-topic
            namespace: kafka
    - cluster:
        allow:
          - create

In resourceSelectors you can provide a list of selectors, telling the system What resource you are binding the operations to.

Note that the cluster part is without a resourceSelector, because the clusterRef at the top of the CR is being used as an anchor for the operation.

Specify optional parameters: reference and use our KafkaRoles CRs. Note that Roles are without a namespace, so you can reuse them across your other KafkaClusters.

  roles:
    - name: consumer
      resourceSelectors:
        - name: schemas-topic
          namespace: kafka
    - name: producer
      resourceSelectors:
        - name: schemas-topic
          namespace: kafka

Roles with resourceSelectors: [] are consider as cluster level acl definitions. There is no need to describe your cluster as a resource.

The entire KafkaACL CR used in the previous examples is the following:

apiVersion: kafka.banzaicloud.io/v1beta1
kind: KafkaACL
metadata:
  name: schema-registry
  namespace: kafka
spec:
  kind: User
      name: CN=<schame-registry-namespace>-<schema-registry-service-account>
  clusterRef:
    name: kafka
    namespace: kafka
  acls:
    - topic:
        operations:
          allow:
            - describe_configs
        resourceSelectors:
          - name : schemas-topic
            namespace: kafka
    - topic:
        operations:
          allow:
            - describe
        resourceSelectors:
          - name : consumer-offsets-topic
            namespace: kafka
    - cluster:
        allow:
          - create
  roles:
    - name: consumer
      resourceSelectors:
        - name: schemas-topic
          namespace: kafka
        - name: schema-registry-group
          namespace: kafka
    - name: producer
      resourceSelectors:
        - name: schemas-topic
          namespace: kafka