Dynamic application security testing in Kubernetes

Banzai Cloud’s Pipeline platform is an operating system that allows enterprises to develop, deploy and scale container-based applications. It leverages best-of-breed cloud components, such as Kubernetes, to create a highly productive, yet flexible environment for developers and operations teams alike. Strong security - multiple authentication backends, fine grained authorization, Vault-based dynamic secret management, automated secure communications between components using TLS, vulnerability scans, static code analysis, pod and network policies etc. - are a tier zero feature of the Pipeline platform, which we strive to automate and enable for all enterprises.

We’re happy to announce that, as of today, we have opensourced a DAST operator and will soon be adding it to Pipeline.

To learn more about the different security aspects of the Pipeline platform, from our Vault operator and dynamic secret injection, to pod security policies, network policies, Dex integration, CIS benchmarks, unpriviledged image builds, vulnarability scans, Istio CNI plugin and more, please follow and read the posts marked with the security tag on our blog.

Motivation 🔗︎

At Banzai Cloud we strive to ensure that our applications are (security)tested from all aspects, to ensure the highest levels of security as well. In order to achieve the best results, we use a variety of methods for our security testing. The two most well-known approaches to security testing are SAST and DAST. The main difference between them is that SAST is a form of white box testing, while DAST is black. Static analysis (like searching dependencies for vulnerabilities) is relatively simple and can be easily integrated into CI workflows. Although examination of the source code may provide solid data about code security, dynamically testing an application during operations is crucial for determining its real security state.

In the world of containers, microservices, Kubernetes and distibuted teams, where applications change fast, the only way to ensure that all of the applications will be security scanned before the services are exposed to the public is to automate testing. Thus, we started to think and work on a solution which will help us reach this goal on Kubernetes.

Application security testing methodologies 🔗︎

SAST, or Static Application Security Testing, is also known as white box testing. It allows developers to find security vulnerabilities in application source code comparatively early in the software development life cycle. It also ensures conformance to coding guidelines and standards without actually executing any underlying code.

DAST, or Dynamic Application Security Testing, also known as black box testing, can find security vulnerabilities and weaknesses in running applications, typically web apps. It does that by employing fault injection techniques, such as feeding malicious data to an app, to identify common security vulnerabilities like SQL injection and cross-site scripting. DAST can also cast a spotlight on runtime problems that can’t be as easily identified by static analysis - for example, authentication and server configuration issues - as well as flaws that only become apparent once a known user logs in.

IAST, or Interactive Application Security Testing, provides the advantages of static analyzers because it can check the source code, and also the advantages of the dynamic analyzers, testing applications during runtime.

How we started to automate DAST 🔗︎

At Banzai Cloud we write lots of Kubernetes operators. Here are just a few of the popular ones:

• Istio operator
• Vault operator
• Koperator
• HPA operator
• Logging operator

Building on our experience of writing operators, we decided to build the DAST tool as a Kubernetes operator. Enter the dast-operator, which uses and leverages OWASP ZAP to perform automated application security testing on K8s.

The overall architecture looks like this:

The features currently available through the dast-operator 🔗︎

• Deploy an OWASP ZAProxy defined in a custom resource
• Scan external URLs defined in a custom resource
• Scan internal services based on their annotations
• Before deploying ingress, backend services checks that indicate what services have been scanned, and if those services are below the defined/accepted thresholds

Structure of the dast-operator 🔗︎

The dast-operator is running two reconcilers and one validating admission webhook to prevent vulnerable services becoming exposed.

The DAST reconciler is responsible for:

• watching DAST custom resources
• deploying ZAProxies
• initiating an analyzer job if the CR contains an external target

The Service reconciler is responsible for:

• watching services
• deploying an analyzer job if the service has the proper annotations

The Validating Webhook is responsible for:

• catching ingress creations if an ingress has the proper annotations
• extracting backend services
• checking their security testing results on the ZAProxy
• making decisions based on ingress annotations

Showtime!

Install the dast-operator 🔗︎

Clone dast-operator

git clone https://github.com/banzaicloud/dast-operator.git
cd dast-operator

Build docker images

make docker-build
make docker-analyzer

Deploy dast-operator

make deploy

Example 🔗︎

The following animated image shows how the example works:

Let’s see the DAST operator in action.

1: Deploy ZAProxy

First, we deploy ZAP to the zaproxy namespace by applying the DAST CustomResource.

kubectl create ns zaproxy
kubectl apply -f config/samples/security_v1alpha1_dast.yaml -n zaproxy

The content of the DAST CR:

apiVersion: security.banzaicloud.io/v1alpha1
kind: Dast
metadata:
  name: dast-sample
spec:
  zaproxy:
    name: dast-test
    apikey: abcd1234

2. Deploy an application and initiate an active scan

The ZAP is now deployed, so let’s create a test with a properly annotated service.

kubectl create ns test
kubectl apply -f config/samples/test_service.yaml -n test

The content of a test_secvice.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
      secscan: dast
  template:
    metadata:
      labels:
        app: nginx
        secscan: dast
    spec:
      containers:
      - name: nginx
        image: nginx:1.16.0-alpine
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: test-service
  annotations:
    dast.security.banzaicloud.io/zaproxy: "dast-test"
    dast.security.banzaicloud.io/zaproxy-namespace: "zaproxy"
spec:
  selector:
    app: nginx
    secscan: dast
  ports:
  - port: 80
    targetPort: 80

3. Test ingress validation

Once our test service has been scanned, let’s deploy an ingress with test-service backend.

kubectl apply -f config/samples/test_ingress.yaml -n test

Example ingress definition:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    dast.security.banzaicloud.io/medium: "2"
    dast.security.banzaicloud.io/low: "5"
    dast.security.banzaicloud.io/informational: "10"
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - http:
      paths:
      - path: /
        backend:
          serviceName: test-service
          servicePort: 80

The ingress deployment will succeed, because the scan results will be below the pre-defined thresholds that have already been set in annotations.

4. External URL scan test

Now we’ll initiate a security test targeted to an external URL by using a separated ZAP in an external namespace.

kubectl create ns external
kubectl apply -f config/samples/security_v1alpha1_dast_external.yaml -n external

Content of DAST CR

apiVersion: security.banzaicloud.io/v1alpha1
kind: Dast
metadata:
  name: dast-sample-external
spec:
  zaproxy:
    name: dast-test-external
    apikey: abcd1234
  analyzer:
    image: banzaicloud/dast-analyzer:latest
    name: external-test
    target: http://example.com

Current limitations: When using the webhook feature, it is only possible to successfully deploy the ingress when the backend service has already been scanned. If you’re deploying something with Helm that contains service and ingress definitions, the ingress deployment will fail, since the scan progress of the backend service won’t be finished.

The dast-operator roadmap 🔗︎

This is the first release of our dast-operator, however, it’s only the beginning. While the operator already automates the detection of many common mistakes, we don’t plan on stopping there. Our short term roadmap looks like this:

• API testing with JMeter and ZAP
• API security testing based on OpenAPI
• Parameterized security payload with fuzz
• Automated SQL injection testing using SQLmap

If you’d like to add your feature requests, or PR, or just star our GitHub repo, please feel free to visit the dast-operator repository. Thank you!

If you’d like to learn more about Banzai Cloud check out our other posts on this blog, the Pipeline, Hollowtrees and Bank-Vaults projects.