Controlling egress traffic with Istio

Network perimeter security is a focal point of any network admin. When it comes to network perimeter control, our first thought is always inbound security (ingress). However, securing what can leave the network (egress) and where is equally important. In this post, we’re not going to go into the theoretical details of discussing why, exactly, controlling egress traffic is so important or where possible exploitations points are, because there are quite a few posts already.

Instead, we will present a working example of how to control egress traffic from specific source workloads to specific external services using Backyards (now Cisco Service Mesh Manager), our Istio distribution.

This blog post was inspired by Monzo’s excellent post: Controlling outbound traffic from Kubernetes. They’ve discussed how they control egress traffic from their Kubernetes platform. In our post, we aim to achieve the same goals, but from an Istio service mesh.

High level goals 🔗︎

First, let’s present the gist of what we would like to accomplish:

• We have the list of applications deployed in our cluster.
• We have a set of external hosts (including hostnames, ports, protcols, etc.) which we would like to access from the applications running inside the Istio service mesh.
• We explicitly define which applications can reach which external services.
• All other egress communication must be denied.

If these goals have been achieved, then we’ve already come a long way toward securing our network, as it implements a default-deny policy for the egress traffic on our cluster.

Use case 🔗︎

Let’s take an example use case, where we have two applications in the test-app namespace: app-a and app-b, and we want to achieve the following goals:

1. Deny access towards all external services from the test-app namespace by default.

2. Explicitly allow the following application → external service connections:

   • app-a → http(s)://github.com
   • app-a → http(s)://httpbin.org
   • app-b → http(s)://google.com

3. Ensure that all calls to external services go through dedicated egress gateways.

4. Support both HTTPS and plain HTTP calls.

To replicate this setup, all you need to do is configure a few Istio resources properly, like egress gateway deployments and services, Sidecars, Gateways, VirtualServices and ServiceEntrys.

I hope I haven’t missed anything. Sounds easy right?

Actually, it’s not that hard once you understand what’s going on. To help us do that, we’ve open-sourced a repository with an example application to demonstrate the setup. If you keep reading, we’ll go into detail about how Istio configurations work for this setup, or you can skip to the Try it out! section to see how it actually works.

Dirty details 🔗︎

You can find all the resources described in this section in this repo: https://github.com/banzaicloud/istio-external-demo

For better understanding, we defined groups of resources with separate functionalities in the repo. Let’s list these resource groups first:

• Common resources: Resources which apply to all applications in the test-app namespace by default
• Sample applications: The app-a and app-b sample applications
• External services: The list of external services which potentially can be reached from the cluster
• Allowed connections: The matrix to define which applications can connect to which external services

Now, let’s dig into the details of what’s being defined in each resource group exactly.

Common resources 🔗︎

The goal of this resource group is to make sure that all egress calls are denied by default from the test-app namespace.

Sidecar resource 🔗︎

• This resource sets outboundTrafficPolicy.mode to REGISTRY_ONLY for the test-app namespace, which means that calls to external services will be denied if there are no ServiceEntrys available for the sidecar proxy configuration.
• It also restricts the sidecar proxies to only receive outbound configurations for services defined in the test-app namespace.

Since there won’t be any ServiceEntrys defined in the test-app namespace, all external calls from this namespace will be denied with 502 responses from the Istio proxy sidecar containers. Furthermore, you can set up alerts for when forbidden access is detected based on istio proxy container access logs.

You can read more on the Sidecar resource and on accessing external services.

Sample applications 🔗︎

In this resource group, the sample app-a and app-b applications are defined.

The only important note about these resources is that they are simple Kubernetes Deployments and Services, there are no modifications in the resource descriptions or in the application codes. All the work to make sure which external calls will be allowed or denied are controlled by Istio resources.

External services 🔗︎

In this resource group, the accessible external services are listed: (github, google, and httpbin), along with the necessary routing, so that requests to them always go through dedicated egress gateways.

For each external services the following resources are defined in the external namespace:

MeshGateway resources (github, google, httpbin) 🔗︎

• The open source Banzai Cloud Istio operator creates an egress gateway deployment and service based on this resource and opens the service’s 80 and 443 ports.

There have been quite a few issues involving multiple gateway support in the Istio community, which is why we came up with our own solution in Banzai Cloud’s Istio operator. For more info on how to create multiple gateways, please refer to this blog post.

Note: If you wish to use this example without Banzai’s Istio operator, you need to create separate egress gateway deployments and services for yourself.

Gateway resources (github, google, httpbin) 🔗︎

• It configures listening ports (80, 443) on the matching egress gateway deployment.
• It sets tls.mode to ISTIO_MUTUAL to enforce mTLS connections for the application → egress gateway communications.

ServiceEntry resources (github, google, httpbin) 🔗︎

• It registers the given external service on the specified ports (80, 443) to Istio’s service registry.

Since these ServiceEntrys are in the external namespace and, as you might remember, we restricted applications in the test-app namespace to receive outbound configurations only from their own namespace, they won’t be able to connect to external hosts at this point.

VirtualService resources (github, google, httpbin) 🔗︎

• It routes all incoming traffic to egress gateways for the appropriate external service.

This way the egress gateway → external host directions are configured properly, what’s left is to allow the application → egress gateway connections for the chosen workloads.

Allowed connections 🔗︎

In this resource group, we finally whitelist which applications can access which external services.

Sidecar resources (app-a, app-b) 🔗︎

• It overrides the namespace-level Sidecar resource and allows us to receive outbound configurations for specific external services defined as ServiceEntrys in the external namespace.

This very important change makes it possible to configure access to external services on a workload-by-workload basis.

VirtualService resources (app-a, app-b) 🔗︎

• The application → egress gateway direction is defined in these VirtualServices for the specific application to make sure that when an external service is called, it is reached through the egress gateway.

This way all calls going through the sidecar proxies will go to dedicated egress gateways for each external host, hence the traffic can be monitored and controlled as usual with VirtualServices (for example, you can set traffic routing, traffic shifting, fault injection, and so on).

Try it out! 🔗︎

Setup 🔗︎

1. Create a Kubernetes cluster.

   If you need a hand with that, you can create a cluster with our free version of Banzai Cloud’s Pipeline platform.

2. Point KUBECONFIG at your cluster.

3. Install Istio with Backyards:

   Register for the free tier version of Cisco Service Mesh Manager (formerly called Banzai Cloud Backyards) and follow the Getting Started Guide for up-to-date instructions on the installation.

   In this demo, we install and use Backyards (now Cisco Service Mesh Manager), the Banzai Cloud Istio distribution, but the resources in this repo can be used with any Istio mesh.

   This step usually takes a few minutes, and installs Istio and Backyards on your cluster.

4. Create the test-app namespace for the example applications:

   kubectl create ns test-app
   

5. Enable sidecar-injection for the test-app namespace:

   backyards sidecar-proxy auto-inject on test-app
   

6. Make sure that the test-app namespace has the necessary istio-injection=enabled label. (If it is not there yet, wait a few seconds and check again.)

   kubectl get ns test-app --show-labels
   

7. Create external namespace for the external services. These will be accessible only for specific applications.

   kubectl create ns external
   

8. Create all the resources for this demo in the test-app and external namespaces with the following commands. These resources apply the rules described in the Dirty details section.

   git clone git@github.com:banzaicloud/istio-external-demo.git
   cd istio-external-demo
   kubectl apply --recursive -f resources/
   

Explore 🔗︎

1. Save the application pod names for easier access:

   APP_A_POD_NAME=$(kubectl get pods -n test-app -l k8s-app=app-a -o=jsonpath='{.items[0].metadata.name}')
   APP_B_POD_NAME=$(kubectl get pods -n test-app -l k8s-app=app-b -o=jsonpath='{.items[0].metadata.name}')
   

2. Test to make sure that app-a can access httpbin.org:

   kubectl exec -n=test-app -ti $APP_A_POD_NAME -- curl -Ls -o /dev/null -w "%{http_code}" httpbin.org
   

   Should be 200, as we specifically whitelisted app-a → http(s)://httpbin.org.

3. Check that traffic has passed through the egress gateway for the httpbin external service:

   kubectl -n=external logs $(kubectl get pods -n external -l app=egress-httpbin -o=jsonpath='{.items[0].metadata.name}') | tail -n 1
   [2020-04-28T08:07:36.714Z] "GET / HTTP/1.1" 200 - "-" "-" 0 9593 211 211 "10.20.1.179" "curl/7.47.0" "c50400c8-bf41-4c39-9956-ab61deded778" "httpbin.org" "34.230.193.231:80" outbound|80||httpbin.org 10.20.2.4:48588 10.20.2.4:80 10.20.1.179:34648 - -
   

4. Test to make sure that app-a can access github.com:

   kubectl exec -n=test-app -ti $APP_A_POD_NAME -- curl -Ls -o /dev/null -w "%{http_code}" github.com
   

   Should be 200, as we specifically whitelisted app-a → http(s)://github.com.

5. Now try to access cnn.com from app-a:

   kubectl exec -n=test-app -ti $APP_A_POD_NAME -- curl -Ls -o /dev/null -w "%{http_code}" cnn.com
   

   Should be 502, as it is not allowed explicitly.

6. Test to make sure that app-b can access google.com:

   kubectl exec -n=test-app -ti $APP_B_POD_NAME -- curl -Ls -o /dev/null -w "%{http_code}" google.com
   

   Should be 200, as we specifically whitelisted app-b → http(s)://google.com.

7. Now try to access cnn.com from app-b:

   kubectl exec -n=test-app -ti $APP_B_POD_NAME -- curl -Ls -o /dev/null -w "%{http_code}" cnn.com
   

   Should be 502, as it is not allowed explicitly.

8. Check the topology of your mesh on the Backyards dashboard:

   backyards dashboard
   

   If you created the traffic we just discussed, check the test-app and external namespaces in the TOPOLOGY view and you should see something like this:

   

   You can see basically the same thing on the Backyards UI, what we described as “use case architecture” earlier.

   There is a feature gap between the Mixer-based and Mixerless telemetry in Istio, which affects the egress telemetry as well. Backyards comes with Mixerless Telemetry V2 by default, so for the sake of this screenshot I switched back to Mixer-based telemetry to gather all necessary egress data.

Cleanup 🔗︎

1. Delete the demo resources:

   kubectl delete --recursive -f resources/
   kubectl delete ns test-app
   kubectl delete ns external
   

2. Remove Backyards and Istio from your cluster:

   backyards uninstall -a
   

Automating egress control 🔗︎

Imagine having a list of your applications and a list of external services you want to reach from your applications, and then simply selecting which connections are allowed (similar to what we envisioned in the High level goals section). You could use a UI or a CLI, or a declarative method if you prefer, for exactly that: all you’d need is the two lists and to determine what connections should be allowed between them. All the necessary resources would be automatically created and maintained in the background. That’d be nice, right?

Well, this is precisely what one of the new features in an upcoming Backyards (now Cisco Service Mesh Manager) release will be!

Until then, make sure to check out the Backyards (now Cisco Service Mesh Manager) 1.3 release (it’s just around the corner) which will already bring some advanced ingress/egress features (among many others). Stay tuned for a release blog post soon!

Outlook 🔗︎

Obviously, this example use case did not cover everything an organization might have in mind in regards to egress traffic. These were simply outside the scope of this blog post, but, before we go, I’d like to take the time to mention a few additional considerations:

• Security: Going through sidecar proxies we restricted connections so that they go through dedicated egress gateways, but what if sidecar proxies are hacked and call external services directly? Or what about connections from pods without a sidecar proxy container running in them? To block these calls as well, consider additional security aspects like using firewalls or Kubernetes Network Policies.
• TCP traffic: We’ve only covered HTTP traffic in this post, but it is also possible to use TCP connections.
• Authorization Policies: Authorization Policy enables access control on workloads in the mesh, but they do not work on egress gateways as of Istio 1.5.2, which was a deal-breaker for our use case; we did not use them.

Takeaway 🔗︎

We showed through a working example how you can fine-tune which workloads can access which external services from an Istio mesh.

Backyards (now Cisco Service Mesh Manager) can help you achieve these goals (and much more) with its advanced control and monitoring capabilities.

We see the service mesh as a key component of every modern Cloud Native stack. To make this a reality, we are on a mission to make Istio simple to use and manage for everyone. We have built a product called Backyards (now Cisco Service Mesh Manager), the Banzai Cloud operationalized and automated service mesh, which makes setting up and operating an Istio-based mesh a cinch.