Istio ingress and egress gateways

In today’s blogpost we’re going to be discussing ingress and egress gateways. First, we’ll cover the basics, then we’ll go into detail and explore how they work through a series of practical examples. Ingress and egress gateways are load balancers that operate at the edges of any network receiving incoming or outgoing HTTP/TCP connections. Ingress gateways make it possible to define an entry points into an Istio mesh for all incoming traffic to flow through. Egress gateways are similar: they define exit points from the mesh, but also allow for the application of Istio features to the traffic exiting the mesh. Some examples of these features are monitoring, routing rules and retries.

In Istio, both gateways are based on Envoy. Envoy handles reverse proxying and load balancing for services running inside a service mesh’s network, and also for external services outside the mesh. An Istio gateway in a Kubernetes cluster consists of, at minimum, a Deployment and a Service. For an ingress gateway the latter is typically a LoadBalancer-type service, or, when an ingress gateway is used solely within a cluster, a ClusterIP-type service. For an egress gateway the service type is almost always ClusterIP.

Although Istio can be configured to support Kubernetes Ingress Resources, a better approach would be to use Istio’s custom resources (Gateway, VirtualService). That way you can use Istio features for more than internal services, including ingresses, giving you access to way more features than you’d have with just Kubernetes’ Ingress Resources.

The Gateway resource describes the port configuration of the gateway deployment that operates at the edge of the mesh and receives incoming or outgoing HTTP/TCP connections. The specification describes a set of ports that should be exposed, the type of protocol to use, TLS configuration – if any – of the exposed ports, and so on. For more information about Gateways, see the Istio documentation.

VirtualService defines a set of traffic routing rules to apply when a host is addressed. Each routing rule defines matching criteria for the traffic of a specific protocol. If the traffic matches a routing rule, then it is sent to a named destination service defined in the registry. For example, it can route requests to different versions of a service or to a completely different service than was requested. Requests can be routed based on the request source and destination, HTTP paths and header fields, and weights associated with individual service versions.

For more information about VirtualServices, see the Istio documentation.

Istio Ingress and Egress Gateways, showtime 🔗︎

Our only prerequisite before exploring these concepts through examples is the creation of a Kubernetes cluster.

You can create a Kubernetes cluster on five different cloud providers, or on-premise via the free developer version of the Pipeline platform. But you can also bring your own cluster.

Install Istio using Backyards 🔗︎

The easiest way to install a production ready Istio and a demo application on a brand new cluster is to use the Backyards CLI.

Register for the free version and install the CLI.

You can read more about the latest Backyards release here

Using the main ingress gateway 🔗︎

Remember, as we talked about earlier in this post, ingress gateways enable us to expose services to the external world. Accordingly, an ingress gateway serves as the entry point for all services running within the mesh.

Determining the ingress IP 🔗︎

In order to expose a service, you must first know the external IP of the ingress gateway. Fortunately, the Banzai Cloud Istio operator helps us with this.

❯ kubectl -n istio-system get istio mesh

NAME   STATUS      ERROR   GATEWAYS                        AGE
mesh   Available           [18.184.240.108 18.196.72.62]   15m

In general, you should manually set an external hostname that points to these addresses, but for demo purposes you can use xip.io, which is a domain name that provides wildcard DNS for any IP address.

Expose a service through the ingress gateway 🔗︎

The demo application that comes with Backyards (now Cisco Service Mesh Manager) contains several microservices. The frontpage service serves as the entry point of that application. Now, let’s create a Gateway and a VirtualService resource to expose the frontpage service.

If you’re using xip.io, the external hostname for the service is going to be either frontpage.18.184.240.108.xip.io or frontpage.18.196.72.62.xip.io.

Create Gateway resource 🔗︎

The following Gateway resource configures listening ports on the matching gateway deployment.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: frontpage
  namespace: backyards-demo
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "frontpage.18.184.240.108.xip.io"
    - "frontpage.18.196.72.62.xip.io"

Create VirtualService resource 🔗︎

The following VirtualService resource configures routing for the external hosts within the mesh.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: frontpage
  namespace: backyards-demo
spec:
  hosts:
  - "frontpage.18.184.240.108.xip.io"
  - "frontpage.18.196.72.62.xip.io"
  gateways:
  - frontpage
  http:
  - route:
    - destination:
        port:
          number: 8080
        host: frontpage.backyards-demo.svc.cluster.local

Access the service on the external address 🔗︎

Try to access the service on the external address you just configured, on host frontpage.18.184.240.108.xip.io. If everything is set correctly, the following command will return an HTTP 200 status code.

❯ curl -i frontpage.18.184.240.108.xip.io

HTTP/1.1 200 OK
content-type: text/plain
date: Sun, 02 Feb 2020 19:15:14 GMT
content-length: 9
x-envoy-upstream-service-time: 16
server: istio-envoy

frontpage

Create and use multiple ingress gateways 🔗︎

Having one ingress and egress gateway to handle incoming and outgoing traffic from the mesh is part of a basic Istio installation and has been supported by the Banzai Cloud Istio operator from day one, but in large enterprise deployments our customers typically use Backyards (now Cisco Service Mesh Manager) with multiple ingress or egress gateways.

The Banzai Cloud Istio operator and multiple gateways 🔗︎

The Banzai Cloud Istio operator has an Istio custom resource that defines mesh configurations. The main ingress/egress gateways are part of the specifications of that resource.

spec:
  ...
  gateways:
    egress:
      enabled: true
      maxReplicas: 1
      minReplicas: 1
      ports:
      - name: http2
        port: 80
        protocol: TCP
        targetPort: 80
      - name: https
        port: 443
        protocol: TCP
        targetPort: 443
      replicaCount: 1
      sds:
        enabled: false
        image: docker.io/istio/node-agent-k8s:1.4.3
      serviceType: ClusterIP
    enabled: true
    ingress:
      enabled: true
      maxReplicas: 1
      minReplicas: 1
      ports:
      - name: status-port
        port: 15020
        protocol: TCP
        targetPort: 15020
      - name: http2
        port: 80
        protocol: TCP
        targetPort: 80
      - name: https
        port: 443
        protocol: TCP
        targetPort: 443
      - name: tls
        port: 15443
        protocol: TCP
        targetPort: 15443
      replicaCount: 1
      sds:
        enabled: false
        image: docker.io/istio/node-agent-k8s:1.4.3
      serviceType: LoadBalancer
    ...

One way to support multiple gateways would have been to add support for specifying them in the existing custom resource. But we chose a radically different approach for the following reasons:

• a separate controller should reconcile gateways, as there could be multiple gateways in multiple namespaces
• RBAC: having a separate CR allows us to properly control who can manage gateways, without having permissions to modify other parts of the Istio mesh configuration

Thus, we have added a new CRD to the Banzai Cloud Istio operator, called the MeshGateway, that can be used to add and configure a new Istio ingress or egress gateway into the mesh. When you create a new MeshGateway CR, the Banzai Cloud Istio operator will take care of configuring and reconciling the necessary resources, including the Envoy deployment and its related Kubernetes service.

Create a new service in the default namespace and expose it 🔗︎

To demonstrate how to create and use multiple ingress gateways, let’s add a simple service to the default namespace.

❯ kubectl apply -f https://raw.githubusercontent.com/banzaicloud/istio-operator/release-1.4/docs/federation/multimesh/echo-service.yaml

deployment.extensions/echo created
service/echo created

It would be possible to expose this echo service through the existing ingress gateway, similar to the way we would for the frontpage service, but let’s assume we need to expose this service on port 8000, without modifying the existing ingress gateway.

Create a new ingress gateway using the MeshGateway resource 🔗︎

Apply the following resource and the operator will create a new ingress gateway deployment, and a corresponding service.

apiVersion: istio.banzaicloud.io/v1beta1
kind: MeshGateway
metadata:
  name: echo-ingress
  namespace: default
spec:
  maxReplicas: 1
  minReplicas: 1
  ports:
  - name: http
    port: 8000
    protocol: TCP
    targetPort: 8000
  serviceType: LoadBalancer
  type: ingress

The MeshGateway resource automatically labels the created Service and Deployment resources with the gateway-name and gateway-type labels and their corresponding values.

Get the echo ingress gateway IP 🔗︎

❯ kubectl -n default get meshgateways echo-ingress

NAME           TYPE      SERVICE TYPE   STATUS      INGRESS IPS                     ERROR   AGE
echo-ingress   ingress   LoadBalancer   Available   [18.185.173.38 18.197.110.20]           29m

Create Gateway and VirtualService resources 🔗︎

Just like in the first example, the following Gateway and VirtualService resources are necessary to configure listening ports on the matching gateway deployment.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: echo
  namespace: default
spec:
  selector:
    gateway-name: echo-ingress
    gateway-type: ingress
  servers:
  - port:
      number: 8000
      name: http
      protocol: HTTP
    hosts:
    - "echo.18.197.110.20.xip.io"
    - "echo.18.185.173.38.xip.io"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: echo
  namespace: default
spec:
  hosts:
  - "echo.18.197.110.20.xip.io"
  - "echo.18.185.173.38.xip.io"
  gateways:
  - echo
  http:
  - route:
    - destination:
        port:
          number: 80
        host: echo.default.svc.cluster.local

Access the echo service on the external address 🔗︎

The service should be accessible on host echo.18.197.110.20.xip.io and port 8000.

❯ curl -i echo.18.197.110.20.xip.io:8000

HTTP/1.1 200 OK
date: Sun, 02 Feb 2020 19:22:15 GMT
content-type: text/plain
server: istio-envoy
x-envoy-upstream-service-time: 1

Hostname: echo-68578cf9d9-874rz
...

Our ability to easily create ingress gateways gives you fine-grained control over how services are exposed to the outside world. That way, teams can manage the exposure of their own services without running the risk of misconfiguring the services of other teams. Another way of tackling this potential issue is to have separate load balancer configurations with, for example, different port level settings.

Accessing External Services 🔗︎

Direct outbound traffic 🔗︎

Any traffic that’s outbound from a pod with an Istio sidecar will also pass through that sidecar’s container, or, more precisely, through Envoy. Therefore, the accessibility of external services depends on the configuration of that Envoy proxy. By default, Istio configures the Envoy proxy to passthrough requests for unknown services. Although this provides a convenient way of getting started with Istio, its generally a good idea to put stricter controls in place.

To read more about the Sidecar object configuration, check out this informative blog post:.

Check the default outbound traffic policy 🔗︎

❯ kubectl -n istio-system get istio mesh -o jsonpath='{@.spec.outboundTrafficPolicy.mode}{"\n"}'

ALLOW_ANY

This traffic policy should be set to ALLOW_ANY by default.

Make an HTTP request to httpbin.org from the echo service container in the default namespace 🔗︎

❯ kubectl -n default exec -ti deploy/echo -- curl -sD /dev/stderr http://httpbin.org -o/dev/null |head -5

Defaulting container name to echo-service.
Use 'kubectl describe pod/echo-68578cf9d9-874rz -n default' to see all of the containers in this pod.
HTTP/1.1 200 OK
date: Sun, 02 Feb 2020 20:43:40 GMT
content-type: text/html; charset=utf-8
content-length: 9593
server: envoy

This should work fine, since, by default, every sidecar sends traffic towards unknown services through its passhtrough proxy.

Change the default outbound traffic policy to block unknown services 🔗︎

Now we’re going to demonstrate a more controlled way of enabling access to external services. Change the spec.outboundTrafficPolicy.mode option from the ALLOW_ANY mode to the REGISTRY_ONLY mode in the mesh Istio resource in the istio-system namespace.

❯ kubectl -n istio-system patch istio mesh -p '{"spec":{"outboundTrafficPolicy":{"mode":"REGISTRY_ONLY"}}}' --type='merge'

istio.istio.banzaicloud.io/mesh patched

Make another HTTP request to httpbin.org from the echo service container in the default namespace 🔗︎

❯ kubectl -n default exec -ti deploy/echo -- curl -sD /dev/stderr http://httpbin.org -o/dev/null |head -5

HTTP/1.1 502 Bad Gateway
location: http://httpbin.org/
date: Sun, 02 Feb 2020 20:53:44 GMT
server: envoy
content-length: 0

Now we’re getting a 502 response code, since now the traffic towards external services is blocked and it is going through Envoy’s blackhole cluster.

Create a ServiceEntry to allow HTTP access to httpbin.org 🔗︎

ServiceEntry resources enable adding additional entries into Istio’s internal service registry, so that auto-discovered services in the mesh can access/route to these manually specified services. A service entry describes the properties of a service (DNS name, VIPs, ports, protocols, endpoints). These services could be external to the mesh (for example, web APIs) or mesh-internal services that are not part of the platform’s service registry. For more information about the ServiceEntry resource, see the Istio documentation.

Apply the following ServiceEntry to allow for HTTP access to httpbin.org

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: httpbin-ext
  namespace: default
spec:
  hosts:
  - httpbin.org
  ports:
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS
  location: MESH_EXTERNAL

Make another HTTP request to httpbin.org from the echo service container in the default namespace 🔗︎

❯ kubectl -n default exec -ti deploy/echo -- curl -sD /dev/stderr http://httpbin.org -o/dev/null |head -5

Defaulting container name to echo-service.
Use 'kubectl describe pod/echo-68578cf9d9-874rz -n default' to see all of the containers in this pod.
HTTP/1.1 200 OK
date: Sun, 02 Feb 2020 21:13:40 GMT
content-type: text/html; charset=utf-8
content-length: 9593
server: envoy

Outbound traffic through an egress gateway 🔗︎

As you probably recall from earlier in this blogpost, egress gateways are exit points from the mesh that allow us to apply Istio features. This includes applying features like monitoring and route rules to traffic that’s exiting the mesh.

Use cases 🔗︎

Let’s take a quick look at some use cases.

Consider an organization which requires some, or all, outbound traffic to go through dedicated nodes. These nodes could be separated from the rest of the nodes for the purposes of monitoring and policy enforcement.

Now imagine a cluster where the application nodes don’t have public IPs, so the in-mesh services that run on them cannot access the internet directly. Defining an egress gateway and routing egress traffic through it, then allocating public IPs to the gateway nodes would allow for controlled access to external services.

Create an egress gateway with the MeshGateway resource 🔗︎

Apply the following resource and the Istio operator will create a new egress gateway deployment and a corresponding service.

apiVersion: istio.banzaicloud.io/v1beta1
kind: MeshGateway
metadata:
  name: egress
  namespace: default
spec:
  maxReplicas: 1
  minReplicas: 1
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  serviceType: ClusterIP
  type: egress

Create a Gateway resource for the egress gateway 🔗︎

Similar to the ingress gateway configuration, a Gateway resource must be created that will be a bridge between Istio configuration resources and the deployment of a matching gateway.

Apply the following Gateway resource to configure the outbound port, 80, on the egress gateway that was just defined in the previous step.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: egress
  namespace: default
spec:
  selector:
    gateway-name: egress
    gateway-type: egress
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"

Define a VirtualService resource to direct traffic from the sidecars to the egress gateway 🔗︎

Apply the following VirtualService to direct traffic from the sidecars to the egress gateway and also from the egress gateway to the external service.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin-egress
spec:
  hosts:
  - httpbin.org
  gateways:
  - egress
  - mesh
  http:
  - match:
    - gateways:
      - mesh
      port: 80
    route:
    - destination:
        host: egress.default.svc.cluster.local
        port:
          number: 80
  - match:
    - gateways:
      - egress
      port: 80
    route:
    - destination:
        host: httpbin.org
        port:
          number: 80

Resend an HTTP request to httpbin.org from the echo service container in the default namespace 🔗︎

❯ kubectl -n default exec -ti deploy/echo -- curl -sD /dev/stderr http://httpbin.org/headers -o/dev/null |head -5

Defaulting container name to echo-service.
Use 'kubectl describe pod/echo-68578cf9d9-874rz -n default' to see all of the containers in this pod.
HTTP/1.1 200 OK
date: Sun, 02 Feb 2020 22:20:28 GMT
content-type: application/json
content-length: 1850
server: envoy

Output should be the same as earlier, but if we check the logs of the egress gateway, it shows that the request actually went through the egress gateway.

❯ kubectl -n default logs -l gateway-name=egress,gateway-type=egress -c istio-proxy | tail -1

[2020-02-02T22:20:29.191Z] "GET /headers HTTP/1.1" 200 - "-" "-" 0 1850 173 173 "10.20.5.1" "curl/7.47.0" "bf26bf45-4a0b-4d78-a968-98515a890a91" "httpbin.org" "52.202.2.199:80" outbound|80||httpbin.org - 10.20.5.16:80 10.20.5.1:58584 - -

Takeaway 🔗︎

Ingress and egress gateways are core concepts of a service mesh. Although Istio itself provides the basic building blocks, having an easy and simple way to create and manage multiple mesh gateways is a must. The Banzai Cloud Istio operator provides support with a new CRD called MeshGateway. Give it a try, and quickstart your Istio experience with Backyards (now Cisco Service Mesh Manager)!

About Banzai Cloud Istio operator 🔗︎

Banzai Cloud Istio operator is a simple way to deploy, manage and maintain Istio service meshes, even in multi-cluster topologies.

About Backyards 🔗︎

Banzai Cloud’s Backyards (now Cisco Service Mesh Manager) is a multi and hybrid-cloud enabled service mesh platform for constructing modern applications. Built on Kubernetes, our Istio operator and the Banzai Cloud Pipeline platform gives you flexibility, portability, and consistency across on-premise datacenters and on five cloud environments. Use our simple, yet extremely powerful UI and CLI, and experience automated canary releases, traffic shifting, routing, secure service communication, in-depth observability and more, for yourself.

About Banzai Cloud 🔗︎

Banzai Cloud is changing how private clouds are built: simplifying the development, deployment, and scaling of complex applications, and putting the power of Kubernetes and Cloud Native technologies in the hands of developers and enterprises, everywhere.

#multicloud #hybridcloud #BanzaiCloud