Access control with Pomerium on Kubernetes

With Pipeline, we strive to provide a unified authentication and authorization experience across our multi- and hybrid-cloud environments. To accomplish this, we rely on dex, an identity service that uses OpenID Connect to drive authentication for apps. Dex and OpenID Connect use ID Tokens that are an OAuth2 extension, but not all the applications we use supports OAuth2 flows. Because of this, we searched for an OAuth proxy solution that handles authentication and basic policies that control access to these applications and services. Fortunately, we found Pomerium, which fits perfectly with our Kubernetes environment. After a chat with Bobby, Pomerium’s maintainer, we resolved to collaborate on a blog post that would demonstrate the power of Pomerium, and how it helps you secure even the simplest applications.

This is a guest post by Bobby DeSimone, core Pomerium maintainer.

What is Pomerium? 🔗︎

Pomerium is an identity-aware proxy that enables secure access to internal applications. It provides a standardized interface to add access control to applications, regardless of whether an application itself has authorization or authentication baked-in. Pomerium gateways both internal and external requests, and can be used in situations where you’d typically reach for a VPN.

Why Pomerium? 🔗︎

For many years, security was synonymous with network security. Firewalls, network segmentation, and VPNs ruled the day. Broadly speaking, this network-focused security posture is what people mean today when they talk about a “perimeter” security model. This so-called “impenetrable fortress” worked only as long as you could reasonably expect your network perimeter to correspond to a physical perimeter: users, devices, and servers. But, as teams, applications, workloads, and users grew increasingly ephemeral, containerized, and distributed, the shortcomings of perimeter-based security became more obvious, both in terms of operational costs and security breaches.

Most networks [have] **big castle walls, hard crunchy outer shells, and soft gooey centers…**
Rob Joyce, Chief, Tailored Access Operations, National Security Agency @ ENIGMA 2016

There’s no such thing as perfect security. Many recent high-profile breaches have demonstrated how difficult it is for even large companies with sophisticated security organizations to avoid a breach. To pick just two of many breaches wherein perimeter security played a major role, consider the Target and Google hacks. In Target’s case, hackers circumvented both the physical and network perimeter by hacking an HVAC system, which was connected to the internal corporate network. The hackers then moved laterally to exfiltrate customer credit card data. In Google’s case, they were on a receiving end of a devastating attack from the Chinese military. Google did a bottom up review of their security following “Operation Aurora”. The actions they took as a result of that review were released as a series of white papers, called “BeyondCorp”. These have since become foundational documents in the effort to articulate how and why an organization should move beyond corporate perimeter-based (BeyondCorp… get it?) security.

In reality, there’s never one front door; there are many front doors… [and] …we’re not securing a single castle. We’re starting to think about securing many different interconnected castles.
Armon Dadgar, Founder, HashiCorp @ PagerDuty Nov 2018

The other side of the security trade-off is operational agility. Perimeter-based approaches tend to focus on network segmentation, which entails creating virtual or physical boundaries around services that need to communicate. This makes those boundaries increasingly difficult to manage in a world of microservices and cloud computing, wherein service communication requirements are constantly in flux. In theory, an organization could “micro/nano/pico-segment” each and every layer of an application stack, so as to ensure an appropriate audience. In practice, however, operators usually choose between very precise boundaries that are high-touch, time-consuming to manage, and error prone, and more lax boundaries, which may involve more risk, but are less time consuming to update and manage, and less prone to break.

In summary, perimeter-based security suffers from the following shortcomings:

• Perimeter security largely ignores the possibility of an inside threat. Given that thirty percent of all breaches are from internal actors, this is a pretty big omission.
• If the last few years have proved anything, it’s that the impenetrable fortress theory fails in practice, even when employed by the most sophisticated security organizations.
• Network segmentation is time-consuming, and is difficult to get exactly right so as to ensure secure communication.
• Just defining what, exactly, the network perimeter is is an increasingly difficult proposition in a remote-work, BYOD, multi-cloud world. Most organizations are a heterogeneous mix of clouds, servers, devices, and organizational units.
• VPNs are often misused. They exacerbate the problems inherent in perimeter-based security by opening yet another door into networks.

Conversly, zero-trust attempts to mitigate these shortcomings by adopting the following principles:

• Trust flows from identity, device-state, and context; not network location.
• Treat both internal and external networks as completely untrustworthy; mutually authenticated encryption is used instead of network segmentation.
• Act like you’ve already been breached, because you probably have been, and an attacker could be anyone and anywhere on your network.
• Every device, user, and application’s communication should be authenticated, authorized, and encrypted.
• Access policy should be dynamic, and built from multiple sources.

To be clear, perimeter security is not defunct, nor is zero-trust security a panacea or a single product. Many of the ideas and principles of perimeter security are still relevant and are part of a holistic, and wide-ranging security policy. After all, we still want our castles to have high walls.

So, to put all this back in context, before zero-trust tools like Pomerium existed, access to internal applications were gated based on whether a user was on the corporate network or not. Trust flowed from, and was anchored in, the security of the perimeter. For all the reasons we’ve discussed, this has made “perimeter” an insufficiently rigorous security model. In contrast, Pomerium adopts the zero-trust position and uses identity, device-state, and context, which is compared to a single-source of rich authorization policies and acts as the basis for the delegation of access to an internal resource. All Pomerium communication is mutually authenticated and encrypted; trust is not justified on the basis of a user or device being inside or outside the network.

How? 🔗︎

Pomerium is an authenticating reverse proxy.

Pomerium is responsible for establishing whether an incoming request can be trusted, and whether it should be handled downstream by an internal application. For a more practical example, let’s take a look at how Pomerium accomplishes this by walking through a typical request.

1. User makes an unauthenticated request to the service
2. Pomerium proxy receives the request and recognizes it as anonymous
3. It redirects the user to the auth provider for authentication
4. Upon successful login, Pomerium provides an auth cookie to the user. In the picture you can see the dex approval page before that.
5. Based on the cookie, Pomerium identifies the user and checks policy to determine whether to permit access. Authorization is based on identity factors like id, email, group, role, or email domain.
6. When the cookie expires, the login flow gets triggered all over again

Securing Prometheus with Pomerium 🔗︎

There are several services we use (and offer) as out-of-the-box solutions for Kubernetes clusters deployed with Pipeline. One of these is federated Prometheus for monitoring and alerts. If you want to reach the original Prometheus UI or the Alertmanager UI - and you need authentication - your only options are external solutions like basic authentication or an authentication proxy. This is where Pomerium comes handy. Pipeline comes with dex for authentication and Prometheus for monitoring and alerts, so it makes sense that we use dex as our authentication frontend. Now let’s take a look at how we’ve automated access to these services.

Install 🔗︎

Installing Pomerium with Helm is relatively easy. First, you need to decide what OAuth2 provider you want to use. You can choose from any of the supported identity providers or use a generic provider like dex as backend.

Configuring dex 🔗︎

The simplest way to enable communication with dex is to add a static client configuration:

staticClients:
- id: "${dex_client_id}"
  secret: "${dex_client_secret}"
  name: Pomerium'
  # Where the app will be running.
  redirectURIs:
  - 'https://my-pipeline.com/oauth2/callback'

Note: in order to use Pomerium with dex you need a complete a dex installation with valid TLS Certificates

Setting up dex as Auth backend 🔗︎

This is an example values.yaml file for Pomerium with dex:

authenticate:
  idp:
    provider: oidc
    url: "https://my-pipeline.com/dex"
    clientID: "${dex_client_id}"
    clientSecret: "${dex_client_secret}"
extraEnv:
  IDP_SCOPES: "openid profile email offline_access groups"
config:
  rootDomain: my-pipeline.com
  policyFile:
    - from: prometheus.my-pipeline.com
      to: http://monitor-prometheus-operato-prometheus.monitor:9090
      allowed_groups:
        - banzaicloud:dev
    - from: alertmanager.my-pipeline.com
      to: http://monitor-prometheus-operato-alertmanager.monitor:9093
      allowed_groups:
        - banzaicloud:dev

Installing Pomerium 🔗︎

You can install Pomerium with the following Helm command.

$ helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
$ helm install banzaicloud-stable/pomerium -f values.yaml

In such a setup the https://prometheus.my-pipeline.com and https://alertmanager.my-pipeline.com URLs are guarded by Pomerium. The idp section is dex’s generic configuration. IDP_SCOPES provides the email and group parameters for authorization, while the allowed_groups attribute in policyFile ensures access only for the banzaicloud organization dev team.

As you can see, wiring services into common and OAuth standard-based flows becomes very simple with Pomerium. If you want to learn more about Pomerium visit https://pomerioum.io.