One of the core feature of Banzai Cloud’s container management platform, Pipeline, is the capability of building hybrid clouds with ease. The most important reason behind the introduction of our own CNCF certified Kubernetes distribution, Pipeline Kubernetes Engine (PKE) was to provide our customers a Kubernetes environment that behaves consistently across public cloud providers, on-premise, and the combination of those.
A single approach does not cover all use-cases, so the Banzai Cloud Pipeline platform provides four different ways to build and use hybrid clouds.
- Cluster groups
- Federation
- Service mesh
- Hybrid cloud controller
To learn more, check the Four ways to build hybrid clouds with Kubernetes post.
Kubernetes has integration with the networking and storage services available for the computing resources of most cloud providers, so exposing a network service to the internet, or creating a persistent volume can be quite effortless — given that you are happy with the default configurations and the costs of these integrated services.
What does this mean in practice?
If you create a LoadBalancer service — for example try to expose your own TCP based service, or install an ingress controller — the cloud provider integration will take care of creating the needed cloud resources, and writing back the endpoint where your service will be available.
If you don’t have a cloud provider integration or a controller for this purpose, your Service resource will remain in Pending state.
We have always had solutions for real-life on-prem use cases, but recently we started to merge the solutions that we use in the majority of our on-prem deployments into our open source PKE project.
Let’s see how we use MetalLB to provide flawless LoadBalancer experience in bare-metal and virtualized environments like VMware vSphere, Hyper-V, or different KVM and Xen-based solutions.
Background 🔗︎
Load-balancing is a solution, not an exact problem. Most of the time when we speak about load balancers, we assume them to take care of two different, but related issues by distributing the incoming traffic to the instances of your horizontally scaled application.
- Performance: in the ideal case, you can sum up the individual performance of your application instances. In general, together they can withstand loads that a single instance wouldn’t be able to handle.
- High availability: if some of your application instances stop working because of a failure, or for the propagation of a change (like a version upgrade), the remaining ones will take the load.
Furthermore, in case of Kubernetes, LoadBalancer services are the easiest and most common way to expose a service (redundant or not) for the world outside of the cluster or the mesh — to other services, to internal users, or to the internet.
Load balancing as a concept can happen on different levels of the OSI network model, mainly on L4 (transport layer, for example TCP) and L7 (application layer, for example HTTP).
In Kubernetes, Services are an abstraction for L4, while Ingresses are a generic solution for L7 routing and load balancing of application protocols (HTTP/HTTPS).
This is not something you have to choose from, because the engines behind Ingress, for example Traefik or Nginx ingress controllers, are typically accessed through LoadBalancer services.
Cloud integrations 🔗︎
Cloud providers have different solutions to implement a LoadBalancer type Service resource in Kubernetes.
For example AWS backs them with Elastic Load Balancers: Kubernetes exposes the service on specific TCP (or UDP) ports of all cluster nodes’, and the cloud integration takes care of creating a classic load balancer in AWS, directing it to the node ports, and writing back the external hostname of the load balancer to the Service resource.
Bare metal environments have no such general solution, and even enterprise network stacks don’t have a generic implementation of a TCP/UDP load balancer. Different solutions for L2 or L3 redundancy might be at your help, but there is no magic here that would solve our issues.
ExternalIP 🔗︎
There is a built-in, low-level solution in Kubernetes for exposing services from a cluster without cloud integrated LoadBalancers.
Kubernetes services of ExternalIP type start serving on the specified IP addresses once the traffic is routed to (and from) any of the nodes.
In the background this is handled with a bunch of netfilter (iptables) rules and kube-proxy, which is part of Kubernetes installations, and directs traffic from service endpoints to the backends (either to a pod scheduled to the node it runs on, or to another).
However, this solution has some drawbacks:
- You have to manually allocate and manage IP addresses for your services.
- Routes for allocated addresses have to be propagated to the network devices, either statically by an administrator, or by a routing protocol.
- Redundancy (availability) depends on the packets being routed to a healthy node, either by changing the routes, or maintaining a virtual IP address.
Routing traffic to multiple nodes 🔗︎
You might think that a network router is smart enough to fall back to redundant routes on failures, but this would require the router to know that the next hop is unreachable (which would not even solve all scenarios). However, this information is readily available only when the link of a directly connected route is down, which is rarely the case, not to speak about the higher level healthiness of nodes.
The most obvious and well-known solution is to use virtual IP addresses: there are more than one devices (Kubernetes nodes acting as routers in our case) which could serve a given address. These nodes keep track of exactly one leader, and this leader has the virtual address.
There are solutions for circumventing the single point of failure caused by external load balancers: a dedicated load balancer service or appliance is usually based on either:
- a highly available and high performance device that can share the load between backends (Kubernetes node ports in our case) in an active-passive architecture backed by:
- gratuitous/unsolicited ARP based virtual IPs + heartbeat, or
- dynamic routing and heartbeat protocols like HSRP, or
- the limited client-side loadbalance/failover offered by round-robin DNS records.
MetalLB 🔗︎
MetalLB is one of the most popular on-prem replacements for LoadBalancer cloud integrations. The whole solution runs inside the Kubernetes cluster.
The main component is an in-cluster Kubernetes controller which watches LB service resources, and based on the configuration supplied in a ConfigMap, allocates and writes back IP addresses from a dedicated pool for new services. It maintains a leader node for each service, and depending on the working mode, advertises it via BGP or ARP (sending out unsolicited ARP packets in case of failovers).
The main difference from the user’s point of view between the two is that the ARP solution requires each node to be in a single L2 network (subnet), while BGP requires dynamic routing configuration on the upstream router.
In addition, MetalLB can operate in two ways: either all requests are forwarded to pods on the leader node, or distributed to all nodes with kubeproxy.
If you have two dedicated Kubernetes nodes for running MetalLB, you can achieve the theoretical performance and reliability of a pair of “classic” active-passive software load balancers.
If you have a clean Ubuntu 18.04 virtual machine in a subnet with a few free addresses you can try this out by installing a single-node PKE cluster and configuring the ARP based MetalLB with the following command (adjust the range to your environment):
pke install single --lb-range=192.168.22.230-192.168.22.239
Alternatives 🔗︎
Layer 7 (usually HTTP/HTTPS) load balancer appliances like F5 BIG-IP, or HAProxy and Nginx based solutions may be integrated with an applicable ingress-controller. If you have such, you won’t need a LoadBalancer implementation in most cases.
About PKE 🔗︎
Banzai Cloud Pipeline Kubernetes Engine (PKE) is a simple, secure and powerful CNCF-certified Kubernetes distribution, the preferred Kubernetes run-time of the Pipeline platform. It was designed to work on any cloud, VM or on bare metal nodes to provide a scalable and secure foundation for private clouds. PKE is cloud-aware and includes an ever-increasing number of cloud and platform integrations.
