At Banzai Cloud we support and manage hybrid Kubernetes clusters for our customers across five clouds and on-prem (bare metal, VMware). Therefore, the ability and fluency required to observe these clusters is an absolute must. Very frequently, the Pipeline control plane is tasked with managing multiple Kubernetes clusters, which it does through our own CNCF certified Kubernetes distribution, PKE, or a cloud provider-managed distribution. When that happens, it’s important that we federate metrics, collect them into a single place for querying, analysis and long term storage.

Without a doubt Prometheus has become the de facto standard monitoring solution for Kubernetes, the same way it has become a core component of the Pipeline platform’s monitoring service. However, Prometheus already has a well defined mission with a focus on alerts and the storage of recent metrics. Prometheus’ local storage is limited by single nodes in its scalability and durability. Instead of trying to solve clustered storage in Prometheus itself, Prometheus has a set of interfaces that allow integration through remote storage systems.

Here at Banzai Cloud we blog a lot about Prometheus and how to use it. One of the problems we have so far neglected to discuss is the inadequate long term storage capability of Prometheus. Luckily a new project called Thanos seeks to address this. If you are not familiar with Prometheus, or are interested in other monitoring related articles, check out our monitoring series, here: Monitoring series: Monitoring Apache Spark with Prometheus Monitoring multiple federated clusters with Prometheus - the secure way Application monitoring with Prometheus and Pipeline Building a cloud cost management system on top of Prometheus Monitoring Spark with Prometheus, reloaded