It’s been a while since we published a new release of One Eye. That doesn’t mean we haven’t been working on it, quite the opposite! We’ve been busy making lots of improvements and adding features. This blog will highlight the many changes made since version 0.3. Let’s start with some great news! Logging Operator (responsible for the One Eye logging sub-system) is now the official logging backend of Rancher.

Finding a root cause can be hard. You need to collect data from several (usually disparate) sources. Often, when we debug a cluster, we need the output of its logging Flows to validate the consistency of its Logging stack. The idea in such cases is to peek into the messages transported by fluentd. Let’s take a look at how we might accomplish that. If you’re not familiar with the Logging Operator, please check out our earlier posts.

Many of you are probably familiar with the moment in Life of Brian when John Cleese lists all the improvements the Roman Empire has brought to Judea then asks, “What have the Romans ever done for us?” I mention this because, one day recently, I was scrolling through twitter when I stumbled onto a treasure trove of users blaming Kubernetes for its complexity, claiming things were better before it existed. So, in the interest of fair play, I’ve decided to write a bit about the glorious past and our mediocre present, and to compare the two.

A couple of weeks have passed since the release of One Eye version 0.3, which introduced some powerful UI and backend improvements. Our goal with One Eye is to provide a singular vantage point from which you can see what’s happening inside your Kubernetes cluster, centralizing our observability tools into one place and onto one dashboard. However, we’re not trying to reinvent the wheel by re-implementing observability, but to organize and configure existing tools to create a seamless experience.

More than a month ago, we announced One Eye, the observability tool for Kubernetes. This has been an ongoing project, and we release a new version of it about once per week. We’ve gathered the features included in those updates here to keep you up to speed. If you are not familiar with One Eye, check out our introductory blog post or browse the official documentation. Who is One Eye for?

Helm version 3 has been out officially for some time (release blog post was published on Wed, Nov 13, 2019). We’ve been using Helm since the early days of Kubernetes, and it’s been a core part of our Pipeline container management platform since day one. We’ve been making the switch to Helm 3 for a while and, as the title of this post indicates, today we’ll be digging into some details of our experience in transitioning between versions, and of using Helm as a Kubernetes release manager.

If you’ve been reading our blog you already know that we’re passionate about observability. We are convinced that the key to operating a reliable system is to know what happens where, and the correlated ability to rapidly dissect issues as they emerge. In previous posts we’ve gone over the base components of our suggested stack, which includes Prometheus, Thanos, Fluentd, Fluentbit, and many others. We’ve created several tools and operators to ease the management of these components, like the Istio operator, Logging operator, Thanos operator as well as using some other very popular operators, like the Prometheus one.

Today we are happy to announce a new release of the Banzai Cloud logging operator. It’s been a long time from the first commits till today, and is always nice to look back, learn and reflect on the evolution of the project. The first major release, June 2018 This was the very first release, and among the first operators we made. The operator pattern was pretty new, and the goal of the first logging operator was fairly simple - automate the manual fluent ecosystem configurations we were doing for our customers with the Pipeline platform.

At Banzai Cloud we support and manage hybrid Kubernetes clusters for our customers across five clouds and on-prem (bare metal, VMware). Therefore, the ability and fluency required to observe these clusters is an absolute must. Very frequently, the Pipeline control plane is tasked with managing multiple Kubernetes clusters, which it does through our own CNCF certified Kubernetes distribution, PKE, or a cloud provider-managed distribution. When that happens, it’s important that we federate metrics, collect them into a single place for querying, analysis and long term storage.

Without a doubt Prometheus has become the de facto standard monitoring solution for Kubernetes, the same way it has become a core component of the Pipeline platform’s monitoring service. However, Prometheus already has a well defined mission with a focus on alerts and the storage of recent metrics. Prometheus’ local storage is limited by single nodes in its scalability and durability. Instead of trying to solve clustered storage in Prometheus itself, Prometheus has a set of interfaces that allow integration through remote storage systems.

Frequent readers of our blog and users of our hybrid cloud container management platform, Pipeline, will be familiar with the integrated cluster services that come with it. These services are automated end-to-end solutions for centralized logging, federated monitoring, security scans, advanced credential management, autoscaling, registries and lots more (see, for example, automated DNS management for Kubernetes). Providing an automated logging solution, and making sure it works seamlessly across multiple clusters, has always been part of Pipeline.

On this blog we’ve already discussed our totally redesigned logging operator, which automates logging pipelines on Kubernetes. Thanks to the tremendous amount of feedback and the numerous contributions we received from our community, we’ve been able to rethink and redesign that operator from scratch, but the improvements aren’t going to stop coming any time soon. Our goal is to continue removing the burden from human operators, and to help them manage the complex architectures of Kubernetes.

About a year ago we published the first release of our popular logging-operator. The initial version of that operator was designed to fit Pipeline, the Banzai Cloud hybrid cloud container management platform. However, since then, all kinds of people have found it to be an extremely useful tool that helps them manage their logs on Kubernetes. Initially, Fluent ecosystem automation was enough to support the disparate needs of our userbase, but, as the popularity of the logging-operator grew, different setups were put in place by our community that revealed some of its limitations.

At Banzai Cloud we are passionate about observability, and we expend a great amount of effort to make sure we always know what’s happening inside our Kubernetes clusters. All clusters provisioned with Pipeline - our multi- and hybrid-cloud container management platform - are provided with, and rely upon, each of the three pillars of observability: federated monitoring, centralized log collection and traces. In order to automate log collection on Kubernetes, we opensourced a logging-operator built on the Fluent ecosystem.

This is the second part of a very popular post, Helm from basics to advanced. In the previous post (we highly suggest you read it, if you haven’t done so already) we covered Helm’s basics, and finished with an examination of design principles. In this post, we’d like to continue our discussion of Helm by exploring best practices and taking a look at some common mistakes. Before we start, if you really want to get deep down into the nitty gritty of Helm, I suggest you to read the official guide to creating templates.

Update: Logging operator v3 (released March, 2020) We’re constantly improving the logging-operator based on feature requests of our ops team and our customers. The main features of version 3.0 are: Log routing based on namespaces Excluding logs Select (or exclude) logs based on hosts and container names Logging operator documentation is now available on the Banzai Cloud site. Check The Kubernetes logging operator reloaded post for details.

Banzai Cloud is on a mission to simplify the development, deployment, and scaling of complex applications and to bring the full power of Kubernetes to all developers and enterprises. Banzai Cloud’s Pipeline provides a platform which allows enterprises to develop, deploy and scale container-based applications. It leverages best-of-breed technology from the Cloud Native Foundation ecosystem to create a highly productive, yet flexible environment for developers and operation teams alike. One of the key tools we use from the Kubernetes ecosystem is Helm.

Here at Banzai Cloud we blog a lot about Prometheus and how to use it. One of the problems we have so far neglected to discuss is the inadequate long term storage capability of Prometheus. Luckily a new project called Thanos seeks to address this. If you are not familiar with Prometheus, or are interested in other monitoring related articles, check out our monitoring series, here: Monitoring series: Monitoring Apache Spark with Prometheus Monitoring multiple federated clusters with Prometheus - the secure way Application monitoring with Prometheus and Pipeline Building a cloud cost management system on top of Prometheus Monitoring Spark with Prometheus, reloaded

Update: Logging operator v3 (released March, 2020) We’re constantly improving the logging-operator based on feature requests of our ops team and our customers. The main features of version 3.0 are: Log routing based on namespaces Excluding logs Select (or exclude) logs based on hosts and container names Logging operator documentation is now available on the Banzai Cloud site. Check The Kubernetes logging operator reloaded post for details.

As part of the Debug 101 series, we’re back hunting a small but annoying bug. This kind of bug is not really a bug, but a side effect of several tools working together. Here comes trouble I deploy a development version of Pipeline on a Kubernetes cluster running on top of AWS infrastructure. For this deployment I use the following Helm chart command. $: helm install --name pipeline banzaicloud-stable/pipeline-cp \ --set=drone.

Monitoring series: Monitoring Apache Spark with Prometheus Monitoring multiple federated clusters with Prometheus - the secure way Application monitoring with Prometheus and Pipeline Building a cloud cost management system on top of Prometheus Monitoring Spark with Prometheus, reloaded At Banzai Cloud we provision and monitor large Kubernetes clusters deployed to multiple cloud/hybrid environments. These clusters and applications or frameworks are all managed by our next generation PaaS, Pipeline.