Banzai Cloud is now part of Cisco

Banzai Cloud Logo Close
Home Products Benefits Blog Company Contact

The content of this page hasn't been updated for years and might refer to discontinued products and projects.

Banzai Pipeline, or simply “Pipeline” is a tabletop reef break located in Hawaii, on Oahu’s North Shore. It is the most famous and infamous reef on the planet, and serves as the benchmark by which all other surf breaks are measured.

Pipeline is a PaaS with a built in CI/CD engine to deploy cloud native microservices to a public cloud or on-premise. It simplifies and abstracts all the details of provisioning cloud infrastructure, installing or reusing a Kubernetes cluster, and deploying an application.

Note: The Pipeline CI/CD module mentioned in this post is outdated and not available anymore. You can integrate Pipeline to your CI/CD solution using the Pipeline API. Contact us for details.

Today we are pleased to announce the third release of Pipeline - 0.3.0. Besides a battery of new features and fixes, the main focus of this release is to add support for the Google Kubernetes Engine GKE, and to simplify/strengthen security using OAuth2 and Vault.

A major change in 0.3.0 is that Pipeline now requires GitHub OAuth authentication similar to a CI/CD flow. Accordingly, it allows both Pipeline and the CI/CD flow to be authenticated by a single mechanism, as opposed to before, when Pipeline requiried only basic authentication. Also, this release adds support for three different serverless/function-as-service frameworks.

Give it a try 🔗︎

A control plane is available for all supported providers. And an end-to-end howto guide (including a full CI/CD flow) is available, here.

tl;dr: 🔗︎

New cloud provider support - Google Cloud GKE 🔗︎

We have added native support for Google’s Kubernetes engine, on which the Pipeline control plane is available as a Kubernetes deployment, and which can also provision and deploy clusters to GKE.

We strongly believe that managed Kubernetes services are the future, and most of our investment in cloud neutrality and provisioning is built on managed Kubernetes services, both in the cloud and on-premise.

Services 🔗︎

Please find below the list of services run by the control plane. Note that this is not an exhaustive list - there are other low level infrastructure services running inside k8s, like Tiller, that Pipeline speaks to, and other beta services like aggregated log collectors, Helm chart orchestrators, namespace operators or K8S watchers/informers.

Service Technology stack Type Remarks
Monitoring Prometheus Full vertical and horizontal stack monitoring for AWS, AKS, GKE, K8S, and deployed microservices Default node exporters and push gateways are deployed when needed
Alerting Prometheus Default alerts for infra and apps, customizable node exporters and push gateways Correlated alerts across the stack, collected for model build
Dashboards Grafana AWS, AKS, GKE, K8S, app deployment specific dashboards Dashboards are based on spotguides, e.g. a Spark deployment that contains all relevant metrics, and differs from a Zeppelin or JavaOne deployment
CI/CD Drone AWS, AKS, GKE, K8S, microservices Vendor independent, can be used with CircleCI or Travis
Pipeline plugins Golang, Docker AWS, AKS, GKE, K8S, Spark, Zeppelin, Kafka, Java, Golang Extensible, custom plugins are built in Go based on an interface
WebHooks GitHub Default GitHub web hooks GitHub marketplace placement and GitLab support are ongoing

Pipeline PaaS release components - DIY, be your own PaaS vendor 🔗︎

In the 0.3.0 release, components of Pipeline have been re-worked to easily streamline the incorporation of new cloud providers, and to provide the same level of automation and easier enablement to new cloud providers.

This release, 0.3.0, adds support for Google’s Managed Kubernetes GKE support, and moves away from username/password-based authentications and towards OAuth2 tokens. Both Pipeline and the CI/CD component use OAuth2 tokens issued by GitHub after successful GitHub OAuth2 authentication.

For further details, read the Pipeline how-to documentation.

Component Source code
Pipeline API
Control Plane Cloudformation template
Cluster images
Control plain images
Banzai Charts
Pipeline CI/CD plugin
Kubernetes CI/CD plugin
Zeppelin CI/CD plugin
Spark CI/CD plugin
HTPassword generator for ingress
Spark on K8S
Zeppelin on K8S
TensorFlow on K8S
TiDB on K8S
Zeppelin on K8S
Spark History Server
Serverless - Fn
Serverless - OpenFaaS
Serverless - Kubeless

Pipeline PaaS release - the hosted version, aka the next generation of Heroku/CloudFoundry 🔗︎

Here, we have demonstrated the complexity and number of building blocks necessary in order to host your own PaaS and become your own microservice provider. We do this to better understand and illustrate the level of work, maintenance, as well as the HA deployments and Kubernetes expertise, that’s required to run these components.

We are moving towards a hosted service that will host, deploy, maintain, patch and support all these services, with the end goal that our users should never know, be exposed to, or care about the underlying systems; at the end of the day your focus should be on writing applications, without having to worry about whether they are built, deployed and automatically operated according to your company’s SLA rules and configured CI/CD workflow.

Supported flows 🔗︎

Once a control plane is up and running (please check the following installation guide), Pipeline is able to serve as a fully customizable CI/CD workflow by placing a .pipeline.yml file under source control, alongside your project. This is similar to commercial CI systems like CircleCI or Travis, however, instead of using these systems (though that is possible), it uses Pipeline’s own CI/CD system with plugins for workflow steps, such as:

  • clone - clones a GH repository inside a k8s clusters (uses PVC)
  • remote_checkout - checks out a GH repository inside a k8s cluster (uses PVC)
  • cluster - provisions, updates, reuses or deletes a cloud infrastructure and Kubernetes cluster, and deploys a runtime required for the spotguide (e.g. if it’s Spark then it understands pre-requisites like RSS and executors running as k8s daemon sets)
  • remote_build - builds an application inside a k8s cluster based on the spotguide
  • run - runs an application on a runtime required by the spotguide Flows are backed by spotguides. Currently, we support the following:
  • Java with cgroups support
  • Apache Spark support
  • Apache Zeppelin support
  • TensorFlow support
  • JupyterHub support
  • TiDB support
  • Fn, container native serverless platform support
  • OpenFaaS, functions as a service support

A typical flow in the UI, which deploys the custom workflow that’s described in the .pipeline.yml looks like this.

What’s next 🔗︎

This is an early release that we’re handing over to a few beta testers across different industries, with an eye toward receiving feedback and allowing them to drive the direction and features of Pipeline’s next generation. We don’t recommend deploying production use cases to Pipeline yet, not because the system is too unstable to run the supported spotguides, but because the current release falls far short of what we at Banzai Cloud believe a next generation PaaS platform should look like. In the coming months we’ll add support for many cloud or managed k8s providers, deliver an end to end security model integrated with Kubernetes (based on OAuth2 tokens and Vault) and a real service mesh to support throttling, as well as canary releases and policy driven ops and many many other features, all of which will be available through the current rest API, UI and CLI. Also, we’re doing extensive work on Hollowtrees, with custom plugins for Pipeline spotguides and SLA policy-driven Kubernetes schedulers that support the resource management needs of your microservices.

Building a PaaS is a challenging yet rewarding engineering experience, which we are all excited about. We encourage you to take part in this open source engagement, and we’ll continue to provide detailed information in our blog in order to share the experience, allowing you to join in and even to drive the process of creation.

About Banzai Cloud Pipeline 🔗︎

Banzai Cloud’s Pipeline provides a platform for enterprises to develop, deploy, and scale container-based applications. It leverages best-of-breed cloud components, such as Kubernetes, to create a highly productive, yet flexible environment for developers and operations teams alike. Strong security measures — multiple authentication backends, fine-grained authorization, dynamic secret management, automated secure communications between components using TLS, vulnerability scans, static code analysis, CI/CD, and so on — are default features of the Pipeline platform.