Fn and Hollowtrees

The adoption of serverless technologies is quickly progressing. According to this survey, it’s on par with the adoption of containers. And, even though ‘serverless’ is a very vague term (it can be argued that it’s rarely used in production, especially in complex applications), it seems set to be one of the most dominant trends in the near future in the cloud computing space. While, once, serverless referred specifically to early stage AWS Lambda, the category has matured rapidly. New frameworks are appearing and the Cloud Native Computing Foundation has created a working group to “explore the intersection of cloud native and serverless technologies”. At Banzai Cloud we recognize the importance of this shift, and, a few weeks ago, we announced our Fn spotguide in a blog post. That post makes it easy for users to deploy the framework on their preferred cloud provider, using Pipeline. It also describes why we chose Fn, and how to deploy and use it.

Note that Pipeline also supports OpenFaaS and Kubeless, however, while writing internal functions used by our Pipeline PaaS, we standardized internally on Fn.

The serverless model and Hollowtrees 🔗︎

Hollowtrees is part of the Pipeline PaaS. It’s an alert-react framework that processes alerts from Prometheus, applies rules and dispatches action flows to plugins using standard CNCF interfaces. If you’d like to learn more about Hollowtrees, and how it can help save you money on the cloud, check out this post.

Hollowtrees normally assumes that there are action plugins running in the cluster, which are listening on gRPC interfaces and accepting events from the Hollowtrees action flow controller. We’ve realized that it’s the nature of these plugins to be idle most of the time, while they wait for instructions from the server. But even when they are idle, they reserve resources in the Kubernetes cluster, which, to us, suggests a serverless model; if a serverless framework is running in our cluster and Hollowtrees is able to send events to functions, they’ll only be called on demand, and will not take resources from the cluster while idle.

So we’ve decided to extend Hollowtrees to support Fn functions as action plugins. This will help cost efficiency because action plugins are only triggered when there is an event to handle, and exit once they are finished processing the event instead of continuously running. The diagram below shows how the architecture changes when we’re using Fn to react to alerts, instead of using gRPC based action plugins.

In the second part of this post we’ll deploy Fn to a Kubernetes cluster, create and deploy a test function and set up Hollowtrees to send events to that function. Before we can deploy Fn and Hollowtrees, we’ll need a Kubernetes cluster. To start one, you should use Pipeline. If you’re not sure how to start a Pipeline control plane and cluster, begin by looking at our Github repo.

Deploy Fn and create a test function 🔗︎

Once we have a Kubernetes cluster and we’ve deployed our Prometheus-based monitoring stack through the Pipeline API’s create deployment endpoint, we can start deploying Fn to our cluster. The easiest way to do this is via our Fn spotguide. Make sure to follow the instructions described in our last Fn blog post, and locate the public endpoint of the Fn service by using the Cluster Public Endpoints REST API call to call the Pipeline API. This address is set in Hollowtrees, and will be needed later to send events to.

For now, we’ll use only a Hello world Fn function and a very simple Prometheus alert, but that should be enough to allow us to take a deeper look at the project’s architecture. Before we can start using the fn command line tool to deploy apps to our server, we should log into a docker hub account with docker login; Fn will use docker hub to store images being built from our code. Let’s set the public API URL of the Fn server, create the hello world example with the init command, and deploy it:

$ export FN_API_URL=http://<the `Host` listed for <fn-release-name>-fn-api/

fn init --runtime go hello
cd hello
export FN_REGISTRY=<DOCKERHUB_USERNAME>
fn deploy --app myapp

Note: if the deploy command fails, try to change the base image of the hello function to golang:1.9:

build_image: golang:1.9
run_image: golang:1.9

Deploy and configure Hollowtrees 🔗︎

The next few steps deploy the Hollowtrees server to the control plane and configure it to work with the Fn plugins. These steps involve some manual configuration, because Hollowtrees is not yet enabled in Pipeline deployments by default, since it’s still in an experimental phase. However, we are planning to include it in a later release. Once it’s included, most of these steps will be automated and Hollowtrees will be enabled out-of-the-box. First, enter the Pipeline control plane and deploy the Hollowtrees Helm chart without the default gRPC action plugins:

helm install --name ht banzaicloud-stable/hollowtrees --set asgplugin.enabled=false --set k8splugin.enabled=false

The above command will start the Hollowtrees dispatcher server, but, first, we should configure our action plugins and rules. The Hollowtrees configuration is stored in a Kubernetes configmap, so let’s edit that and change the action_plugins and rules sections to:

action_plugins:
- name: "hello-fn-plugin"
  address: "<the `Host` listed for <fn-release-name>-fn-api"
  type: "fn"
  properties:
    app: myapp
    function: hello

rules:
- name: "hello_fn"
  description: "says 'hello world' when a spot instance is terminated"
  event_type: "prometheus.server.alert.SpotTerminationNotice"
  action_plugins:
    - "hello-fn-plugin"

Hollowtrees should be restarted to process the new config. Simply delete the Hollowtrees pod, and the deployment will restart. Before we continue, let’s take another look at the configuration above. In the action_plugins section we set up the fn function we just deployed by setting its address, app name and function name. In the rules section we put together a rule that calls this action plugin when a specific type of event occurs. The event type is prometheus.server.alert.SpotTerminationNotice. That means that if the Prometheus alert called SpotTerminationNotice is firing, Hollowtrees will call our function.

Let’s take a look at the Prometheus section. We’d like Hollowtrees to receive alerts from the Prometheus server we deployed to the control plane. The Prometheus configuration is also stored in a configmap, and it must be edited so that the alertmanagers block is extended as below:

kubectl edit configmap <release-name>-prometheus-server

Add the following snippet to the alerting block:

    alerting:
      alertmanagers:
      - static_configs:
        - targets:
          - ht-hollowtrees:8080
        scheme: http
        path_prefix: /
        timeout: 2s

Prometheus is now sending its alerts to Hollowtrees but we don’t have an alert set up yet. To test things out, we’ll use our spot termination exporter but we can add another dummy alert as well. Deploy the exporter to the cluster through the Pipeline API’s create deployment endpoint:

{"name": "banzaicloud-stable/spot-termination-exporter", "release_name": "spot-exporter"}

Add the alert through the same configmap we used for Prometheus. Make sure that rule_files is set to /etc/config/alerts, and add a new alert rule to the alerts section. Our alert isn’t particularly useful in real world situations, but right now we want to see if our stack is working. This alert will keep firing as long as AWS spot instance termination is NOT imminent:

    alerts: |
      groups:
      - name: hello
        rules:
        - alert: SpotTerminationNotice
          expr: |
            aws_instance_termination_imminent == 0
          for: 10s

After the config is saved, our last building block is in place. Head over to the Prometheus UI and check the Alerts page. The alert we’ve set up should be there and should be firing. Also, check the Hollowtrees logs with kubectl logs. See if it’s able to call the fn function. If yes, you should see more log lines like this:

time="2018-03-21T14:25:12Z" level=info msg="sync fn function completed, call id: 01C94F64REA02HJ00000000000, response body: {\"message\":\"Hello World\"}\n" package=engine

If you made it this far, congrats: you’ve got a working Hollowtrees deployment and a serverless function that can react to Prometheus alerts! That’s it for now, but we’ll return soon enough with some real world examples of how to use Hollowtrees and Fn together. If you’ve tried it and have a new use case, feel free to share it with us.