SLO-based alerting requires you to configure Backyards to use an Alertmanager deployment. This procedure describes how to configure Backyards to use an existing Alertmanager deployment.

Note: If you don’t have and existing Alertmanager deployment, or you want to use a separate deployment, see Deploy a new Alertmanager for Backyards.

Prerequisites 🔗︎

Prometheus Alertmanager installed and configured.

Note: We recommend configuring alert grouping in the following way:

   route:
     receiver: 'slack-notifications'
     group_by: [ service, severity ]

That way you can route notifications based on severity, using the severity label of Backyards-generated alerts.

Steps 🔗︎

To configure Backyards to use an existing Alertmanager deployment, complete the following steps.

  1. Download the following configuration snippet as enable-alertmanager.yaml.

    spec:
  backyards:
    prometheus:
      enabled: true
      alertmanager:
       static:
        - host: <your-alert-manager-0-host>
          port: <your-alert-manager-0-port>
        - host: <your-alert-manager-1-host>
          port: <your-alert-manager-1-port>

  2. Replace the your-alert-manager-X-host part with your Alertmanager’s fully qualified domain name, and the your-alert-manager-X-port with the port Alertmanager is listening on.

  3. Backyards is controlled by a ControlPlane custom resource found in the Backyard’s namespace (default: backyards-system) named backyards.

    The following command changes the spec.backyards.prometheus.alertmanager value to connect to the existing Alertmanagers. Run the following command:

    $ kubectl patch controlplane --type=merge --patch "$(cat enable-alert-manager.yaml)" backyards

  4. If you are using Backyards in operator mode, skip this step.

    Otherwise, execute a reconciliation so Backyards updates your Kubernetes cluster to the desired state described by the ControlPlane Custom Resource. Run the following command:

    $ backyards operator reconcile
  1. In case your Alertmanagers are not part of your service mesh setup, create destination rules in the backyards-system namespace to allow communication with your Alertmanagers.