Backyards

Circuit Breaking

Circuit Breaking is a pattern for creating resilient microservices applications. In microservices architecture, services are deployed across multiple nodes or clusters and have different response times or failure rate. Downstream clients need to be protected from excessive slowness of upstream services. Upstream services, in turn, must be protected from being overloaded by a backlog of requests.

A circuit breaker can have three states:

Closed: requests succeed or fail until the number of failures reach a predetermined threshold, with no interference from the breaker. When the threshold is reached, the circuit breaker opens.
Open: the circuit breaker trips the requests, which means that it returns an error without attempting to execute the call
Half open: the failing service is given time to recover from its broken behavior. If requests continue to fail in this state, then the circuit breaker is opened again and keeps tripping requests. Otherwise, if the requests succeed in the half open state, then the circuit breaker will close and the service will be allowed to handle requests again.

Backyards is using Istio’s - and therefore Envoy’s - circuit breaking feature under the hood.

Try it out! 🔗︎

Circuit breaking using the backyards-cli 🔗︎

Set circuit breaking configurations 🔗︎

Let’s put this to the test by creating the Circuit Breaker through the CLI.

You can do this in interactive mode:

$ backyards r cb set backyards-demo/notifications
? Maximum number of HTTP1/TCP connections 1
? TCP connection timeout 3s
? Maximum number of pending HTTP requests 1
? Maximum number of requests 1024
? Maximum number of requests per connection 1
? Maximum number of retries 1024
? Number of errors before a host is ejected 1
? Time interval between ejection sweep analysis 1s
? Minimum ejection duration 3m
? Maximum ejection percentage 100
INFO[0043] circuit breaker rules successfully applied to 'backyards-demo/notifications'
Connections  Timeout  Pending Requests  Requests  RPC  Retries  Errors  Interval  Ejection time  percentage
1            3s       1                 1024      1    1024     1       1s        3m             100

Or, alternatively, in a non-interactive mode, by explicitly setting the values:

$ backyards r cb set backyards-demo/notifications --non-interactive --max-connections=1 --max-pending-requests=1 --max-requests-per-connection=1 --consecutiveErrors=1 --interval=1s --baseEjectionTime=3m --maxEjectionPercent=100
Connections  Timeout  Pending Requests  Requests  RPC  Retries  Errors  Interval  Ejection time  percentage
1            3s       1                 1024      1    1024     5       1s        3m             100

After the command is issued, the circuit breaking settings are fetched and displayed right away.

View circuit breaking configurations 🔗︎

You can list the circuit breaking configurations of a service in a given namespace with the following command:

$ backyards r cb get backyards-demo/notifications
  Connections  Timeout  Pending Requests  Requests  RPC  Retries  Errors  Interval  Ejection time  percentage
  1            3s       1                 1024      1    1024     5       1s        3m             100

By default, the results are displayed in a table view, but it’s also possible to list the configurations in JSON or YAML format with -o json or -o yaml flags.

Monitor circuit breaking 🔗︎

Trigger circuit breaker trips by calling the service from multiple connections and then issue the following command:

$ backyards r cb graph backyards-demo/notifications

You should see something like this:

The first dashboard details the percentage of total requests that were tripped by the circuit breaker. When there are no circuit breaker errors, and your service works as expected, this graph will show 0%. Otherwise, you’ll be able to see what percentage of the requests were tripped by the circuit breaker right away.

The second dashboard provides a breakdown of the trips caused by the circuit breaker by source. If no circuit breaker trips occurred, there will be no spikes in this graph. Otherwise, you’ll see which service caused the circuit breaker to trip, when, and how many times. Malicious clients can be tracked by checking this graph.

Remove circuit breaking configurations 🔗︎

To remove circuit breaking configurations:

$ backyards r cb delete backyards-demo/notifications
INFO[0000] current settings
Connections  Timeout  Pending Requests  Requests  RPC  Retries  Errors  Interval  Ejection time  percentage
1            3s       1                 1024      1    1024     5       1s        3m             100
? Do you want to DELETE the circuit breaker rules? Yes
INFO[0008] circuit breaker rules set to backyards-demo/notifications successfully deleted

Circuit breaking using the Backyards UI 🔗︎

Set circuit breaking configurations 🔗︎

You don’t need to create or edit a Destination Rule resource manually, you can easily change the circuit breaker configurations from the UI. Let’s create a demo circuit breaker.

View circuit breaking configurations 🔗︎

You don’t have to fetch the Destination Rule (e.g. with kubectl) to see the circuit breaker’s configurations, you can see them on the right side of the Backyards UI when you click on the notifications service icon and then click on the circuit breaker tab.

Monitor circuit breaking 🔗︎

With this configuration I’ve just set, when traffic begins to flow from two connections simultaneously, the circuit breaker will start to trip requests. In the Backyards UI, you will see this being vizualized via the graph’s red edges. If you click on the service, you’ll learn more about the errors involved, and will see two live Grafana dashboards which specifically show the circuit breaker trips.

These are live Grafana dashboards customized in order to display circuit breaker-related information.

Remove circuit breaking configurations 🔗︎

You can easily remove circuit breaking configurations with the REMOVE button.