Fault Injection

Fault injection is a system testing method which involves the deliberate introduction of network faults and errors into a system. It can be used to identify design or configuration weaknesses and to ensure that the system is able the handle faults and recover from error conditions.

With Backyards, failures can be injected at the application layer to test the resiliency of the services. You can configure faults to be injected into requests that match specific conditions to simulate service failures and higher latency between services.

• Delay can delay requests before forwarding, emulating various failures such as network issues, an overloaded upstream service, etc.

• Abort can abort HTTP request attempts and return error codes to a downstream service, giving the impression that the upstream service is faulty.

Backyards is using Istio’s - and therefore Envoy’s - fault injection feature under the hood

Try it out! 🔗︎

Using the Backyards CLI 🔗︎

These examples work out of the box with the demo application packaged with Backyards. Change the service name and namespace to match your service.

Setting fault injection requires that at least one routing rule is present for the service. First, make sure you have one:

backyards routing route get backyards-demo/payments

Settings for backyards-demo/payments

Matches  Routes         Redirect  Timeout  Retry  Rewrite  Mirror To
any      100% payments  -         -        -      -        -

If you don’t have one (the previous command returned the message no http route found for backyards-demo/payments), then create a new routing rule that matches any incoming requests:

backyards routing route set backyards-demo/payments -m any -d payments -w 100

To learn more about configuring routing rules, head to the routing section.

Now you can create fault injection rules like these:

1. Abort 50% of all HTTP requests with a 503 status code:

   backyards routing fault-injection set backyards-demo/payments \
   --match any \
   --abort-percentage 50 \
   --delay-fixed-delay 5s \
   --delay-percentage 0 \
   --abort-status-code 503 \
   --non-interactive
   

2. Delay 50% of all HTTP requests by 5 seconds:

   backyards routing fault-injection set backyards-demo/payments \
   --match any \
   --abort-percentage 0 \
   --delay-fixed-delay 5s \
   --delay-percentage 50 \
   --abort-status-code 503 \
   --non-interactive
   

To delete the fault injection settings for the service, use the delete command:

backyards routing fault-injection delete backyards-demo/payments -m any --non-interactive

Or if you want to delete the whole routing rule:

backyards routing route delete backyards-demo/payments --match any --non-interactive

Tip 1: all CLI commands and switches have short names, check the CLI docs to get to know them.

Tip 2: the CLI has an interactive mode: try the above commands with --match any as the only argument

Using the Backyards UI 🔗︎

Fault injection can also be configured from the dashboard. Select the service on the Services or the Topology view, and create a new rule on the traffic management tab: