Upgrade to Backyards 1.5

The Backyards upgrade process consists of two phases:

1. In-place upgrade of the Backyards control plane
2. Canary upgrade of the Istio control plane

Upgrade the Backyards control plane 🔗︎

1. Upgrade Backyards CLI to the version you want to install.

   Note: For details on a specific Backyards release, see the release announcements.

   • Backyards 1.4 announcement
   • Backyards 1.5 announcement

2. Deploy a new version of Backyards. The following command in-place upgrades the Backyards control plane. It also installs the new Istio control plane, but the applications keep using the old control plane until you upgrade the Istio control plane.

   • In case you want to have custom settings for your Istio control plane, you can provide that during the installation:

     $ backyards install -a --istio-cr-file <custom-istio-cr-file.yaml>
     

   • Otherwise, simply run:

     $ backyards install -a
     

3. Validate that the Backyards control plane is upgraded and already uses the new Istio control plane:

   $ kubectl get po -n=backyards-system -L istio.io/rev
   NAME                                            READY   STATUS    RESTARTS   AGE     REV
   backyards-74b79fbd48-xfpg5                      3/3     Running   1          6m48s   cp-v18x.istio-system
   backyards-als-57568857c9-dm9hf                  2/2     Running   0          6m48s   cp-v18x.istio-system
   backyards-grafana-55b54d8895-kvdhm              3/3     Running   0          6m48s   cp-v18x.istio-system
   backyards-ingressgateway-64f4df4555-shpcg       1/1     Running   0          6m46s   cp-v18x.istio-system
   backyards-kubestatemetrics-8fd988999-vsrh7      2/2     Running   0          6m48s   cp-v18x.istio-system
   backyards-leo-6d7f474d9-wdhg2                   2/2     Running   1          6m48s   cp-v18x.istio-system
   backyards-prometheus-operator-9d54d6796-929tc   3/3     Running   1          6m48s   cp-v18x.istio-system
   backyards-sre-alert-exporter-8577475bd9-vhfn6   2/2     Running   1          6m48s   cp-v18x.istio-system
   backyards-sre-controller-f846b9448-cph44        2/2     Running   1          6m48s   cp-v18x.istio-system
   backyards-tracing-7c5c88cf-mlnc8                2/2     Running   0          6m47s   cp-v18x.istio-system
   prometheus-backyards-prometheus-0               4/4     Running   1          6m3s    cp-v18x.istio-system
   prometheus-node-exporter-crpq2                  1/1     Running   0          12m
   prometheus-node-exporter-j6bkx                  1/1     Running   0          12m
   prometheus-node-exporter-n2sbh                  1/1     Running   0          12m
   

Upgrade the Istio control plane 🔗︎

With Backyards, you can upgrade the Istio control plane to a new version using a process similar to canary upgrades.

Note: Upgrading to a new version of Backyards involves upgrading Istio as well.

Let’s suppose that you have Backyards 1.4 with an Istio 1.7 control plane running in your cluster and you’d like to upgrade to Backyards 1.5 with an Istio 1.8 control plane. Here are the high level steps necessary to accomplish this using the canary upgrade flow.

1. Deploy an Istio 1.8 control plane next to the Istio 1.7 control plane.
2. Migrate the data plane gradually to the new control plane.
3. When migration is fully finished, uninstall the Istio 1.7 control plane.

This canary upgrade model is much safer than the in-place upgrade workflow, because it can be done gradually. If any issues arise, they probably affect only a small portion of your applications, and if something happens, it is much easier to rollback only a part of your applications to use the original control plane.

This upgrade flow gives you more flexibility and confidence when making Istio upgrades, dramatically reducing the potential for service disruptions.

Limitations 🔗︎

Currently, you can migrate your applications to the new control plane on a per namespace basis.

Steps 🔗︎

To upgrade the Istio control plane to a newer version, complete the following steps. The examples in this procedure assume that you are upgrading from Istio 1.7 to Istio 1.8. Adjust the version numbers as needed for your environment.

Complete the pre-upgrade checks for the Istio control plane 🔗︎

1. Verify that the Backyards upgrade has deployed the new Istio control plane. Run the following command: there should be two istio-operator- and two istiod- entries in the output, one corresponding to Istio version 1.8, one to 1.7.

   $ kubectl get po -n=istio-system
   NAME                                            READY   STATUS    RESTARTS   AGE
   istio-ingressgateway-5df9c6c68f-vnb4c           1/1     Running   0          3m47s
   istio-operator-v17x-0                           2/2     Running   0          3m59s
   istio-operator-v18x-0                           2/2     Running   0          3m43s
   istiod-cp-v17x-c58e46c49-e16qs                  1/1     Running   0          2m45s
   istiod-cp-v18x-b58f95c49-r24tr                  1/1     Running   0          3m17s
   

   The backyards install -a command has installed the new Istio control plane, but the applications still use the old control plane.

Complete the pre-upgrade checks for the Istio configurations 🔗︎

1. The Istio 1.8 release removes all Mixer-related features and functionality. If you haven’t already done so, start using the Telemetry V2 model before upgrading to 1.8.

2. Protocol detection timeout is disabled by default in Istio 1.8 to avoid potential traffic failures on slow connections or in certain cases of increased latency. If you have misconfigured “server first” protocols, this change will cause immediate connection timeouts.

   To avoid issues involving “server first” protocols, make sure to explicitly specify the protocol for your service using one of the following methods:

   • By the name of the port: name: <protocol>[-<suffix>].
   • In Kubernetes 1.18+, by the appProtocol field: appProtocol: <protocol>.

   The following examples show the two possible declarations:

   kind: Service
   metadata:
       name: httpbin
   spec:
       ports:
       - number: 443
       name: https-web
       - number: 3306
       name: db
       appProtocol: mysql
   

   For more details, see the Istio’s documentation.

3. The new remoteIpBlocks/notRemoteIpBlocks fields of the AuthorizationPolicy resource are used to allow/deny requests based on the original source IP of a request, populated from the X-Forwarded-For header or from a proxy protocol.

   The ipBlocks/notIpBlocks fields on the AuthorizationPolicy resource are now used to allow/deny requests based on the source address of IP packets that arrive at the sidecar.

   If you’d like to allow/deny requests based on their original source IP addresses (either because you use the X-Forwarded-For header or a proxy protocol), then update your AuthorizationPolicy resources to use the new remoteIpBlocks/notRemoteIpBlocks fields instead of the ipBlocks/notIpBlocks fields.

4. By default, the Trust Domain Validation feature rejects any incoming requests to sidecars, if the request is neither from the same trust domain nor in the TrustDomainAliases list.

   If you want to allow traffic to your mesh from different trust domains on Istio 1.8, you need to add the domains to your TrustDomainAliases list, otherwise the traffic will be rejected.

Migrate the gateways 🔗︎

1. Move the gateways so they use the new Istio control plane.

   $ kubectl -n istio-system patch mgw <your-gateway> --type=merge --patch "
   spec:
     istioControlPlane:
       name: "<name-of-the-new-istio-control-plane>"
       namespace: "istio-system"
   "
   

   For example:

   $ kubectl -n istio-system patch mgw <your-gateway> --type=merge --patch "
   spec:
     istioControlPlane:
       name: "cp-v18x"
       namespace: "istio-system"
   "
   

Complete the pre-migration checks 🔗︎

1. Verify that your namespaces are labeled for the Istio control plane you want to upgrade from.

   $ kubectl get ns <name-of-your-namespace> -L istio.io/rev
   NAME                       STATUS   AGE     ISTIO-REV
   <name-of-your-namespace>   Active   2m21s   cp-v17x.istio-system
   

2. Make sure that the pods of your applications use the sidecar proxy of the Istio version you want to upgrade from, for example, 1.7.

   $ POD_NAME=$(kubectl get pods -n <name-of-your-namespace> -l app=<name-of-your-application> -o=jsonpath='{.items[0].metadata.name}')
   $ kubectl get po -n=<name-of-your-namespace> $POD_NAME -o yaml | grep istio-proxyv2:
       image: banzaicloud/istio-proxyv2:1.7.3-bzc
       image: docker.io/banzaicloud/istio-proxyv2:1.7.3-bzc
   

Migrate your data plane to the new control plane 🔗︎

Note that the two Istio control planes share trust, because they use the same certificates for encrypted communication. Therefore, when an application pod that still uses the old Istio’s control plane calls another pod that already uses the new Istio’s control plane, the encrypted communication will succeed because of that shared trust. This means that you can migrate the data plane on a namespace by namespace basis, the communication between the pods won’t be affected.

1. Label the namespace of your application you want to migrate for sidecar injection with the new control plane’s label. The label name is: istio-control-plane-cr-name.istio-control-plane-cr-namespace, for example, cp-v17x.istio-system, or cp-v18x.istio-system.

   $ kubectl label ns <name-of-your-namespace> istio-injection- istio.io/rev=cp-v18x.istio-system --overwrite
   

2. Make sure that the labeling is correct.

   $ kubectl get ns <name-of-your-namespace> -L istio.io/rev
   NAME                       STATUS   AGE   REV
   <name-of-your-namespace>   Active   19m   cp-v18x.istio-system
   

3. Restart the pod in the namespace.

   $ kubectl rollout restart deployment -n <name-of-your-namespace>
   

4. Wait for the rollout to finish, then make sure that the new sidecar proxy is used for one of the application pods.

   $ POD_NAME=$(kubectl get pods -n <name-of-your-namespace> -l app=<name-of-your-application> -o=jsonpath='{.items[0].metadata.name}')
   $ kubectl get po -n=<name-of-your-namespace> $POD_NAME -o yaml | grep istio-proxyv2:
       image: banzaicloud/istio-proxyv2:1.8.0-bzc
       image: docker.io/banzaicloud/istio-proxyv2:1.8.0-bzc
   

5. Test your application to verify that it works with the new control plane as expected. In case of any issues, refer to the rollback section to roll back to the original Istio control plane.

6. Repeat this procedure for other namespaces that you want to migrate. In the meantime, make sure that your applications can still communicate with each other.

Roll back the data plane to the old control plane in case of issues 🔗︎

1. If there is a problem and you want to roll the namespace back to the old control plane, set the istio.io/rev label on the namespace to point to the old Istio control plane, and restart the pod using the kubectl rollout restart deployment command:

   $ kubectl label ns <name-of-your-namespace-with-issues> istio.io/rev=cp-v17x.istio-system
   $ kubectl rollout restart deployment -n <name-of-your-namespace-with-issues>
   

Uninstall the old Istio control plane 🔗︎

1. After you have migrated all your namespaces and verified that your applications work, check that the only pod using the old control plane is the old istiod pod.

   It’s recommended that you take some time to make sure that everything is working on the new control plane before turning off the old one.

   $ kubectl get po -A -l istio.io/rev -L istio.io/rev
   NAMESPACE          NAME                                            READY   STATUS    RESTARTS   AGE     REV
   backyards-system   backyards-74b79fbd48-xfpg5                      3/3     Running   1          20m     cp-v18x.istio-system
   backyards-system   backyards-als-57568857c9-dm9hf                  2/2     Running   0          20m     cp-v18x.istio-system
   backyards-system   backyards-grafana-55b54d8895-kvdhm              3/3     Running   0          20m     cp-v18x.istio-system
   backyards-system   backyards-ingressgateway-64f4df4555-shpcg       1/1     Running   0          20m     cp-v18x.istio-system
   backyards-system   backyards-kubestatemetrics-8fd988999-vsrh7      2/2     Running   0          20m     cp-v18x.istio-system
   backyards-system   backyards-leo-6d7f474d9-wdhg2                   2/2     Running   1          20m     cp-v18x.istio-system
   backyards-system   backyards-prometheus-operator-9d54d6796-929tc   3/3     Running   1          20m     cp-v18x.istio-system
   backyards-system   backyards-sre-alert-exporter-8577475bd9-vhfn6   2/2     Running   1          20m     cp-v18x.istio-system
   backyards-system   backyards-sre-controller-f846b9448-cph44        2/2     Running   1          20m     cp-v18x.istio-system
   backyards-system   backyards-tracing-7c5c88cf-mlnc8                2/2     Running   0          20m     cp-v18x.istio-system
   backyards-system   prometheus-backyards-prometheus-0               4/4     Running   1          19m     cp-v18x.istio-system
   demo-a             app-a-989f8b7d5-r7gxp                           2/2     Running   0          2m46s   cp-v18x.istio-system
   demo-b             app-b-6fd9976f8-8fnsw                           2/2     Running   0          91s     cp-v18x.istio-system
   istio-system       istio-ingressgateway-74cffbd5d5-58kgl           1/1     Running   0          4m43s   cp-v18x.istio-system
   istio-system       istiod-65484c4dbc-dxchq                         1/1     Running   0          20m     mesh.istio-system
   istio-system       istiod-cp-v18x-556b8f9fdc-8zj2t                 1/1     Running   0          20m    cp-v18x.istio-system
   istio-system       istiod-cp-v17x-b58f95c49-xxpwm                  1/1     Running   0          20m     cp-v17x.istio-system
   

2. Uninstall the old Istio control plane.

   $ kubectl delete -n=istio-system istio <name-of-old-istio-control-plane>
   

3. Delete the Istio operator related to the old control plane, for example:

   kubectl delete statefulset -n=istio-system istio-operator-v17x