Istio 1.8 has just been released and is one of the best Istio releases so far. The new version contains exciting experimental features, numerous enhancements, as well as deprecations and removals. The core focus of the release, however, is to increase operational stability. In this post, we’ll review what’s new in Istio 1.8, and highlight a few potential snags to look out for when upgrading to the latest version.
Istio 1.8 changes 🔗︎
Caveat: The Istio 1.8.0 release has a few known issues pointed out in the official change notes. Either make sure you won’t be affected by them or wait a few patch releases before upgrading your cluster to make sure these issues are sorted out.
First of all, it’s important that we point out that the supported Kubernetes versions for Istio 1.8 are 1.16, 1.17, 1.18 and 1.19. If you are running Istio 1.7 in your environment, you should already be on at least Kubernetes 1.16 (as that is also the oldest supported K8s version for Istio 1.7). As a result, you should be ready to upgrade to Istio 1.8 without having to upgrade your K8s cluster.
High impact changes (can cause issues when upgrading the mesh):
- Inbound cluster name format changed
- Protocol detection timeout disabled by default
- AuthorizationPolicy has new remoteIpBlocks/notRemoteIpBlocks fields
- Trust Domain Validation enforced by default
- Completely removes Mixer
- Removes support to install add-ons with istioctl
- Istio CoreDNS Plugin Deprecated
- Envoy filter names deprecated
Under the radar changes (smaller changes, but which we believe are interesting/useful):
Other notable changes:
- Helm3 Istio installation
- Observed generation added for Istio resource statuses
- Wasm Grafana dashboards added
- Metrics properly labeled when destination not reached
For the full list of changes please refer to the change notes.
High impact changes 🔗︎
Inbound cluster name format changed 🔗︎
Previously, the Envoy inbound cluster name format looked like this:
An example cluster name with this format was:
For Istio 1.8.0 this format has changed, the service port name and the service’s hostname are now omitted, the new format looks like this:
For the example above the new format is simply:
There were issues when multiple services selected the same container port in the same pod and this change is an attempt to solve those issues.
The offical release note states that:
"For most users, this is an implementation detail, and will only impact debugging or tooling that directly interacts with Envoy configuration."
Well, from what we’ve seen so far this change causes undesirable behaviour in the following scenario. When there are two services, which use the same service port number and select the same pod, but they are targeting different port numbers inside the pod. Here’s an example when a backend and a frontend container are in the same pod:
- name: http
- name: http
With this setup you’ll see the following warning in the istiod logs when using the upstream
docker.io/istio/pilot:1.8.0 docker image:
"message": "Duplicate cluster inbound|80|| found while pushing CDS"
And the issue will be that when calling either the
backyards-web or the
backyards service that in both cases the same target port will be called inside the pod.
So it will be impossible to reach the other container port through its service, which is definitely undesired behaviour and seems to be a bug.
We’re planning to open an issue about this with more details to sort out this issue upstream. A workaround for this problem can be to use different service port numbers to avoid such conflicts. What we did in Backyards is that we reverted this pilot change in our own docker image, which we use in Backyards (now Cisco Service Mesh Manager), so that this issue can never happen.
Protocol detection timeout disabled by default 🔗︎
In the past, Istio introduced a detection timeout for certain “server first” protocols, such as the protocol used by MySQL, which triggers when there are no initial bytes of traffic to sniff. This protocol detection timeout has been disabled by default to avoid potential traffic failures on slow connections or in certain cases increased latency.
If you have misconfigured “server first” protocols, this change will cause immediate connection timeouts.
To avoid issues involving “server first” protocols, make sure to explicitly specify the protocol for your service through one of the following methods:
- By the name of the port:
- In Kubernetes 1.18+, by the
Here are examples of two possible declarations:
- number: 443
- number: 3306
For more info, please refer to Istio’s documentation.
AuthorizationPolicy has new remoteIpBlocks/notRemoteIpBlocks fields 🔗︎
remoteIpBlocks/notRemoteIpBlocks fields on the AuthorizationPolicy resource are used to allow/deny requests based on the original source IP of a request, populated from the
X-Forwarded-For header or from a proxy protocol.
ipBlocks/notIpBlocks fields on the
AuthorizationPolicy resource are now used to allow/deny requests based on the source address of IP packets that arrive at the sidecar.
If you’d like to allow/deny requests based on their original source IP addresses (either because you use the
X-Forwarded-For header or a proxy protocol), then update your
AuthorizationPolicy resources to use the new
remoteIpBlocks/notRemoteIpBlocks fields instead of the
Trust Domain Validation enforced by default 🔗︎
By default, the Trust Domain Validation feature will reject any incoming requests to sidecars, if the request is neither from the same trust domain nor in the
If you want to allow traffic to your mesh from different trust domains on Istio 1.8, you need to add it to your
TrustDomainAliases list, otherwise it will be rejected.
Complete removal of Mixer 🔗︎
Mixer has been the control plane component to which application sidecar proxies reported their telemetry data since the early days of Istio. Mixer had the potential to cause resource consumption and latency issues in larger environments, which is why a new Mixerless telemetry model was introduced:
- first, as an experimental feature in Istio 1.3
- then in Istio 1.4 the feature was made available under the name Telemetry V2
- in Istio 1.5 Telemetry V2 was turned on by default
- in Istio 1.6 the original Mixer-based telemetry model was deprecated in favor of the new Telemetry V2 model
Now, in Istio 1.8, all Mixer-related features and functionality have been removed. So, if you haven’t already, you need to adapt to the Telemetry V2 model before upgrading to 1.8.
Remove support to install add-ons with istioctl 🔗︎
Open-source software that enhances Istio’s capabilities (like Prometheus, Grafana, Zipkin, Jaeger, and Kiali) can no longer be installed with
istioctl, and it is recommended that they be installed separately.
If you want the easiest way to install all these integrated components - with a production-ready setup and with many additional features - check out Backyards (now Cisco Service Mesh Manager).
Istio CoreDNS Plugin deprecated 🔗︎
The Istio sidecar proxy now provides native support for DNS resolution with ServiceEntries, which is why the Istio CoreDNS plugin is deprecated in Istio 1.8 and will be removed in a future release.
Envoy filter names deprecated 🔗︎
For the EnvoyFilter API, it is recommended that you use the new Envoy filter names, as some filter names were deprecated and will be removed in a future release.
Experimental features 🔗︎
Smart DNS proxying 🔗︎
DNS is essential for any cloud native application, but the way it works in Kubernetes results in a few challenges in Istio:
- It can be difficult for VMs outside the Kubernetes cluster to provide DNS resolution to route services inside the cluster
- If services do not have unique virtual IPs and are on the same port (e.g. databases), then DNS servers cannot distinguish between them
- In multi-cluster meshes we cannot resolve the IP addresses of a service on another cluster (this is why stub services are likewise created on remote clusters)
To solve all these issues for Istio 1.8, there is a DNS proxy implemented in the Istio sidecar agent, written in GO. This DNS proxy is dynamically updated by Istiod (based on services and ServiceEntries on the cluster) and it acts as a cache. It will try to resolve the required hostname based on its cache, and only turn to external DNS servers when the cache does not have the hostname registered.
This is an exciting new feature, which can be tried right now, but is not enabled by default. To read an in-depth explanation on this new feature, make sure to check out this blog post.
Auto registration for VMs 🔗︎
To register VMs for the Istio mesh, so far the only way was to manually create a WorkloadEntry resource. This was previously a manual step which registered new VMs, in contrast to how the same process was automated for nodes in a Kubernetes cluster. It could cause issues, for example, in case of auto scaling VMs, a circumstance in which it’s is not easy to register new VMs automatically.
There is a pre-alpha feature that solves this problem, making it possible to auto register VMs.
This is done by a new WorkloadGroup resource, which is responsible for automatically creating the
WorkloadEntry resources when new VMs are available.
The Istio 1.8 release laid the groundwork for a wide variety of features around VM support, which should become apparent in future releases as it matures.
Integrate with external CAs with K8s CSR API 🔗︎
Beginning with Kubernetes 1.18, there is a CSR API feature, which automates the request and retrieval of certificates from a Certificate Authority (CA). In Istio 1.8, experimental support has been added to allow Istio to integrate with external CAs, using the Kubernetes CSR API.
This feature only lays the foundation necessary to work with external CAs, now only istiod is supported (and Google CA somewhat), but new implementations should come in future releases as this feature matures both on the Istio and Kubernetes side.
Under the radar changes 🔗︎
Istiod to handle gateway certificates 🔗︎
Previously, the gateways read the certificates for external communications from Kubernetes secrets. These gateways were usually public facing entities, and, if they were compromised, the certificates could be read from the Kubernetes secrets that compromised the whole cluster.
In Istio 1.8 gateways fetch the certs from Istiod. They do not even have the privileges they need to read secrets from the cluster to reduce potential collateral damage.
This feature is interesting because it paves the way for us to be able to fetch certificates from external secret stores (e.g. Vault or cloud key stores) in future releases, which would be convenient for many Istio users already using other secret stores. Furthermore, this is a totally backward compatible change, and does not require any user interaction.
EnvoyFilter enhancements 🔗︎
To specify the order of HTTP_ROUTE specs for the Envoy configurations, new
INSERT_AFTER operations were added to the EnvoyFilter API.
To be able to override http and network filters, a new
REPLACE operation was added to the EnvoyFilter API.
Debug archive option added 🔗︎
istioctl bug-report command was added to be able to get an archive of the Istio cluster - mainly for debugging purposes.
Other notable changes 🔗︎
Install Istio with Helm3 🔗︎
Almost two years ago now, when we open sourced the Banzai Cloud Istio operator, we did so partly because the only option for installing Istio was through Helm, which had its issues.
After a while,
istioctl, and then the official Istio operator, were introduced, and, slowly, the method of installing by Helm became unsupported.
It turned out, however, that there was still demand among Istio users for a way of installing and upgrading Istio with Helm (mostly for users who had already deployed all their apps with Helm), and because of that, Helm 3 support was added for Istio 1.8.
As a result, it is possible to install and upgrade Istio with Helm 3, but I would note here that this is NOT recommended. Only in-place upgrades are supported with Helm, whilst the canary upgrade model is the recommended flow today.
Observed generation added for Istio resource status 🔗︎
Since the 1.6 release, there has been an interesting alpha feature in Istio, in which various useful pieces of information about Istio resources are exposed in their
These fields include, but are not limited to resource readiness, the number of data plane instances associated with the resource, and validation messages.
In Istio 1.8 a new observed generation field is also present, which, when it matches the generation in the resource’s metadata, indicates that all Istio updates have been completed. This is useful for detecting when requested changes to an Istio configuration have been served and are ready to receive traffic.
Wasm Grafana dashboards added 🔗︎
As WebAssembly (Wasm) gains more and more traction in Istio, it is also gradually becoming required in order to monitor it properly. Therefore, it can come handy to use the newly added Grafana dashboard on Wasm related metrics.
Metrics properly labeled when a destination is not reached 🔗︎
Previously, there were cases in which requests did not reach a destination proxy (e.g. fault injection, circuit breaking, no proxy injected etc.), and there were no destination labels to identify what would have been the end target of the request. This is remedied in 1.8, which is a welcome bug fix.
Istio 1.8 largely increases operational stability and also lays the foundation for exciting new features in future releases.
If you’d like to kickstart your Istio experience, try out Backyards (now Cisco Service Mesh Manager), our Istio distribution. Backyards (now Cisco Service Mesh Manager) operationalizes the service mesh to bring deep observability, convenient management, and policy-based security for modern container-based applications.
Check out Backyards in action on your own clusters!