Aspen Mesh Carrier-Grade 1.11.5-am1 release notes#

Introduction#

These release notes describe the differences between Aspen Mesh Carrier-Grade 1.9.9-am1 and 1.11.5-am1.

This release includes security updates and important bug fixes.

Important

Secure ingress is no longer supported. Therefore, the number of Custom Resource Definitions (CRDs) installed or upgraded is now 15.

Supported platforms#

This release is officially supported on these platforms and versions:

Platform

Version

Recommended Helm version

OpenShift

4.7

3.6

Security updates#

Istio 1.11.5#

(No security updates)

Istio 1.11.4#

(No security updates)

Istio 1.11.3#

(No security updates)

Istio 1.11.2#

(No security updates)

Istio 1.11.1#

(The same security updates shown for open source Istio 1.10.4 were included in open source Istio 1.11.1.)

Istio 1.11.1 proxy (Envoy)#

(The same security updates shown for the open source Istio 1.10.4 sidecar proxy were included in the open source Istio 1.11.1 sidecar proxy.)

Istio 1.11#

(No security updates)

Istio 1.10.6#

(No security updates)

Istio 1.10.5#

(No security updates)

Istio 1.10.4#

(The following security updates in open source Istio 1.10.4 were included in Aspen Mesh 1.9.8-am1: CVE-2021-39155 (CVE-2021-32779) and CVE-2021-39156.)

Istio 1.10.4 proxy (Envoy)#

  • CVE-2021-32780 (CVSS score 8.6, High): Envoy contains a remotely exploitable vulnerability where an untrusted upstream service may cause Envoy to terminate abnormally by sending the GOAWAY frame followed by the SETTINGS frame with the SETTINGS_MAX_CONCURRENT_STREAMS parameter set to 0. Note: this vulnerability does not impact downstream client connections.

(The following security updates in the open source Istio 1.10.4 sidecar proxy were included in Aspen Mesh 1.9.8-am1: CVE-2021-32777, CVE-2021-32778, and CVE-2021-32781.)

Istio 1.10.3#

(No security updates)

Istio 1.10.2#

  • CVE-2021-34824 (CVSS score 9.1, Critical): Istio contains a remotely exploitable vulnerability where credentials specified in the Gateway and DestinationRule credentialName field can be accessed from different namespaces. See the ISTIO-SECURITY-2021-007 bulletin for more details.

Istio 1.10.1#

(No security updates)

Istio 1.10#

(No security updates)

Aspen Mesh features#

(No security updates)

Other changes#

Istio 1.11.5#

  • Fixed istiod deployment respect values.pilot.nodeSelector. (Issue #36110)

  • Fixed the in-cluster operator can’t prune resources when the Istio control plane have active proxies connected. (Issue #35657)

  • Fixed the release tar URL by adding the patch version.

  • Fixed LbEndpointValidationError.LoadBalancingWeight: value must be greater than or equal to 1 from Envoy when multi-network gateways are configured via MeshNetworks.

  • Fixed workload name metric labels are not correctly populated for CronJob at k8s 1.21+. (Issue #35563)

Istio 1.11.4#

  • Fixed VMs are able to use a revisioned control plane specified by --revision on the istioctl x workload entry command.

  • Fixed an issue when creating a Service and Gateway at the same time, causing the Service to be ignored. (Issue #35172)

  • Fixed an issue causing stale endpoints for service entry selecting pods (Issue #35404)

Istio 1.11.3#

  • Updated to allow specifying NICs that bypassing traffic capture in Istio iptables. (Issue #34753)

  • Added values to the Istio Gateway Helm charts for configuring annotations on the ServiceAccount. Can be used to enable IAM Roles for Service Accounts on AWS EKS.

  • Fixed istioctl analyze command to not output [IST0132] message when analyzing the gateway of the virtual service. (Issue #34653)

  • Fixed a bug using a Service’s pointer address to get its instances in the case where a sidecar’s egress listener has a port.

  • Fixed a bug in the “image: auto” analyzer causing it to fail to take into account the Deployment namespace. (Issue #34929)

  • Fixed istioctl x workload command output to set the correct discoveryAddress for revisioned control-planes. (Issue #34058)

  • Fixed gateway analyzer message reporting if there is no selector in the gateway spec. (Issue #35093)

  • Fixed an issue causing memory to not be freed after XDS clients disconnect.

  • Fixed an issue occurring when multiple VirtualServices with the same name exist in different namespaces. (Issue #35127)

Istio 1.11.2#

  • Improved istioctl install to give more details during installation failures.

  • Added support for gRPC configuring workloads via xDS without an Envoy proxy.

  • Added two mutually-exclusive flags to istioctl x workload entry configure

    • --internal-ip configures the VM workload with a private IP address used for workload auto registration and health probes.

    • --external-ip configures the VM workload with a public IP address used for workload auto registration. Meanwhile, it configures health probes to be performed through localhost by setting the environment variable REWRITE_PROBE_LEGACY_LOCALHOST_DESTINATION to true. (Issue #34411)

  • Added topology label topology.istio.io/network to IstioEndpoint if it does not exist in pod/workload label.

  • Added a configuration FILE_DEBOUNCE_DURATION that allows users to configure the duration SDS server should wait after it sees first file change event. This is useful in File mounted certificate flows to ensure key and cert are fully written before they are pushed to Envoy. Default is 100ms.

  • Fixed unexpected info logs for Istio when using command line tool istioctl profile diff and istioctl profile dump.

  • Fixed issue of deployment analyzer ignoring service namespaces during the analysis process.

  • Fixed DestinationRule updates not triggering an update for AUTO_PASSTHROUGH listeners on gateways. (Issue #34944)

Istio 1.11.1#

(No changes)

Istio 1.11#

  • Promoted CNI to beta. (Issue #86)

  • Improved resolution of headless services via in-agent DNS to include endpoints from other clusters that are on the same network. (Issue #27342)

  • Improved usage of AUTO_PASSTHROUGH Gateways to no longer require configuring the ISTIO_META_ROUTER_MODE environment variable on the gateway deployment; instead, it is automatically detected. (Issue #33127)

  • Improved CNI network plugin to send logs to the CNI DaemonSet. This allows viewing CNI logs using kubectl logs, instead of looking at kubelet logs. (Issue #32437)

  • Improved service conflict resolution to favor Kubernetes Services over ServiceEntries with the same hostname.

  • Updated CNI install container and race condition repair container are combined into one container. (Issue #33712)

  • Updated the Istiod debug interface to be only accessible over localhost or with proper authentication (mTLS or JWT). The recommended way to access the debug interface is through istioctl experimental internal-debug, which handles this automatically.

  • Added the shutdownDuration flag to pilot-discovery so that users can configure the duration istiod needs to terminate gracefully. The default value is 10s.

  • Added an environment variable PILOT_STATUS_UPDATE_INTERVAL that is the interval to update the XDS distribution status and its default value is 500ms.

  • Added the HTTP endpoint localhost:15004/debug/<typeurl> to the Istio sidecar agent. GET requests to that URL will be resolved by sending an xDS discovery “event” to istiod. This can be disabled by setting the following in the Istio Operator: meshConfig.defaultConfig.proxyMetadata.PROXY_XDS_DEBUG_VIA_AGENT=false. (Issue #22274)

  • Added support for overriding the locality of the WorkloadGroup template in an auto registered WorkloadEntry. Locality overrides can be passed in through Envoy bootstrap configuration. (Issue #33426),(Issue #33426)

  • Added new metric for tracking distribution of configuration resource sizes being pushed by istiod. (Issue #31772)

  • Added experimental support for the Kubernetes Multi-Cluster Services (MCS) host (clusterset.local). This feature is off by default, but can be enabled by setting the following environment variables for your Istiod deployment: ENABLE_MCS_HOST and ENABLE_MCS_SERVICE_DISCOVERY. When enabled Istio will include the MCS host as a domain in the service’s HTTP route. Additionally, Istio will support the MCS host during a DNS lookup. For now, the MCS host is just an alias for cluster.local and resolves to the same service IP. Future work will give the MCS host a separate IP as is defined by the MCS spec. (Issue #33949)

  • Added experimental support for controlling service endpoint discoverability with Kubernetes Multi-Cluster Services (MCS). This feature is off by default, but can be enabled by setting the ENABLE_MCS_SERVICE_DISCOVERY flag in Istio. When enabled, Istio will make service endpoints only discoverable from within the same cluster by default. To make the service endpoints within a cluster discoverable throughout the mesh, a ServiceExport CR must be created within the same cluster as the service endpoints. this process can be automated by enabling the Istio flag ENABLE_MCS_AUTOEXPORT. With this enabled, Istio will automatically create ServiceExport in all clusters for each service. (Issue #29384)

  • Fixed an issue to enableCoreDump using the sidecar annotation. (reference) (Issue #26668)

  • Fixed where both inbound and outbound apps were unable to intercept traffic when using podIP in TPROXY interception mode. (Issue #31095)

  • Fixed an issue where subject alternate names specified in service entry are not considered while building TLS context. (Issue #32539)

  • Fixed a bug where multiple gateways on the same port with SIMPLE and PASSTHROUGH modes was not working correctly. (Issue #33405)

  • Fixed a bug where Istio config generation fails when the sum of endpoint weights was over uint32 max. (Issue #33536)

  • Fixed smart DNS support in Istio CNI. (Issue #29511)

  • Fixed a bug in Kubernetes Ingress causing paths with prefixes of the form /foo to match the route /foo/ but not the route /foo.

  • Fixed an issue allowing a ServiceEntry to act as an instance in other namespaces.

  • Fixed an issue causing proxies to send Transfer-Encoding headers with 1xx and 204 responses.

  • Fixed reconciliation logic in the validation webhook controller to rate-limit the retries in the loop. This should drastically reduce churn (and generated logs) in cases of misconfiguration. (Issue #32210)

  • Optimized generated routing configuration to merge virtual hosts with the same routing configuration. This improves performance for Virtual Services with multiple hostnames defined. (Issue #28659)

  • Added validation for the jwks field in the request authentication policy. (Issue #33053)

  • Updated Prometheus telemetry behavior for inbound traffic to disable host header fallback by default. This will prevent traffic coming from out-of-mesh locations from potentially polluting the destination_service dimension in metrics with junk data (and exploding metrics cardinality). With this change, it is possible that users relying on host headers for labeling the destination service for inbound traffic from out-of-mesh workloads will see that traffic labeled as unknown. The behavior can be restored by modifying Istio configuration to remove the disable_host_header_fallback: true configuration.

  • Added support for Apache SkyWalking tracer. Now you can run the istioctl dashboard skywalking command to view SkyWalking dashboard UI. (Issue #32588)

  • Added a new metric to istiod to report server uptime.

  • Added a new metric (istiod_managed_clusters) to istiod to track the number of clusters managed by an istiod instance.

  • Fixed Prometheus metrics merging to correctly handle the case where the application metrics are exposed as OpenMetrics. (Issue #33474)

  • Promoted external control plane to beta. (Pull Request #93)

  • Improved the installation of Istio on remote clusters using an external control plane. The istiodRemote component now includes all of the resources needed for either a basic remote or config cluster. (Issue #33455)

  • Improved the size of container images, decreasing each image by up to 50Mb. As a result, the linux-tools-generic package, as well as dependencies (including python) are no longer installed.

  • Updated the base image versions to be built on ubuntu:focal and debian10 (for distroless).

  • Updated Jaeger addon to version 1.22.

  • Fixed the upgrade and downgrade message of the control plane. (Issue #32749)

  • Removed the empty caBundle default value from Chart to allow a GitOps approach. (Issue #33052)

  • Promoted the istioctl experimental revision tag command group to istioctl tag.

  • Added --workloadIP flag to istioctl x workload entry configure, which sets the configuration for the workload IP that the sidecar proxy uses to auto register a workload Entry. Usually required when the VM workloads aren’t in the same network as the primary cluster to which they register. (Issue #32462)

  • Added --dry-run flag for istioctl x uninstall. (Issue #32513)

  • Added istioctl proxy-config bootstrap now has a short output option (-o short) that shows the Istio and Envoy version summary. (Issue #21517)

  • Added a new analyzer to check for image: auto in Pods and Deployments that will not be injected.

  • Added support for auto-completion of the namespace for istioctl.

  • Added istioctl now supports completion for Kubernetes pods, services.

  • Added --vklog option to enable verbose logging in client-go. (Issue #28231)

  • Fixed user-agent in all Istio binaries to include version.

Istio 1.10.6#

  • Fixed an issue that prevented the in-cluster operator from pruning resources when the Istio control plane had active proxies connected. (Issue #35657)

  • Fixed an issue causing workload name metric labels to be incorrectly populated for CronJobs for k8s 1.21+. (Issue #35563)

Istio 1.10.5#

  • Improved istioctl install to give more details when encountering installation failures.

  • Added values to the Istio Gateway Helm charts for configuring ServiceAccount annotations. Can be used to enable IAM Roles for Service Accounts on AWS EKS. (Issue #34837)

  • Fixed an issue causing istioctl profile diff and istioctl profile dump to output unexpected info logs.

  • Fixed an issue causing istioctl analyze to show an unexpected IST0132 message when analyzing the gateway associated with a virtual service. (Issue #34653)

  • Fixed an issue causing the deployment analyzer to ignore service namespaces during the analysis process.

  • Fixed an issue resulting in DestinationRule updates not triggering updates for AUTO_PASSTHROUGH listeners on gateways. (Issue #34944)

  • Fixed an issue causing memory to not be freed after XDS clients disconnect.

Istio 1.10.4#

  • Added a validator to prevent for empty regex match. (Issue #34065)

  • Added a new analyzer to check for image: auto in Pods and Deployments that will not be injected.

  • Fixed a bug where having multiple gateways on the same port with SIMPLE and PASSTHROUGH modes does not work correctly. (Issue #33405)

  • Fixed a bug in Kubernetes Ingress causing paths with prefixes of the form /foo to match the route /foo/ but not the route /foo.

Istio 1.10.3#

  • Fixed a bug where wildcard hosts were incorrectly added even when a Sidecar resource only specified particular hosts. (Issue #33387)

  • Fixed a bug where setting the retryRemoteLocalities on a VirtualService would produce configuration that Envoy would reject. (Issue #33737)

  • Improved the meshConfig.defaultConfig.proxyMetadata field to do a deep merge when overridden rather than replacing all values.

Istio 1.10.2#

  • Fixed an issue where IPv6 iptables rules were incorrect when the traffic.sidecar.istio.io/includeOutboundPorts annotation was used. (Issue #30868)

  • Fixed a bug where secret files were not watched after being removed and then added back. (Issue #33293)

  • Fixed an issue causing Envoy Filters that merged the transport_socket field and had a custom transport socket name to be ignored.

Istio 1.10.1#

  • Fixed an issue causing the Host header to not be modifiable for specific destinations in a VirtualService (Issue #33226)

  • Fixed an issue that made it impossible to set the PDB maxUnavailable field in IstioOperator (Issue #31910)

Istio 1.10#

  • Changed the networking behavior to align with the standard behavior present in Kubernetes. The Envoy sidecar proxy, which binds to the eth0 interface, previously redirected all inbound traffic to the lo interface (typically 127.0.0.1); the sidecar now forwards inbound traffic on the eth0 interface (typically the pod’s IP address). Therefore, applications that bind to either lo or eth0 (but not both) are affected. For more information and a temporary workaround, see Upcoming networking changes in Istio 1.10. The temporary workaround will eventually be removed, perhaps as early as Istio 1.14.

  • Deprecated the values.global.jwtPolicy=first-party-jwt option. This option is less secure and intended for backwards compatibility with older Kubernetes clusters without support for more secure token authentication but is now enabled by default in new Kubernetes versions. See this documentation for more information.

  • Deprecated the values.global.arch option in favor of the affinity Kubernetes settings. (Issue #30027)

  • Deprecated the remote installation profile and added the external profile for installing Istio with an external control plane. (Issue #32370)

  • Added meshConfig.discoverySelectors to dynamically restrict the set of namespaces for Services, Pods, and Endpoints that istiod processes when pushing xDS updates to improve performance on the data plane. (Blog, Issue #26679)

  • Added the ISTIO_GATEWAY_STRIP_HOST_PORT environment variable to control whether gateways strip the host port before any processing of requests by HTTP filters or routing. This option is disabled by default. (Issue #25350)

  • Fixed configuration of TLS parameters (TLS version, TLS cipher suites, curves, etc.) with EnvoyFilter. (Issue #28996)

  • Fixed an issue where the filter chain name was ignored when processing EnvoyFilter match. (Issue #31166)

  • Improved the full push scoping by adding Sidecar config to sidecarScopeKnownConfigTypes.

  • Improved virtual machine integration to clean up iptables rules when the service is stopped. (Issue #29556)

  • Updated istio-proxy drain notification strategy from gradual to immediate. (Issue #31403)

  • Added CNI metrics counting repair operations. (Issue #19300)

  • Added /debug/connections istiod debug interface to list the current connected clients. (Issue #31075)

  • Added SDS secrets fetch failure metric pilot_sds_certificate_errors_total. (Issue #31779)

  • Added metrics for istiod informer errors.

  • Fixed a bug where ISTIO_META_IDLE_TIMEOUT is not reflected when set to 0s. (Issue #30067)

  • Fixed a bug causing unnecessary full push in service entry store. (Issue #30683)

  • Fixed a bug where the EnvoyFilter HTTP_FILTER didn’t support INSERT_FIRST. (Issue #31573)

  • Fixed an issue where services with PASSTHROUGH load balancing were always sent mTLS traffic, even if the destinations did not support mTLS. (Issue #23494)

  • Fixed a bug where EnvoyFilter with service match did not work for inbound clusters.

  • Added an experimental feature to allow dry-run of an AuthorizationPolicy without actually enforcing the policy. (Usage, Design, PR #1933)

  • Updated configuration to sign istiod certificates using Kubernetes CA (PILOT_CERT_PROVIDER=kubernetes) will not be honored in clusters with version 1.22 and greater. (Issue #22161)

  • Improved the experimental External Authorization feature with new capabilities:

    • Added the timeout field to configure the timeout (default is 10m) between the ext_authzfilter and the external service.

    • Added the include_additional_headers_in_check field to send additional headers to the external service.

    • Added the include_request_body_in_check field to send the body to the external service.

    • Supported prefix and suffix match in the include_request_headers_in_check, headers_to_upstream_on_allow and headers_to_downstream_on_deny field.

    • Deprecated the include_headers_in_check field with the new include_request_headers_in_check field for better naming. (Reference, PR #1926)

  • Added experimental option to configure Envoy to fetch the JWKS by itself. This should be enabled if the jwks_uri is a mesh cluster URL for mTLS and has other benefits like retries, JWKS caching etc. This is disabled by default and can be enabled by setting PILOT_JWT_ENABLE_REMOTE_JWKS to true. (Issue #28742)

  • Added an environment variable PILOT_JWT_PUB_KEY_REFRESH_INTERVAL to configure the interval of istiod fetching the jwks_uri for the JWKS public key. Users can set the refresh interval with --set values.pilot.env.PILOT_JWT_PUB_KEY_REFRESH_INTERVAL=<duration> during installation. The default interval is 20m. Valid time units are “ns”, “us”, “ms”, “s”, “m”, “h”.

  • Update the istiod JWT public key refresh job to retry the failed fetch of the jwks_uri with exponential backoff. (Issue #30261)

  • Removed ability to configure trustDomain from Helm global.values. Now it is configured through meshConfig.trustDomain (Issue #27734)

  • Fixed an issue causing simple TLS traffic to ports not exposed by a Service to be rejected by servers when in PERMISSIVE mTLS mode. (Issue #31297)

  • Added experimental support for the Telemetry API. (Issue #24284)

  • Fixed the missing destination_cluster metric label reported by client proxy on request failures. (Issue #29373)

  • Fixed an issue where Envoy did not start up properly when duplicate stats tags were configured. (Issue #31270)

  • Added reliable Wasm module remote load with istio-agent. (Issue #29989)

  • Added istioctl experimental revision tag command group. Revision tags act as aliases for control plane revisions. Users can label their namespaces with a revision tag rather than pointing them directly at a revision and selectively decide the granularity of their namespace labels. This makes it possible to perform upgrades with the ease of in-place upgrades while having the safety of revision-based upgrades under the hood. Read more about using revision tags here.

  • Improved ConfigMaps to be read directly rather than from volume mounts. This improves the speed of updates and ensures that for external istiod installations that the configmaps are read from the config cluster. (Issue #31410)

  • Improved the sidecar injector to better utilize pod labels to determine if injection is required. (Issue #30013)

  • Updated non-revisioned installs to target the label istio.io/rev=default for injection in addition to the existing default injection labels (istio-injection=enabled and sidecar.istio.io/inject=true).

  • Added support for slash characters in environment variables on injectionURL. (Issue #31732)

  • Added an external profile for installing Istio with an external control plane and deprecated the remote profile. (Issue #32370)

  • Fixed a bug preventing istioctl kube-inject from working with revisions. (Issue #30991)

  • Improved the output of istioctl YAML diff commands. (Issue #31186)

  • Removed the 15012 and 15443 ports from the default gateway installation. These can be explicitly added if desired, although it is recommended to follow the new multicluster installation guide instead.

  • Updated Kiali addon to the latest version v1.34.

  • Updated the istioctl experimental precheck command to identify potential upgrade issues prior to actually running an upgrade.

  • Updated istioctl kube-inject to call the webhook server to get the injection template by default. (Issue #29270)

  • Added istioctl experimental internal-debug to retrieve istiod debug information via a secured debug interface. (Issue #31338)

  • Added istioctl validate and the validating webhook now report duplicate or unreachable virtual service matches. (Issue #31525)

  • Added istioctl proxy-config -o yaml to display in YAML along with the current JSON and short format. (Usage, Issue #31695)

  • Added the istioctl proxy-config all command to view the full proxy configuration.

  • Added tooling for revision-centric view of current Istio deployments in a cluster. This is to provide a better understanding of deployments- such as the number of istiod, gateway pods, IstioOperator custom resources- defining a particular revision, and the number of pods with sidecars pointing to a particular revision. (Issue #23892)

  • Added a new analyzer for invalid webhook configurations.

  • Fixed an issue where istioctl x create-remote-secret --secret-name failed incorrectly when pointing to a non-existent secret in the remote cluster. (Issue #30723)

Aspen Mesh features#

(No changes)

Known issues#

  • AM-3069: OpenShift clusters using either the Multi-Primary or the Multi-Primary on different networks configuration for multicluster connectivity fail to create the remote secret with the following error:

    $ istioctl x create-remote-secret --name=cluster1

error: could not get access token to read resources from local kube-apiserver: wrong number of secrets (2) in serviceaccount istio-system/istio-reader-service-account

error: could not get access token to read resources from local kube-apiserver: wrong number of secrets (2) in serviceaccount istio-system/istio-reader-service-account

  • AM-3547: Pods with Istio sidecars get evicted when a node runs low on storage because they don’t request ephemeral storage.

Feature request AM42: Sidecar-listener exact balancing#

This feature is supported but requires configuration. Learn how to configure sidecar listeners for exact balancing.

Download#

Use either of the following methods to download the release archive file: