Troubleshooting Istio Service Mesh Add-on on Azure: A Practical Guide

Table of Contents

This article provides a practical guide to troubleshooting common issues encountered when using the Istio service mesh add-on for Microsoft Azure Kubernetes Service (AKS). It outlines general strategies employing tools like kubectl and istioctl, along with a list of specific error messages, their underlying causes, and recommended solutions. By leveraging these tools and following the outlined steps, users can effectively diagnose and resolve problems within their Istio-enabled AKS clusters.

Prerequisites¶

Before diving into the troubleshooting steps, ensure you have the necessary tools installed and configured to interact with your Azure AKS cluster and Istio service mesh.

• The Azure CLI is essential for managing Azure resources, including AKS clusters. Install it to authenticate and interact with your Azure subscription.
• The Kubernetes kubectl tool is the standard command-line tool for interacting with Kubernetes clusters. You can install it directly or conveniently via the Azure CLI using the az aks install-cli command. This tool allows you to inspect cluster resources, view logs, and execute commands within pods.
• The Istio istioctl command-line tool is specifically designed for diagnosing and troubleshooting Istio service meshes. It provides Istio-specific commands to check the mesh status, analyze configurations, and generate reports.
• The Client URL (cURL) tool is useful for testing connectivity and interacting with services within or outside the mesh. It’s often used in conjunction with kubectl exec to test service endpoints from within a pod.

Having these prerequisites in place will equip you with the fundamental capabilities needed to effectively troubleshoot your Istio service mesh add-on.

Troubleshooting Checklist: Using kubectl¶

kubectl is your primary tool for interacting with the Kubernetes cluster where Istio is deployed. These steps focus on using kubectl to inspect the state of Istio components and application pods within the mesh.

Step 1: Get Istiod Pod Logs¶

The Istiod pod is the control plane of the Istio service mesh. Its logs often contain valuable information about configuration distribution, errors, and warnings. To retrieve the logs for the Istiod pod, use the following command, targeting the aks-istio-system namespace where the add-on components reside:

kubectl logs --selector app=istiod --namespace aks-istio-system

Reviewing these logs can provide initial clues about potential issues within the Istio control plane itself. Look for error messages or abnormal behavior patterns recorded in the logs.

Step 2: Bounce (Delete) a Pod¶

Sometimes, simply restarting a problematic pod can resolve transient issues. For Istiod, which is managed by a Deployment, you can safely delete the pod, and Kubernetes will automatically recreate it. This action is analogous to restarting the Istiod service.

To delete the Istiod pod, use the kubectl delete command:

kubectl delete pods <istio-pod> --namespace aks-istio-system

Replace <istio-pod> with the actual name of the Istiod pod. This is a quick method to see if a fresh start for the control plane resolves any issues.

Step 3: Check the Status of Resources¶

If Istiod or other related Istio components are not behaving as expected (e.g., pods are stuck in a pending state or crashing), checking the status of the Deployment and ReplicaSet resources can provide insight. These resources manage the desired state and scaling of your Istio pods.

Use the kubectl get command to inspect the status:

kubectl get <resource-type> [[--selector app=istiod] | [<resource-name>]]

For example, you might run kubectl get deployment istiod --namespace aks-istio-system or kubectl get replicaset --selector app=istiod --namespace aks-istio-system. The output will show the current state, including desired, current, ready, and available replicas, as well as recent events related to the resource’s lifecycle.

Step 4: Get Custom Resource Definition Types¶

Istio heavily relies on Custom Resource Definitions (CRDs) to define its configuration objects like Gateways, VirtualServices, and DestinationRules. Understanding the available Istio CRDs installed in your cluster is fundamental to diagnosing configuration issues.

To list all installed CRDs related to Istio, filter the CRD list using grep:

kubectl get crd | grep istio

This command helps confirm that all necessary Istio CRDs have been successfully installed by the add-on. Once you know the CRD types, you can list all resources of a specific type across all namespaces.

To list resources based on a particular Istio CRD, such as virtualservices, run:

kubectl get virtualservices --all-namespaces

This allows you to quickly see all instances of a specific Istio configuration object deployed in your mesh.

Step 5: View the List of Istiod Pods¶

To get a detailed list and status of all Istiod pods, including their current state and events, retrieve the pod list in YAML format. This output provides more comprehensive information than the default table format.

Use the kubectl get command with the -o yaml flag:

kubectl get pod --namespace aks-istio-system --output yaml

Examining the YAML output for each pod can reveal details like container statuses, restart counts, and conditions, which are crucial for diagnosing pod-level issues.

Step 6: Get More Information About the Envoy Configuration¶

Envoy proxy runs as a sidecar container in each service pod within the mesh. It handles all inbound and outbound traffic for the application container. Debugging connectivity issues often requires inspecting the Envoy configuration received from Istiod. Envoy exposes an admin interface (usually on port 15000) for this purpose.

You can access the Envoy admin endpoint from within the pod using kubectl exec and curl. The following command retrieves the configured clusters (endpoints) from the Envoy proxy of a pod labeled app=sleep in a specified namespace:

kubectl exec --namespace <pod-namespace> \
    "$(kubectl get pods \
        --namespace <pod-namespace> \
        --selector app=sleep \
        --output jsonpath='{.items[0].metadata.name}')" \
    --container sleep \
-- curl -s localhost:15000/clusters

Replace <pod-namespace> with the actual namespace and adjust the selector (app=sleep) as needed to target your specific application pod. Exploring other endpoints on the Envoy admin interface (e.g., /config_dump, /routes) can provide deep insights into the proxy’s dynamic configuration.

Step 7: Get the Sidecar Logs for the Source and Destination Sidecars¶

Envoy sidecar logs are essential for understanding the traffic flow and identifying errors at the data plane level. When diagnosing connectivity problems between two services, checking the logs of the Envoy proxies in both the source and destination pods is highly recommended.

Use the kubectl logs command, specifying the istio-proxy container name:

kubectl logs <pod-name> --namespace <pod-namespace> --container istio-proxy

Execute this command for the source pod and then again for the destination pod involved in the communication issue. Look for error messages related to connection failures, handshake issues, or policy enforcement rejections in these logs.

Troubleshooting Checklist: Using istioctl¶

istioctl is a command-line tool specifically built for Istio, offering powerful diagnostic capabilities tailored to the service mesh. When using istioctl with the AKS add-on, always include the --istioNamespace aks-istio-system flag to ensure the commands target the correct Istio control plane installation.

Step 1: Make Sure That Istio is Installed Correctly¶

The istioctl verify-install command checks the Istio installation against the expected state. It verifies that all necessary components are running and healthy according to the installation manifests.

To verify the installation of the AKS add-on version, run:

istioctl verify-install --istioNamespace aks-istio-system --revision <tag>

Replace <tag> with the specific Istio revision installed (e.g., asm-1-18). This command is a crucial first step to confirm the basic health and correctness of your Istio deployment before investigating deeper issues.

Step 2: Analyze Namespaces¶

The istioctl analyze command examines your Kubernetes and Istio configuration for common issues and misconfigurations. It can detect problems like missing sidecars, incorrect destination rules, or policy violations.

You can analyze all namespaces or focus on a specific one:

istioctl analyze --istioNamespace aks-istio-system \
    --revision <tag> \
    [--all-namespaces | --namespace <namespace-name>] \
    [--failure-threshold {Info | Warning | Error}]

This command provides actionable feedback on potential configuration errors in your Istio resources and the resources they apply to. Adjust the --failure-threshold to filter the output based on the severity of detected issues.

Step 3: Get the Proxy Status¶

The istioctl proxy-status command reports the synchronization status between Istiod and the Envoy sidecar proxy in a specific pod. This is vital for ensuring that the Envoy proxy has received the latest configuration from the control plane.

Check the proxy status for a specific pod using its name:

istioctl proxy-status pod/<pod-name> \
    --istioNamespace aks-istio-system \
    --revision <tag> \
    --namespace <pod-namespace>

If the status shows that the proxy is not synced or is out of sync, it indicates a problem with configuration distribution from Istiod, which can cause traffic routing or policy enforcement failures.

Step 4: Download the Proxy Configuration¶

To perform a detailed inspection of the configuration that an Envoy proxy has received, you can download its full configuration dump using istioctl proxy-config. This dump includes listeners, routes, clusters, endpoints, and more.

Retrieve the configuration dump for a pod:

istioctl proxy-config all <pod-name> \
    --istioNamespace aks-istio-system \
    --namespace <pod-namespace> \
    --output json

Reviewing this detailed configuration in JSON format allows you to verify whether the intended Istio policies (VirtualServices, DestinationRules, etc.) have been correctly translated and pushed down to the data plane proxy.

Step 5: Check the Injection Status¶

Sidecar injection is the process by which the Istio proxy is added to your application pods. It’s a prerequisite for a workload to be part of the mesh. The istioctl experimental check-inject command helps determine if a resource is configured for automatic sidecar injection and if the injection was successful.

Use this command to check the injection status of a namespace, specific pod, or deployment:

istioctl experimental check-inject --istioNamespace aks-istio-system \
    --namespace <pod-namespace> \
    --labels <label-selector> | <pod-name> | deployment/<deployment-name>

This is particularly useful if you suspect that a pod is not participating in the mesh because the sidecar wasn’t injected correctly. It provides information on whether injection is enabled and the status of the injection process.

Step 6: Get a Full Bug Report¶

For complex issues or when seeking support, a comprehensive bug report is invaluable. The istioctl bug-report command collects a wealth of diagnostic information from your cluster, including logs, configurations, and resource states for Istio components and potentially application workloads.

Generating a bug report can take time, especially on large clusters, as it gathers data from many pods. You can scope the report to specific namespaces to reduce the collection time and size.

Generate a bug report using the following command:

istioctl bug-report --istioNamespace aks-istio-system \
    [--include <namespace-1>[, <namespace-2>[, ...]]]

This command packages the collected information into a file that can be easily shared for further analysis or support requests. It’s the most thorough method for capturing the state of your mesh at a given time.

Troubleshooting Checklist: Miscellaneous Issues¶

Beyond using kubectl and istioctl for direct inspection, several common patterns and configurations can lead to issues in an Istio service mesh. Addressing these requires understanding Istio’s resource management, networking, and interaction with the underlying Kubernetes platform.

Step 1: Fix Resource Usage Issues¶

High resource consumption, particularly memory in Envoy proxies, is a common operational challenge. Overly detailed telemetry collection or a large number of services in the mesh can contribute to this. Envoy needs to hold state and configuration for all reachable endpoints.

Examine your Envoy statistics configuration. High cardinality metrics (metrics with many unique label values) significantly increase memory usage. Customizing Istio metrics via MeshConfig should be done cautiously, considering the impact on cardinality. Also, review the concurrency setting in MeshConfig, which affects CPU usage.

By default, Istio pushes the configuration for all services in the cluster to every Envoy proxy. This can be inefficient in large clusters. The Sidecar resource allows you to limit the scope of configuration distributed to proxies, restricting it to services only within the proxy’s own namespace and specified allowed namespaces. Deploying a Sidecar resource in the aks-istio-system namespace can establish a default egress policy for the entire mesh, reducing configuration size for all proxies.

Consider the following Sidecar resource example applied to the root namespace (aks-istio-system), which restricts egress to the local namespace and the Istio control plane namespace:

apiVersion: networking.istio.io/v1alpha3
kind: Sidecar
metadata:
  name: sidecar-restrict-egress
  namespace: aks-istio-system  # This applies as a default to all namespaces
spec:
  egress:
  - hosts:
    - "./*" # Allow traffic to services in the same namespace
    - "aks-istio-system/*" # Allow traffic to services in the istio control plane namespace

Another approach to reduce Istiod’s resource consumption and the size of configurations pushed to proxies is using discoverySelectors in MeshConfig. This feature allows you to limit the namespaces that Istiod watches and includes in its configuration distribution, effectively partitioning the mesh for control plane scaling.

Step 2: Fix Traffic and Security Misconfiguration Issues¶

Traffic management and security are core functions of Istio, but misconfigurations are frequent sources of problems. Issues such as incorrect routing in VirtualServices, missing DestinationRules, or misapplied PeerAuthentication policies can lead to connectivity failures or denied requests.

Istio’s official documentation provides detailed guides on troubleshooting common problems in these areas. Refer to the sections on “Traffic management problems” and “Security problems” in the Istio documentation. These resources cover typical scenarios and debugging techniques specific to Istio’s networking and security features. Exploring the “Common problems” page is also recommended for issues related to injection, observability, and upgrades.

Step 3: Avoid CoreDNS Overload¶

In some environments, Istio’s DNS lookup behavior can potentially put excessive load on the cluster’s CoreDNS service. Istio proxies might perform frequent DNS lookups, especially if dnsRefreshRate is set to a low value in the MeshConfig.

If you observe high CPU or request load on your CoreDNS pods, consider adjusting Istio’s DNS settings. Increasing the dnsRefreshRate value in the MeshConfig can reduce the frequency of DNS queries originating from the Envoy sidecars, thereby alleviating pressure on CoreDNS.

Step 4: Fix Pod and Sidecar Race Conditions¶

A known issue can occur if an application container starts before the Istio sidecar proxy within the same pod is fully initialized and ready to intercept network traffic. The application might attempt to make network connections that bypass the proxy or fail because the network stack is not yet managed by Envoy, leading to application errors or restarts.

To mitigate this race condition, Istio provides the holdApplicationUntilProxyStarts setting in the MeshConfig under defaultConfig. Setting this field to true instructs the application container to wait until the istio-proxy container is ready before its entrypoint command is executed. This ensures that the Envoy proxy is always the first point of network interaction for the pod.

Step 5: Configure a Service Entry When Using an HTTP Proxy for Outbound Traffic¶

If your Azure AKS cluster is configured to use an HTTP proxy for all outbound internet access, the Istio sidecars also need to be aware of and correctly route traffic through this proxy for external services. Without proper configuration, outbound calls from mesh-enabled pods to external endpoints will fail.

When using an HTTP proxy, you must configure an Istio ServiceEntry to define the external service and instruct the sidecar proxies on how to reach it, often including specifying the proxy’s address. Refer to the Azure Kubernetes Service documentation on HTTP proxy support, specifically the section regarding the Istio add-on, for detailed configuration steps and examples.

Step 6: Enable Envoy Access Logging¶

Envoy access logs provide detailed records of requests passing through the sidecar proxies and ingress/egress gateways. Enabling and analyzing these logs is a fundamental troubleshooting technique for understanding traffic flow, identifying connection issues, policy enforcement outcomes (e.g., denied requests), and latency problems.

You can configure Envoy access logging via the Istio MeshConfig or the newer Telemetry API. The AKS documentation on Istio covers configuring mesh settings (MeshConfig), using the Telemetry API for observability, and setting up metrics collection with Managed Prometheus. Enabling verbose access logging can generate a large volume of data, so consider the impact on your logging backend and storage costs.

Error Messages¶

This section provides a table listing common error messages you might encounter when deploying, configuring, or upgrading the Istio service mesh add-on on AKS, along with the reasons for the errors and recommended actions.

Error	Reason	Recommendations
Azure service mesh is not supported in this region	The Istio add-on feature is not yet available in the specific Azure region during the preview phase.	Consult the official public documentation for the Istio service mesh add-on on AKS to confirm the list of supported regions.
Missing service mesh mode: {}	When enabling the service mesh profile for the AKS cluster, the required mode property was not specified in the API request.	In the ServiceMeshProfile field of your managedCluster API request (e.g., using ARM templates), ensure the mode property is set to Istio.
Invalid istio ingress mode: {}	An invalid value was provided for the ingress mode property when attempting to configure an ingress gateway within the service mesh profile settings.	The ingress mode property in the API request must be set to one of the valid values: External for a public IP or Internal for a private IP within the VNet.
Too many ingresses for type: {}. Only {} ingress gateway are allowed	You attempted to create more ingress gateways than allowed for a specific type (External or Internal) through the add-on configuration.	The AKS Istio add-on supports, at most, one external ingress gateway and one internal ingress gateway concurrently. Adjust your configuration to stay within these limits.
Istio profile is missing even though Service Mesh mode is Istio	The Istio add-on was enabled on the cluster, but the necessary configuration parameters for the Istio profile (which define components like ingress gateways or plugin CA) were not provided.	When enabling the Istio add-on, you must specify the Istio profile details, including configuration for specific components and the desired revision, in the API request.
Istio based Azure service mesh is incompatible with feature %s	You tried to enable the Istio add-on while another feature, add-on, or extension that is currently incompatible with Istio was already active on the cluster (e.g., Open Service Mesh).	Before enabling the Istio add-on, ensure that any incompatible features or add-ons are disabled first. You might also need to clean up any associated resources created by the conflicting feature.
ServiceMeshProfile is missing required parameters: %s for plugin certificate authority	You attempted to enable the plug-in certificate authority (CA) feature for Istio but did not provide all the mandatory parameters required for its configuration.	Review the documentation for setting up the Istio-based service mesh add-on with plug-in CA certificates and ensure all required parameters, such as Key Vault details, are correctly specified in the Istio profile configuration.
AzureKeyvaultSecretsProvider addon is required for Azure Service Mesh plugin certificate authority feature	The plug-in CA feature relies on the Azure Key Vault Secrets Store CSI Driver add-on to securely access certificates. This dependency was not met.	Before configuring the Istio plug-in CA feature, you must first enable and set up the Azure Key Vault Secrets Store CSI Driver add-on on your AKS cluster, ensuring it can access the required certificates in Key Vault.
'KeyVaultId': '%s' is not a valid Azure keyvault resource identifier. Please make sure that the format matches '/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.KeyVault/vaults/<vault-name>'	The provided Azure Key Vault resource ID format was incorrect, preventing Istio’s plug-in CA from accessing the vault.	Double-check the format of the Azure Key Vault ID you provided. It must follow the exact Azure Resource Manager ID structure as specified in the error message. Correct the format and retry the configuration.
Kubernetes version is missing in orchestrator profile	An API request related to the Istio add-on (often during upgrade) did not include the Kubernetes version of the cluster, preventing version compatibility checks.	When performing operations related to the Istio add-on, especially upgrades, ensure that the Kubernetes version of the target cluster is explicitly included in the orchestrator profile section of the API request.
Service mesh revision %s is not compatible with cluster version %s. To find information about mesh-cluster compatibility, use 'az aks mesh get-upgrades'	You attempted to install or upgrade to an Istio add-on revision that is not compatible with the current Kubernetes version of your AKS cluster.	Use the az aks mesh get-upgrades Azure CLI command to list the Istio add-on revisions that are compatible with your specific AKS cluster version. Choose a compatible revision for your installation or upgrade.
Kubernetes version %s not supported. Please upgrade to a supported cluster version first. To find compatibility information, use 'az aks mesh get-upgrades'	Your AKS cluster is running an unsupported Kubernetes version for the Istio add-on you are trying to use.	You must upgrade your AKS cluster to a supported Kubernetes version before you can proceed with installing or upgrading the Istio add-on to the desired revision. Use the az aks upgrade command.
ServiceMeshProfile revision field must not be empty	You attempted an upgrade operation for the Istio add-on but failed to specify the target revision you wish to upgrade to.	When performing a minor revision upgrade for the Istio add-on, you must explicitly specify the target revision in the ServiceMeshProfile configuration. Ensure all necessary parameters for the upgrade are included.
Request exceeds maximum allowed number of revisions (%d)	You tried to perform an upgrade operation when the cluster already has the maximum allowed number of Istio revisions installed simultaneously.	The AKS Istio add-on has a limit on the number of concurrently installed revisions during a canary upgrade process. You must either complete or roll back the existing in-progress upgrade before attempting another revision change.
Mesh upgrade is in progress. Please complete or roll back the current upgrade before attempting to retrieve versioning and compatibility information	You tried to query versioning or compatibility information (az aks mesh get-upgrades) while an Istio add-on upgrade process was already underway.	Wait for the current Istio add-on upgrade or rollback operation to finish before attempting to retrieve information about available versions and compatibility.

References¶

For further information and related troubleshooting guides, consult the following resources:

• The official Istio documentation provides a comprehensive guide to Istio diagnostic tools, covering istioctl and other methods in detail.
• Specific troubleshooting articles for the AKS Istio add-on cover common issues related to MeshConfig, ingress gateway, minor revision upgrades, and plug-in CA certificates.

Third-party information disclaimer:

This article references products manufactured by companies independent of Microsoft. Microsoft makes no warranty, implied or otherwise, about the performance or reliability of these products.

We hope this guide helps you effectively troubleshoot issues with your Istio service mesh add-on on Azure AKS. What specific challenges have you encountered, or what tips have you found most useful? Share your experiences in the comments below!