Is Your AKS Load Balancer Unhealthy? Troubleshoot Health Probes in Azure

Table of Contents

The health probe mode feature in Azure Kubernetes Service (AKS) is designed to enhance the reliability and performance of your cluster’s services. It allows you to fine-tune how the Azure Load Balancer assesses the health of nodes within your AKS cluster. This feature offers two distinct modes: Shared and ServiceNodePort. Understanding the nuances of these modes is crucial for optimal configuration and troubleshooting.

In Shared mode, a single health probe is utilized for all external traffic policy cluster services that share the same load balancer. This approach can significantly reduce the number of health probes, leading to improved load balancer efficiency and potentially lower costs. However, it introduces dependencies on additional components to function correctly. Conversely, ServiceNodePort mode employs a dedicated health probe for each service. While this method might increase the number of probes, it provides more granular health monitoring and avoids the complexities associated with Shared mode.

This article is designed to guide you through common challenges encountered when implementing the health probe mode feature in an AKS cluster. We will delve into practical troubleshooting steps and effective resolutions to ensure your AKS load balancer functions as expected.

Symptoms¶

When you enable the health probe mode feature, specifically using the Shared mode, during AKS cluster creation or updates via the Azure CLI, you might encounter several symptoms indicating an issue. These symptoms can manifest in different ways, disrupting traffic flow and overall service availability. Recognizing these symptoms is the first step towards effective troubleshooting.

• Unexpected Traffic Distribution: The load balancer may fail to distribute traffic evenly or correctly across the nodes in your AKS cluster. This can lead to some nodes being overloaded while others remain underutilized, impacting the performance and responsiveness of your applications. Traffic might be routed to fewer nodes than expected, or even seemingly randomly, rather than following the intended load balancing rules.

• False Unhealthy Node Reports: The load balancer might incorrectly report nodes as unhealthy even when they are functioning properly. This misreporting can trigger unnecessary node removals or traffic diversions, further disrupting service availability. You might observe nodes being marked as “Out of Service” in the load balancer backend pool despite the nodes being healthy and responsive within the Kubernetes cluster.

• health-probe-proxy Sidecar Container Issues: The health-probe-proxy sidecar container, a critical component for Shared mode, might crash repeatedly or fail to start altogether. This sidecar is responsible for correctly routing health probe requests in Shared mode, and its malfunction directly impacts the health probe mechanism. Logs for this container might indicate errors related to connectivity, configuration, or resource limits.

• cloud-node-manager Pod Problems: Similarly, the cloud-node-manager pod, which also plays a role in managing health probes in AKS, might experience crashes or startup failures. This pod is responsible for orchestrating various cloud provider functionalities within the AKS cluster, including health probe configurations. Issues with this pod can stem from incorrect configurations, resource constraints, or dependencies on other cluster components.

Behind the scenes, enabling the health probe mode involves a series of orchestrated operations within the Azure Resource Provider (RP) and AKS control plane:

1. RP Frontend Validation: The Azure Resource Provider frontend validates the incoming request to enable health probe mode. It checks if the request is syntactically correct and semantically valid before proceeding. If the request passes validation, the RP frontend updates the LoadBalancerProfile property of the AKS cluster configuration to reflect the desired health probe mode.

2. Cloud Provider Config Secret Reconciliation: The RP asynchronously calls the cloud provider config secret reconciler. This component is responsible for updating the cloud-provider-config secret within the Kubernetes cluster based on the changes made to the LoadBalancerProfile. This secret contains configuration details required by the cloud provider integration within Kubernetes, including health probe settings.

3. Overlay Manager Chart Reconciliation: The Overlay Manager (overlaymgr) reconciles the cloud-node-manager chart. This reconciliation process ensures that the necessary Kubernetes resources, specifically the cloud-node-manager DaemonSet, are updated to enable the health-probe-proxy sidecar container. The DaemonSet is modified to include the sidecar, ensuring it’s deployed on each node in the cluster.

Initial Troubleshooting¶

When faced with the symptoms described above, a systematic troubleshooting approach is crucial to pinpoint the root cause. The following initial troubleshooting steps will help you gather valuable information and narrow down the potential issues.

1. Inspect RP Frontend Logs: Examine the logs of the Azure Resource Provider frontend to verify if the health probe mode configuration in the LoadBalancerProfile is correctly set. You can use the Azure CLI command az aks show --resource-group <your-resource-group> --name <your-aks-cluster-name> --query loadBalancerProfile to retrieve the LoadBalancerProfile property of your cluster. Analyze the output to confirm that the clusterServiceLoadBalancerHealthProbeMode is set to the intended value (e.g., “Shared”). Discrepancies here indicate a problem during the initial configuration request processing.

2. Check Overlay Manager Logs and Cloud Provider Secret: Investigate the Overlay Manager logs for entries related to cloudConfigSecretResolver. This will help you determine if the cloud provider secret was successfully updated based on the LoadBalancerProfile. Alternatively, directly inspect the contents of the cloud-provider-config secret in the ccp namespace using the kubectl get secret cloud-provider-config -n ccp -o yaml command. Verify that the secret reflects the expected health probe mode configuration. Failures in secret updates can indicate issues with the communication between the RP and the cluster control plane or problems within the Overlay Manager itself.

3. Verify cloud-node-manager DaemonSet: Check the cloud-node-manager DaemonSet to confirm if the health-probe-proxy sidecar container is correctly enabled. Use the command kubectl get ds cloud-node-manager -n kube-system -o yaml to retrieve the DaemonSet definition. Examine the containers section to ensure the health-probe-proxy container is present and properly configured. Absence of the sidecar container or incorrect configuration within the DaemonSet indicates a problem with the Overlay Manager’s reconciliation of the cloud-node-manager chart.

Cause 1: Incorrect Health Probe Mode¶

The health probe mode feature is specifically designed to operate with only two modes: Shared and ServiceNodePort. Attempting to use any other mode will render the feature non-functional and can lead to unexpected behavior in your load balancer and service health monitoring. Using an unsupported or incorrectly specified mode will prevent the necessary components from being configured and deployed correctly.

Solution 1: Utilize Valid Health Probe Modes¶

To resolve this issue, ensure that you are explicitly using either the Shared or ServiceNodePort mode when creating or updating your AKS cluster. You can specify the desired mode using the --cluster-service-load-balancer-health-probe-mode flag within the az aks create or az aks update commands. For example, to enable Shared mode, use:

az aks update --resource-group <your-resource-group> --name <your-aks-cluster-name> --cluster-service-load-balancer-health-probe-mode Shared

Similarly, for ServiceNodePort mode (which is the default), you can use:

az aks update --resource-group <your-resource-group> --name <your-aks-cluster-name> --cluster-service-load-balancer-health-probe-mode ServiceNodePort

Verify the configured mode after applying the changes by re-running the az aks show command and inspecting the loadBalancerProfile.

Cause 2: Disabled Health Probe Mode Toggle¶

The health probe mode feature is governed by a feature toggle that can be enabled or disabled by the AKS engineering team at a subscription level. If this toggle is inadvertently or intentionally disabled for your subscription, the health probe mode feature, regardless of your configuration, will not operate as intended. This is a backend control mechanism to manage feature rollout and stability.

Solution 2: Enable the Feature Toggle¶

To address this, you need to contact the AKS support team or your Microsoft account representative to inquire about the status of the health probe mode feature toggle for your Azure subscription. If the toggle is indeed disabled, request them to enable it for your subscription. This might involve internal processes within Microsoft to activate the feature for your specific subscription. Once enabled, retry configuring or using the health probe mode feature on your AKS cluster.

Cause 3: Basic Load Balancer SKU¶

The health probe mode feature is exclusively compatible with the Standard Load Balancer SKU in Azure. If your AKS cluster is configured to use the Basic Load Balancer SKU, the health probe mode feature will not function. The Basic SKU lacks the advanced features and capabilities required to support the health probe mode effectively.

Solution 3: Migrate to Standard Load Balancer SKU¶

To resolve this incompatibility, you must ensure your AKS cluster utilizes the Standard Load Balancer SKU. When creating a new AKS cluster, you can specify the SKU using the --load-balancer-sku flag:

az aks create --resource-group <your-resource-group> --name <your-aks-cluster-name> --load-balancer-sku Standard ...

For existing clusters using the Basic SKU, migrating to the Standard SKU requires a more involved process. Direct in-place SKU upgrade is not supported. You would typically need to create a new AKS cluster with the Standard Load Balancer SKU and migrate your workloads to the new cluster. Consult the official Azure documentation for detailed guidance on migrating from Basic to Standard Load Balancer in AKS.

Cause 4: Feature Unregistered¶

Before utilizing the health probe mode feature, it’s necessary to register the feature for your Azure subscription. If the feature is not properly registered, attempting to enable and use it will fail, and the expected functionalities will not be activated. Feature registration is a prerequisite for accessing and utilizing certain Azure features.

Solution 4: Register the Feature¶

To register the health probe mode feature for your subscription, use the Azure CLI command az feature register. You will need to know the specific feature name to register. Consult the AKS documentation or feature release notes to identify the correct feature name for health probe mode. Once you have the feature name, execute the registration command:

az feature register --namespace Microsoft.ContainerService --name <FeatureName>

Replace <FeatureName> with the actual feature name. After submitting the registration request, it might take some time for the registration to complete. You can check the registration status using az feature show:

az feature show --namespace Microsoft.ContainerService --name <FeatureName>

Ensure the registration state is “Registered” before attempting to use the health probe mode feature on your AKS cluster.

Cause 5: Incompatible Kubernetes Version¶

The health probe mode feature has a minimum Kubernetes version requirement. Specifically, it mandates Kubernetes version v1.28.0 or later. If your AKS cluster is running on an earlier Kubernetes version, the health probe mode feature will not be supported and will not function correctly. Older Kubernetes versions might lack the necessary APIs or functionalities to support this feature.

Solution 5: Upgrade Kubernetes Version¶

To address this version dependency, you must upgrade your AKS cluster to Kubernetes version v1.28.0 or a later supported version. You can initiate a Kubernetes version upgrade using the Azure CLI command az aks upgrade:

az aks upgrade --resource-group <your-resource-group> --name <your-aks-cluster-name> --kubernetes-version v1.28.0

Replace v1.28.0 with the desired target Kubernetes version if you are aiming for a later version. The upgrade process can take some time as it involves upgrading the control plane and agent nodes of your AKS cluster. After the upgrade is complete, verify the Kubernetes version using az aks show and then attempt to configure or use the health probe mode feature.

Known Issues¶

It’s important to be aware of known issues that might interact with the health probe mode feature. One notable issue specifically impacts Windows nodes within AKS clusters.

Windows Node Kube-proxy Startup Delay: On Windows nodes, the kube-proxy component might experience a delay in starting up until the first non-HPC (High-Performance Computing) pod is created on the node. This startup delay can interfere with the health probe mode feature, potentially causing the load balancer to incorrectly report Windows nodes as unhealthy during the initial phase of node initialization. This issue is expected to be resolved in a future AKS update. For environments heavily reliant on Windows nodes and the health probe mode, it’s crucial to be aware of this potential behavior and monitor node health accordingly, especially during node startup or scaling events.

How to Enable Health Probe Mode via Azure CLI¶

Enabling the health probe mode feature using the Azure CLI is straightforward. You can specify the desired mode during AKS cluster creation or when updating an existing cluster.

To enable health probe mode, use one of the following commands, depending on your desired mode:

• Shared Mode:

  az aks create --resource-group <your-resource-group> --name <your-aks-cluster-name> --cluster-service-load-balancer-health-probe-mode Shared ...
  

  or to update an existing cluster:

  az aks update --resource-group <your-resource-group> --name <your-aks-cluster-name> --cluster-service-load-balancer-health-probe-mode Shared
  

• ServiceNodePort Mode (Default):

  az aks create --resource-group <your-resource-group> --name <your-aks-cluster-name> --cluster-service-load-balancer-health-probe-mode ServiceNodePort ...
  

  or to update an existing cluster:

  az aks update --resource-group <your-resource-group> --name <your-aks-cluster-name> --cluster-service-load-balancer-health-probe-mode ServiceNodePort
  

  If you omit the --cluster-service-load-balancer-health-probe-mode flag, ServiceNodePort mode will be enabled by default.

By following these troubleshooting steps and understanding the potential causes and solutions, you can effectively diagnose and resolve issues related to unhealthy AKS load balancer health probes when using the health probe mode feature in Azure. Proper configuration and awareness of known limitations are key to ensuring the reliability and optimal performance of your AKS services.

Do you have any further questions or experiences to share regarding AKS load balancer health probes? Feel free to leave a comment below!