Azure Node Failure: Troubleshooting Custom Script Extension Errors

Table of Contents

This article provides guidance on troubleshooting scenarios where a Microsoft Azure Kubernetes Service (AKS) cluster is not in a Succeeded state, and nodes within a node pool are not ready due to Custom Script Extension (CSE) errors. These situations can be disruptive to your applications and require prompt resolution to ensure the health and availability of your AKS cluster. Understanding the common causes and effective solutions is crucial for maintaining a stable Kubernetes environment on Azure.

Prerequisites¶

Before you begin troubleshooting, ensure you have the Azure Command-Line Interface (CLI) installed and configured. The Azure CLI is a powerful tool that allows you to manage Azure resources from the command line, making it essential for diagnosing and resolving issues within your AKS cluster. You can find installation instructions and the latest version of the Azure CLI on the official Microsoft Azure documentation website. Having the Azure CLI ready will enable you to execute the necessary commands to investigate and remediate CSE errors.

Symptoms¶

The primary symptom of CSE errors manifesting as node failures is an AKS cluster that is not in the Succeeded state. Furthermore, within the affected cluster, individual nodes within a node pool will report a “Not Ready” status. This “Not Ready” status indicates that the node is experiencing issues that prevent it from properly functioning within the Kubernetes cluster. These issues are directly linked to failures during the Custom Script Extension execution, which is a critical part of node provisioning and configuration in AKS. This state prevents workloads from being scheduled on these nodes, impacting application availability.

Cause¶

The root cause of these “Node Not Ready” failures often lies in the failure of the node extension deployment. This deployment is responsible for provisioning the kubelet, the primary node agent in Kubernetes, along with other essential components necessary for the node to function correctly. When the extension deployment fails and returns one or more error codes, it disrupts the node’s ability to join the cluster and achieve a “Ready” state. This failure during kubelet provisioning is the most prevalent reason for CSE-related node failures.

To confirm if the node extension deployment is indeed failing during kubelet provisioning, you can follow these steps to examine the cluster’s state and debug information:

1. Initiate Debugging with Azure CLI: Utilize the Azure CLI commands az aks show and az resource update to set up debugging for your AKS cluster. These commands will provide detailed output that can help identify the source of the problem.

clusterResourceId=$(az aks show \
    --resource-group <resource-group-name> --name <cluster-name> --output tsv --query id)
az resource update --debug --verbose --ids $clusterResourceId

Replace <resource-group-name> and <cluster-name> with your actual resource group and cluster names. The --debug and --verbose flags will ensure detailed output, crucial for diagnosing issues.

2. Analyze Debugging Output and Error Messages: Carefully examine the debugging output generated by the az resource update command. Cross-reference the error messages you find with the error list documented in the CSE helper executable file on GitHub. This file, cse_helpers.sh, contains a comprehensive list of potential errors that can occur during CSE execution. You can access this file at CSE helper executable file.

By comparing the error messages from your debugging output with the CSE helper error list, you can pinpoint if the CSE deployment of the kubelet is indeed the source of the “Node Not Ready” failure.

Exit codes are critical indicators of the specific problem. Common examples include error messages like “Unable to communicate with API server” or “Unable to connect to internet.” These messages, along with exit codes, can also signal API network time-outs or underlying node faults that necessitate node replacement. Examining these codes is essential for accurate diagnosis and targeted resolution.

Solution 1: Ensure Correct Custom DNS Server Configuration¶

Incorrectly configured custom DNS servers are a frequent source of network connectivity issues within AKS clusters, leading to CSE errors and node failures. If you are using custom DNS servers, it is essential to verify that they are correctly set up to perform name resolution for your AKS nodes. Proper DNS configuration is vital for nodes to communicate with the Kubernetes API server and other cluster components.

To ensure your custom DNS server is correctly configured, adhere to the following requirements:

• Health and Reachability: Verify that all custom DNS servers are healthy and reachable over the network. Nodes must be able to reliably communicate with these DNS servers to resolve domain names. Network connectivity issues to DNS servers will directly impact node functionality.

• Conditional Forwarders to Azure DNS: Custom DNS servers must be configured with the necessary conditional forwarders to the Azure DNS IP address (or a forwarder to that address). This ensures that DNS queries for Azure-specific services and resources are correctly resolved by Azure DNS. This is especially important for private AKS clusters or when using private endpoints. For more detailed information on conditional forwarders, refer to Private endpoint DNS configuration.

• Private AKS DNS Zone Linking: If your custom DNS virtual networks are hosted on Azure, ensure that your private AKS DNS zone is properly linked to them. This linkage enables name resolution for services within the private AKS cluster from your custom DNS infrastructure. Incorrect linking can lead to resolution failures within the cluster.

• Avoid Mixing Azure DNS IP with Custom DNS IPs: It is strongly discouraged to use the Azure DNS IP address in conjunction with the IP addresses of your custom DNS servers in DNS settings. This mixed configuration can lead to unpredictable DNS resolution behavior and is not a recommended practice.

• Check for IP Addresses in DNS Settings: Avoid using raw IP addresses instead of DNS server names in your DNS configurations. Using IP addresses directly can bypass your custom DNS setup and lead to resolution issues. You can use Azure CLI commands to check for this configuration on Virtual Machine Scale Sets (VMSS) and availability sets that constitute your AKS nodes.

  • For Virtual Machine Scale Set Nodes: Use the az vmss run-command invoke command to execute commands directly on the VMSS instances. The following commands use telnet and nslookup to test DNS resolution using a specified DNS IP address.

  az vmss run-command invoke \
      --resource-group <resource-group-name> \
      --name <vm-scale-set-name> \
      --command-id RunShellScript \
      --instance-id 0 \
      --output tsv \
      --query "value[0].message" \
      --scripts "telnet <dns-ip-address> 53"
  az vmss run-command invoke \
      --resource-group <resource-group-name> \
      --name <vm-scale-set-name> \
      --instance-id 0 \
      --command-id RunShellScript \
      --output tsv \
      --query "value[0].message" \
      --scripts "nslookup <api-fqdn> <dns-ip-address>"
  

  
  Replace <resource-group-name>, <vm-scale-set-name>, <dns-ip-address>, and <api-fqdn> with your specific values. The first command tests connectivity to the DNS server on port 53 (DNS port), and the second command attempts to resolve the Kubernetes API server’s Fully Qualified Domain Name (FQDN) using the specified DNS server.
  • For VM Availability Set Nodes: Similarly, use the az vm run-command invoke command for nodes in availability sets. The commands are analogous to the VMSS commands, but targeted at availability set VMs.

  az vm run-command invoke \
      --resource-group <resource-group-name> \
      --name <vm-availability-set-name> \
      --command-id RunShellScript \
      --output tsv \
      --query "value[0].message" \
      --scripts "telnet <dns-ip-address> 53"
  az vm run-command invoke \
      --resource-group <resource-group-name> \
      --name <vm-availability-set-name> \
      --command-id RunShellScript \
      --output tsv \
      --query "value[0].message" \
      --scripts "nslookup <api-fqdn> <dns-ip-address>"
  

  
  Again, replace placeholders with your actual resource group, availability set name, DNS IP address, and API FQDN.

For comprehensive information on DNS resolution within Azure virtual networks and custom DNS configurations in AKS, consult these resources:

• Name resolution for resources in Azure virtual networks
• Hub and spoke with custom DNS in AKS

Solution 2: Resolve API Network Time-outs¶

API network time-outs can also lead to CSE failures and “Node Not Ready” status. These time-outs occur when nodes cannot reliably communicate with the AKS API server due to network connectivity issues or delays. Ensuring the API server is reachable and responsive is critical for node health.

To troubleshoot API network time-outs, follow these steps:

• Check AKS Subnet Network Security Group (NSG): Examine the Network Security Group (NSG) associated with the AKS subnet. Verify that the NSG rules are not blocking egress traffic on port 443, which is the standard port for HTTPS communication used by nodes to communicate with the API server. Restrictive NSG rules can inadvertently block necessary API traffic.

• Inspect Node-Level NSGs: Nodes themselves might have NSGs applied directly. Check for any NSGs at the node level that could be blocking egress traffic. These NSGs could be applied at a more granular level than the subnet NSG and might be the source of the blockage.

• Examine AKS Subnet Route Table: Investigate the route table associated with the AKS subnet. If a route table is in place and it directs traffic through a Network Virtual Appliance (NVA) or firewall, ensure that port 443 is allowed for egress traffic through the NVA or firewall. NVAs and firewalls can introduce latency or block traffic if not configured correctly. For more details on controlling egress traffic in AKS, see Control egress traffic for cluster nodes in AKS.

• Address API Time-outs: If DNS resolution is successful and the API server is reachable, but node CSE still fails due to API time-outs, take corrective actions based on the node type:

| Set Type | Action