Azure VHD Error: Solve the ERR_VHD_FILE_NOT_FOUND Problem Quickly

Table of Contents

Creating a Microsoft Azure Kubernetes Service (AKS) cluster involves deploying various Azure resources, including Virtual Machine Scale Sets (VMSS) for the worker nodes. During the provisioning process of these nodes, a Custom Script Extension (CSE) is often utilized to perform setup tasks required for Kubernetes integration. Occasionally, this process can encounter errors, preventing the cluster from being created successfully. One such error is indicated by a specific exit code during the extension’s execution.

This article focuses on understanding and resolving an error encountered during AKS cluster creation that manifests as a VMExtensionProvisioningError with a specific exit status code, often associated internally or in diagnostics with scenarios like ERR_VHD_FILE_NOT_FOUND, even when the root cause points to network-related issues. Understanding the underlying reason for the extension failure is key to a quick resolution.

Symptoms¶

When attempting to deploy or update an Azure Kubernetes Service cluster, you might encounter a provisioning failure. The error details provided in the Azure portal or via Azure CLI output will indicate an issue with a Virtual Machine Scale Set extension, specifically the vmssCSE. The relevant error message typically includes details about the extension failing to process.

A common manifestation of this issue is the following error pattern:

VMExtensionProvisioningError: VM has reported a failure when processing extension ‘vmssCSE’.

Error message: “Enable failed: failed to execute command: command terminated with exit status=65”

The presence of exit status=65 within the vmssCSE failure message is a strong indicator of this particular problem scenario. While the error code 65 itself is a generic exit status indicating command termination with a non-zero value, its occurrence in the context of the vmssCSE during AKS provisioning points towards a specific class of issues.

Understanding the Cause (Exit Code 65)¶

The Azure VM Scale Set Custom Script Extension (vmssCSE) is a powerful tool used by Azure services, including AKS, to automate the installation and configuration of software on virtual machines. During AKS node provisioning, this extension executes scripts to join the nodes to the Kubernetes cluster, configure networking, install container runtimes, and perform other essential setup tasks. These scripts often require downloading components, images, or configuration files from various endpoints on the internet.

The exit code 65 reported by the vmssCSE signifies that the script executed by the extension terminated abnormally. In the specific context of AKS node provisioning, Microsoft documentation indicates that this generic exit code might appear under certain, often rare, circumstances instead of more specific network connectivity errors like those represented by exit codes 50, 51, or 52.

Common network-related error codes during AKS provisioning include:

• OutboundConnFailVMExtensionError (Code 50): Indicates a general failure in outbound connectivity from the VM.
• K8SAPIServerConnFailVMExtensionError (Code 51): Specifically points to a failure in connecting to the Kubernetes API server.
• K8SAPIServerDNSLookupFailVMExtensionError (Code 52): Indicates a failure in resolving the DNS name of the Kubernetes API server or other required endpoints.

The reason exit code 65 might appear instead of these more specific codes is often related to the stage at which the network failure occurs or the nature of the script failure triggered by the lack of connectivity. For example, if the script attempts to download an initial dependency or configuration file from a required endpoint and fails immediately due to a network block or DNS issue, the script might exit prematurely with a generic error code (like 65) before it even attempts subsequent steps that would trigger the more specific 50, 51, or 52 codes.

Despite the seemingly generic nature of exit code 65, when it occurs during AKS vmssCSE provisioning, it is frequently a symptom of underlying network connectivity issues between the AKS nodes (or the VNet subnet where they reside) and essential Azure endpoints required for AKS cluster operation and node setup. These critical endpoints include, but are not limited to, container registries like mcr.microsoft.com (Microsoft Container Registry) and distribution points like acs-mirror.azureedge.net.

The association with “VHD file not found” in some diagnostic contexts might stem from the provisioning process failing to download or access necessary components (potentially packaged in a way related to VHDs or required for disk setup) due to network blocks, leading to a failure that is interpreted or reported in a misleading way by a higher-level system component. However, based on the provided documentation guidance, the primary focus for troubleshooting this specific error code (65) should be on outbound network connectivity.

Troubleshooting and Solution¶

Resolving the ERR_VHD_FILE_NOT_FOUND error, as signaled by vmssCSE exiting with status 65, primarily involves diagnosing and correcting outbound network connectivity problems from your AKS worker nodes’ subnet. The AKS control plane and worker nodes require access to specific Azure and Microsoft services over the internet to function correctly, especially during initial provisioning and scaling operations.

The recommended solution is to thoroughly review and validate your network configuration to ensure that the AKS nodes have the necessary outbound access. This includes examining Network Security Group (NSG) rules, User Defined Routes (UDRs), and any firewalls or Network Virtual Appliances (NVAs) that might be inspecting or blocking traffic originating from the AKS subnet.

Here’s a detailed approach to troubleshooting:

1. Understand AKS Outbound Requirements:
   AKS clusters need to connect to specific FQDNs (Fully Qualified Domain Names) and IP addresses on particular ports. The exact requirements depend on your cluster configuration (e.g., basic vs. standard load balancer, UDRs, Azure Firewall). You must ensure that traffic to these endpoints is permitted.

   Essential endpoints often include:
   * mcr.microsoft.com and related CDNs: For pulling base images and components.
   * acs-mirror.azureedge.net: Another endpoint for distributing AKS-related components.
   * Azure API endpoints (e.g., *.management.azure.com): For control plane communication with Azure APIs.
   * Azure Container Registry (ACR) if you use it to pull images: Requires connectivity to your specific ACR instance.
   * Microsoft Entra ID endpoints: For authentication and authorization.
   * Specific Kubernetes endpoints: Depending on add-ons or features.

   The primary ports required are typically TCP 443 (HTTPS) and sometimes TCP 9000 (for tunnel communication in certain network configurations).

2. Review Network Security Groups (NSGs):
   Check the NSG associated with the subnet where your AKS nodes are deployed. Ensure that outbound rules do not block traffic to the required FQDNs or their corresponding IP ranges/service tags on the necessary ports. Azure provides Service Tags (like AzureContainerRegistry, AzureActiveDirectory, AzureResourceManager, MicrosoftContainerRegistry) which can simplify NSG rule management, allowing outbound access to known Azure service IP ranges without needing to maintain specific IP lists. Ensure you are using appropriate service tags or explicit rules that allow outbound traffic to the internet (if not routing through a firewall) or to your firewall appliance.

3. Inspect User Defined Routes (UDRs):
   If your subnet uses UDRs, traffic might be routed away from the default internet path, typically towards a firewall appliance (like Azure Firewall, or a third-party NVA) or a VPN gateway.

   • If traffic is routed to a firewall, verify that the firewall’s rules explicitly allow the necessary outbound connections (FQDNs, ports). Check the firewall logs for denied connections originating from the AKS subnet IP ranges.
   • If traffic is routed elsewhere, ensure that path correctly handles the required outbound internet traffic for AKS.

4. Examine Firewall / NVA Configuration:
   If you have a firewall or NVA, this is a common point where outbound traffic is filtered.

   • FQDN Filtering: Ensure your firewall permits access to all required AKS FQDNs.
   • Port Filtering: Verify that TCP 443 and any other required ports are open for outbound connections to these FQDNs.
   • TLS/SSL Inspection: If your firewall performs TLS/SSL inspection, it might interfere with AKS communication. Ensure that traffic to AKS-required endpoints is exempted from inspection, or that the inspection is configured correctly to not break the connections.
   • Logs: Analyze firewall logs for any denied connections originating from the AKS worker node IPs during the cluster creation time.

5. Verify DNS Resolution:
   Even if network paths are open, failed DNS resolution will prevent connections.

   • If using custom DNS servers in your VNet (e.g., on-premises DNS or DNS forwarders), ensure they can correctly resolve public FQDNs like mcr.microsoft.com, acs-mirror.azureedge.net, etc.
   • You can test DNS resolution from a VM deployed in the same subnet as the AKS nodes would be. Use tools like nslookup or dig to query the public FQDNs.

   # Example using nslookup from a test VM in the AKS subnet
   nslookup mcr.microsoft.com
   nslookup acs-mirror.azureedge.net
   nslookup <your-acr-name>.azurecr.io # If using ACR
   

   Ensure that the DNS queries return valid public IP addresses.

6. Test Connectivity from a Test VM:
   Deploy a simple test Virtual Machine into the same VNet and subnet where you are trying to create the AKS cluster. This VM will inherit the same NSGs and UDRs as the AKS nodes.
   From this test VM, attempt to establish connections to the required endpoints on the necessary ports using tools like curl, wget, or telnet.

   # Example using curl
   curl -v https://mcr.microsoft.com
   
   # Example using telnet (useful for port check)
   telnet mcr.microsoft.com 443
   

   Successful curl commands should return HTTP responses (even an error page indicates connectivity). Successful telnet should connect and show a blinking cursor or a welcome message. Connection timeouts or failures indicate a network block.

7. Check VM Extension Logs (If Possible):
   If the VM Scale Set instances are briefly available before deletion, or if you can configure a non-expiring VMSS for debugging, you might be able to access the logs from the failed vmssCSE execution.
   On a Linux VM, logs for extensions are typically located under /var/log/azure/. Look for directories or files related to the custom script extension or VMSS extension. The logs might contain more detailed error messages from the script itself, which could pinpoint exactly which command or network call failed, even if the final reported status was a generic 65.

8. Review AKS Service Health:
   Occasionally, regional Azure service issues can impact cluster creation. Check the Azure Service Health dashboard for any ongoing incidents affecting Azure Kubernetes Service, Virtual Machine Scale Sets, Networking, or other related services in the region you are deploying to.

9. Verify Azure Resource Provider Registration:
   Ensure that necessary resource providers like Microsoft.ContainerService, Microsoft.Compute, Microsoft.Network, and Microsoft.Storage are registered in your Azure subscription. Unregistered providers can prevent resource creation. This is less likely to cause a vmssCSE exit 65 specifically, but it’s a good general troubleshooting step for creation failures.

10. Consider Transient Issues:
    Network issues can sometimes be transient. If you’ve verified your configuration seems correct, waiting a few minutes and retrying the AKS cluster creation might resolve the issue, especially if the problem was temporary congestion or a backend service hiccup. However, consistently getting exit code 65 strongly suggests a persistent configuration problem.

By systematically checking NSGs, UDRs, firewalls, DNS resolution, and performing connectivity tests from within the affected subnet, you can pinpoint the network blockage preventing the vmssCSE from completing its setup tasks. Once the outbound connectivity to the required AKS endpoints is restored, the AKS cluster creation should proceed successfully without the exit status=65 error.

To help visualize the potential network path, consider this simplified diagram:

```mermaid
graph LR
subgraph “Azure VNet”
subgraph “AKS Subnet”
A[AKS Node VMSS Instance]
end
B(Network Security Group
Applied to Subnet)
C(User Defined Routes
Applied to Subnet)
end
D(Azure Firewall
or NVA)
E(Internet)
subgraph “Microsoft Endpoints”
F(mcr.microsoft.com)
G(acs-mirror.azureedge.net)
H(Azure APIs)
end

A --> B
A --> C
C --> D
D --> E
B --> E
E --> F
E --> G
E --> H
F --> A
G --> A
H --> A

```
Diagram: Simplified view of network traffic flow from an AKS node instance.

The diagram illustrates how outbound traffic from an AKS node instance might flow through NSGs and UDRs, potentially being routed via a firewall/NVA before reaching the internet and the required Microsoft endpoints. Any block along this path will cause the connectivity failure.

It is critical to ensure that the network configuration allows bidirectional communication as needed, although the initial vmssCSE failure with exit code 65 is most often tied to the inability to initiate outbound connections or resolve names.

Summary¶

The VMExtensionProvisioningError with exit status=65 during Azure AKS cluster creation, often conceptually linked to “VHD not found” in some diagnostic contexts, is a strong indicator of underlying outbound network connectivity issues. The AKS node provisioning scripts, executed via the vmssCSE, require access to essential Microsoft endpoints like mcr.microsoft.com and acs-mirror.azureedge.net. Failure to reach these endpoints due to restrictive NSG rules, UDRs routing traffic through blocking devices, firewall configurations, or DNS resolution problems will cause the extension script to fail, resulting in exit code 65.

Troubleshooting involves a detailed review of your VNet’s network configuration, including NSGs, UDRs, and any firewalls, ensuring that outbound access to the required AKS FQDNs and ports is permitted. Performing connectivity and DNS tests from a test VM within the affected subnet is a highly effective way to diagnose the specific network blockage.

By systematically addressing potential network restrictions based on the required AKS outbound rules, you can resolve the vmssCSE exit code 65 error and allow your AKS cluster creation to complete successfully.

Have you encountered this specific error? What network troubleshooting steps did you find most effective? Share your experiences and insights in the comments below!