Azure VM Extension Error 50: Troubleshooting Outbound Connection Failures

Table of Contents

When attempting to start, create, or deploy a Microsoft Azure Kubernetes Service (AKS) cluster, you might encounter the OutboundConnFailVMExtensionError. This error is also known by its code ERR_OUTBOUND_CONN_FAIL or simply error number 50. This issue specifically indicates that the cluster nodes are unable to establish necessary outbound network connections required for their provisioning and operation. Resolving this error is critical for successful AKS cluster deployment.

Prerequisites for Troubleshooting¶

Effective troubleshooting of network connectivity issues requires specific tools. Ensure you have access to the following command-line utilities on a machine or environment capable of interacting with your Azure resources and the cluster nodes:

• The Netcat (nc) command-line tool, useful for testing direct TCP/UDP connections to specific ports.
• The dig command-line tool, essential for diagnosing Domain Name System (DNS) issues and verifying name resolution.
• The Client URL (cURL) tool, valuable for testing connectivity to web endpoints, especially when dealing with HTTP/HTTPS and proxy configurations.

Having these tools readily available will significantly aid in diagnosing the root cause of the outbound connectivity failure directly from the perspective of the problematic node.

Symptoms¶

The primary symptom of this error is the failure of the AKS cluster creation or start operation. The Azure portal or command-line interface (CLI) output will display error messages indicating a provisioning failure related to a virtual machine extension. A common message pattern includes references to VMExtensionProvisioningError and a specific exit status code.

The error details typically mention that a Custom Script Extension (vmssCSE) failed to execute a command with an exit status of 50. This status code is a direct indicator of the outbound connection failure. You will often find accompanying stderr output from the command execution attempt on the node, which might show specific network errors like “Connection timed out” when trying to reach a critical endpoint such as mcr.microsoft.com on port 443.

A representative error message excerpt might look like this:

Unable to establish outbound connection from agents, please see https://aka.ms/aks-required-ports-and-addresses for more information.
Details: Code="VMExtensionProvisioningError"
Message="VM has reported a failure when processing extension 'vmssCSE'.
Error message: "Enable failed: failed to execute command: command terminated with exit status=50\n[stdout]\n\n[stderr]\nnc: connect to mcr.microsoft.com port 443 (tcp) failed: Connection timed out\nCommand exited with non-zero status
Error details : "vmssCSE error messages : {vmssCSE exit status=50, output=pt/apt.conf.d/95proxy...}"

This output clearly shows that the vmssCSE failed with status 50 because a nc command to mcr.microsoft.com:443 timed out, indicating a block in outbound traffic.

Cause¶

The fundamental cause of the OutboundConnFailVMExtensionError (error 50) is the inability of the AKS cluster nodes to connect to required external endpoints. During the provisioning phase, the Custom Script Extension (CSE) runs scripts on the nodes to download and install necessary software components, configure networking, and join the cluster. For public AKS clusters, a critical step involves downloading container images and other dependencies from the Microsoft Container Registry (MCR) at mcr.microsoft.com over Transmission Control Protocol (TCP) port 443.

If the network path from the AKS node to mcr.microsoft.com:443 is obstructed, the CSE cannot complete its task, leading to the provisioning failure and the specific error message observed. This obstruction can stem from various sources within the network configuration surrounding the AKS cluster’s virtual network. Common culprits include restrictive firewall rules, incorrect Network Security Group (NSG) configurations, issues with proxy server settings, or problems with DNS resolution preventing the node from finding the correct IP address for the required endpoint. Identifying the exact point of failure requires testing connectivity directly from the affected node.

The best approach to pinpoint the specific network blockage is to connect to an affected node and perform connectivity tests. You can use the Secure Shell (SSH) protocol to connect directly to a node if SSH access is enabled and configured. Detailed instructions on how to connect via SSH are available in the official AKS documentation regarding node access for troubleshooting and maintenance. This direct access allows you to execute network diagnostic commands from the node’s perspective, precisely mimicking the environment where the provisioning script failed.

If SSH access is not feasible or enabled, Azure provides the az vmss run-command invoke functionality. This Azure CLI command allows you to execute scripts or commands on a Virtual Machine Scale Set (VMSS) instance, which is what AKS node pools are based on, without requiring a direct SSH connection. This method is particularly useful for initial diagnostics or automated checks.

Once you have a method to run commands on the node, you can begin testing outbound connectivity. The specific tests depend on whether your cluster uses a direct internet connection for outbound traffic or routes traffic through an HTTP/HTTPS proxy server.

For clusters with direct outbound internet access, you should test the connection to mcr.microsoft.com on port 443 using tools like nc and dig. The nc -vz mcr.microsoft.com 443 command attempts a verbose, zero-I/O connection to verify if the TCP handshake completes successfully. The dig mcr.microsoft.com command checks if the node can resolve the domain name to an IP address via its configured DNS server.

Example commands to run directly on the node (via SSH or az vmss run-command invoke):

# Test direct TCP connectivity to MCR on port 443
nc -vz mcr.microsoft.com 443

# Test DNS resolution for MCR
dig mcr.microsoft.com

If you are using az vmss run-command invoke, you first need to identify an instance ID from your node pool’s VMSS.

# Get the VMSS instance IDs for a node pool (replace placeholders)
az vmss list-instances --resource-group <mc-resource-group-name> \
    --name <vmss-name> \
    --output table

# Use an instance ID to test outbound connectivity via nc (replace placeholders)
az vmss run-command invoke --resource-group <mc-resource-group-name> \
    --name <vmss-name> \
    --command-id RunShellScript \
    --instance-id <vmss-instance-id> \
    --output json \
    --scripts "nc -vz mcr.microsoft.com 443"

# Use an instance ID to test DNS resolution via dig (replace placeholders)
az vmss run-command invoke --resource-group <mc-resource-group-name> \
    --name <vmss-name> \
    --command-id RunShellScript \
    --instance-id <vmss-instance-id> \
    --output json \
    --scripts "dig mcr.microsoft.com"

Interpreting the output: If nc reports “Connection timed out” or “Connection refused”, it indicates a network path issue (firewall, NSG, UDR). If dig fails to return an A or CNAME record for mcr.microsoft.com, it suggests a DNS problem.

For AKS clusters configured to use an HTTP or HTTPS proxy for outbound traffic, the troubleshooting steps involve verifying connectivity first to the proxy server from the node, and then testing connectivity through the proxy to the external endpoint (mcr.microsoft.com). This requires using tools like nc to test the proxy server’s reachability and curl to test web connectivity via the proxy.

Example commands for proxy environments (run on the node via SSH or az vmss run-command invoke):

# Test connectivity from the AKS node to the HTTP/S proxy server
nc -vz <http-s-proxy-address> <port>

# Test traffic from the HTTP proxy server to HTTPS endpoint (MCR)
# Requires the node/environment to be configured to use the proxy for curl
curl --proxy http://<http-proxy-address>:<port>/ --head https://mcr.microsoft.com

# Test traffic from the HTTPS proxy server to HTTPS endpoint (MCR)
# Requires the node/environment to be configured to use the proxy for curl
curl --proxy https://<https-proxy-address>:<port>/ --head https://mcr.microsoft.com

# Test DNS functionality from the node (still relevant even with a proxy)
dig mcr.microsoft.com

When using az vmss run-command invoke for proxy scenarios, you would structure the commands similarly:

# Test connectivity from the AKS node to the HTTP/S proxy server (replace placeholders)
az vmss run-command invoke --resource-group <mc-resource-group-name> \
    --name <vmss-name> \
    --command-id RunShellScript \
    --instance-id <vmss-instance-id> \
    --output json \
    --scripts "nc -vz <http-s-proxy-address> <port>"

# Test traffic from the HTTP proxy server to HTTPS (MCR) using curl via run-command (replace placeholders)
az vmss run-command invoke --resource-group <mc-resource-group-name> \
    --name <vmss-name> \
    --command-id RunShellScript \
    --instance-id <vmss-instance-id> \
    --output json \
    --scripts "curl --proxy http://<http-proxy-address>:<port>/ --head https://mcr.microsoft.com"

# Test traffic from the HTTPS proxy server to HTTPS (MCR) using curl via run-command (replace placeholders)
az vmss run-command invoke --resource-group <mc-resource-group-name> \
    --name <vmss-name> \
    --command-id RunShellScript \
    --instance-id <vmss-instance-id> \
    --output json \
    --scripts "curl --proxy https://<https-proxy-address>:<port>/ --head https://mcr.microsoft.com"

# Test DNS functionality using dig via run-command (replace placeholders)
az vmss run-command invoke --resource-group <mc-resource-group-name> \
    --name <vmss-name> \
    --command-id RunShellScript \
    --instance-id <vmss-instance-id> \
    --output json \
    --scripts "dig mcr.microsoft.com"

For proxy tests, if nc to the proxy server fails, the issue is between the node and the proxy. If nc to the proxy succeeds but curl through the proxy to MCR fails, the issue lies either in the proxy configuration itself, or the proxy server is being blocked from reaching the internet endpoint. DNS resolution tested via dig is still important as the proxy server might perform DNS lookups on behalf of the node, or the node might need to resolve the proxy server’s address.

Solution¶

Resolving the OutboundConnFailVMExtensionError (error 50) involves identifying and removing the network obstruction preventing the AKS nodes from reaching required outbound endpoints like mcr.microsoft.com:443. Based on the troubleshooting steps performed in the Cause section, you can determine the specific type of blockage and apply the corresponding solution. The primary areas to investigate are firewalls, Network Security Groups (NSGs), proxy configurations, and DNS settings.

Here is a detailed breakdown of common issues and their solutions:

Issue	Cause Description	Solution
Traffic blocked by Firewall, Proxy, or NSG	Outbound connections from the AKS nodes to necessary endpoints (like MCR, Azure Active Directory, Azure Monitor, etc.) on specific ports (e.g., 443) are being blocked. This is often due to restrictive rules in an Azure Firewall, Network Virtual Appliance (NVA), NSG applied to the node subnet, or proxy server filtering.	Configure Network Security Rules: Review your Azure Firewall rules, NVA configurations, and NSGs applied to the AKS node pool subnet. Ensure that outbound traffic is allowed to all required FQDNs and IP ranges on the necessary ports. For mcr.microsoft.com, this is TCP port 443. Refer to the official AKS documentation for a comprehensive list of required outbound network rules. If using a proxy, ensure the proxy allows this traffic.
AAAA (IPv6) Record Blocking	Some firewalls or network devices might implicitly or explicitly block DNS queries for AAAA records (IPv6 addresses) or block IPv6 traffic itself. While MCR is reachable via IPv4, blocking IPv6 resolution or traffic could potentially cause issues in dual-stack environments or specific configurations.	Verify Firewall/DNS Filtering: Check your firewall or DNS filtering rules to ensure they are not blocking AAAA record queries or IPv6 traffic to necessary Microsoft endpoints. If your environment is IPv4-only, ensure no configurations are mistakenly attempting or blocking IPv6 connections. Verify that your firewall allows outbound DNS (typically UDP/TCP 53) to your configured DNS servers.
Private Cluster DNS Resolution Failure	In private AKS clusters that use custom DNS servers, the nodes need to resolve internal Azure endpoints (like API servers, MCR via Private Link) as well as external ones. If the custom DNS server cannot resolve these Azure-specific names, or if it doesn’t forward resolution requests for Azure domains to Azure DNS (168.63.129.16), name resolution fails.	Configure Custom DNS Forwarding: If using custom DNS servers with a private AKS cluster, ensure that your DNS servers are configured to forward queries for Azure public zones (like *.blob.core.windows.net, *.mcr.microsoft.com, etc.) to the Azure provided DNS IP address 168.63.129.16. Alternatively, ensure you have configured Azure Private DNS zones correctly and linked them to the AKS VNet. Verify that the custom DNS server is reachable from the AKS nodes.
Incorrect Proxy Configuration on Nodes	If your AKS cluster is configured to use an HTTP/HTTPS proxy, but the proxy settings are incorrectly applied to the nodes or the proxy server is misconfigured or unreachable.	Verify Proxy Settings: Double-check the HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables or system-wide proxy configurations applied to the AKS nodes. Ensure the proxy address and port are correct, the proxy server is running and accessible from the node subnet, and that the proxy server itself is configured to allow connections to the required internet endpoints. Test connectivity to the proxy from the node first, then through the proxy.
User Defined Routes (UDR) Routing Issues	If your AKS subnet uses UDRs to route traffic (e.g., to an NVA or Azure Firewall), incorrect UDR configuration can misdirect or drop outbound traffic intended for required endpoints.	Review UDRs: Examine the Route Table associated with your AKS node pool subnet. Ensure there are routes defined for necessary public endpoints (or a default route 0.0.0.0/0) that correctly point traffic towards your intended egress path (e.g., internet gateway, Azure Firewall, NVA). Verify that there are no conflicting or overly specific UDRs misrouting traffic away from the correct path.

Beyond these specific points, a holistic review of the network path from the AKS nodes to the required endpoints is crucial. This includes:
* Tracing the Network Path: Mentally or physically trace the network path from a representative node’s IP address through the subnet, NSGs, Route Tables (UDRs), potentially through a VNet Peering, to any NVA or Azure Firewall, and finally out to the internet or via Private Link.
* Reviewing NSG Flow Logs: If NSG flow logs are enabled for the AKS node subnet, analyze them to see if traffic destined for mcr.microsoft.com or other required endpoints is being denied by an NSG rule.
* Utilizing Azure Network Watcher: Tools within Azure Network Watcher, such as Connection Troubleshoot, can test connectivity between a source (like an AKS node NIC) and a destination (like mcr.microsoft.com:443) considering all intervening network security controls.

Addressing the identified blockage by adjusting firewall rules, NSG rules, proxy settings, DNS configurations, or UDRs should resolve the OutboundConnFailVMExtensionError. After making network configuration changes, attempt the AKS cluster creation or start operation again. It might be necessary to delete and recreate the failed node pool or the entire cluster if the nodes entered an unrecoverable state.

Best Practices for AKS Outbound Connectivity¶

To prevent OutboundConnFailVMExtensionError and other network-related issues, adopt best practices for managing AKS outbound traffic:

1. Centralize Egress Control: Implement a centralized egress solution using Azure Firewall or a Network Virtual Appliance (NVA) deployed in a hub network. Route all AKS outbound traffic through this central point using User Defined Routes (UDRs) from the AKS subnet. This provides a single point for managing and auditing outbound connections.
2. Use Required FQDNs and Ports: Always allow outbound traffic for the list of Required FQDNs and IP addresses documented by Microsoft for AKS. Do not rely solely on allowing traffic to broad IP ranges if more specific FQDN rules are possible with your firewall technology. Crucially, ensure port 443 (HTTPS) is open to these endpoints.
3. Configure DNS Correctly: Ensure AKS nodes can reliably resolve the FQDNs of required services. If using custom DNS servers, they must be able to forward or resolve public and potentially private Azure endpoints. Test DNS resolution from the nodes frequently during troubleshooting.
4. Implement Azure Private Link: For private clusters or scenarios requiring enhanced security, use Azure Private Link for connecting to supported Azure services like Azure Container Registry (ACR) and potentially MCR (check latest features). This keeps traffic within the Azure backbone network.
5. Regularly Review Network Configuration: Periodically review NSG rules, UDRs, firewall rules, and proxy settings affecting the AKS subnet. Network configurations can change, and unintended rules can block necessary traffic.
6. Enable Network Logging: Utilize NSG Flow Logs, Azure Firewall logs, or NVA logs to gain visibility into network traffic and quickly identify denied connections. Forward these logs to a Log Analytics workspace for querying and analysis.

Adhering to these practices creates a more robust and manageable network environment for your AKS clusters, significantly reducing the likelihood of outbound connectivity failures during provisioning and operation.

More Information¶

For further details on troubleshooting AKS cluster creation issues and managing network egress, consult the following resources:

• General troubleshooting steps for AKS cluster creation problems.
• Detailed documentation on the required outbound network rules, FQDNs, and ports for Azure Kubernetes Service clusters, including considerations for various deployment models (public, private, using proxies, etc.).

Understanding the network dependencies and employing systematic troubleshooting methods are key to quickly resolving errors like OutboundConnFailVMExtensionError and ensuring the smooth operation of your AKS environment.

Disclaimer: This article provides general guidance. Network configurations can be complex and specific to your Azure environment. When consulting third-party documentation or utilizing third-party tools mentioned herein (like Netcat, dig, cURL), be aware that information might change. Microsoft does not guarantee the accuracy of third-party information or the functionality of third-party tools.

Have you encountered this error before? What specific steps did you take to identify and resolve the outbound connectivity issue in your environment? Share your experiences and tips in the comments below!