Azure Virtual Network Connectivity: Troubleshooting External Endpoint Access

Table of Contents

Connecting from a Microsoft Azure Kubernetes Service (AKS) cluster to endpoints located outside its virtual network, specifically over the public internet, is a fundamental requirement for many applications. This connectivity allows AKS pods to pull container images from external registries, access external APIs, communicate with third-party services, or even connect to Azure services exposed publicly. However, network configuration complexities within Azure, coupled with application-specific requirements and potential resource constraints, can sometimes lead to issues with this outbound connectivity. Understanding the various components involved in AKS outbound traffic and having a structured approach to troubleshooting is crucial for maintaining reliable communication with external services. This article provides a comprehensive guide to diagnosing and resolving common problems encountered when AKS clusters fail to reach external endpoints.

Outbound connectivity from an AKS cluster is managed by its network configuration, which determines how traffic exits the cluster’s virtual network. The chosen outbound type significantly impacts the troubleshooting steps required when connectivity issues arise. Various Azure networking components such as Network Security Groups (NSGs), User Defined Routes (UDRs), Azure Load Balancer, or Azure NAT Gateway play a role in directing and securing this outbound traffic. Misconfigurations or limitations within any of these components can manifest as connectivity failures or intermittent issues. Therefore, diagnosing the root cause involves examining the network path, validating configurations, and monitoring resource utilization within the cluster nodes and the underlying Azure infrastructure.

Prerequisites¶

Before diving into the troubleshooting steps, ensure you have the necessary tools installed and configured to interact with your Azure environment and the AKS cluster. These tools provide the interface to query configurations, execute commands within the cluster, and test connectivity. Having these prerequisites ready streamlines the diagnostic process, allowing you to quickly gather information and perform tests.

• Azure CLI: The Azure Command-Line Interface (CLI) is essential for managing Azure resources from your terminal. It is used to query AKS cluster configuration, network settings, and other related Azure services like Load Balancers or NAT Gateways. Make sure you have a recent version installed and are logged in to your Azure account with appropriate permissions.
• Curl: The curl command-line tool is widely used for making requests to web servers and is invaluable for testing connectivity to external HTTP/HTTPS endpoints. It allows you to see the response code, headers, and body of a request, providing critical clues about the nature of the connectivity problem. Ensure curl is available on the system you are using to troubleshoot, or ideally, within a test pod deployed to your AKS cluster.
• Kubectl: The Kubernetes command-line tool, kubectl, is used to interact with the AKS cluster’s control plane. You will use kubectl to deploy test pods, execute commands within pods (like curl), check pod and node status, and monitor resource usage. Install kubectl and configure it to connect to your target AKS cluster. You can easily install kubectl using the Azure CLI command az aks install-cli.

With these tools at your disposal, you are equipped to begin the troubleshooting process effectively.

Troubleshooting Checklist¶

Troubleshooting network connectivity issues requires a systematic approach. The steps you take will depend on whether the issue is constant (persistent) or happens occasionally (intermittent). This checklist guides you through the common areas to investigate for each scenario.

Is the Issue Persistent?¶

A persistent outbound connectivity issue means that connections to external endpoints consistently fail every time you attempt them. This usually points to a fundamental configuration problem in the network path or a blocking rule preventing traffic flow.

Step 1: Do Basic Troubleshooting¶

Start with fundamental checks to ensure there isn’t a simple misconfiguration. Although this section focuses on external connectivity, confirming basic internal and outbound connectivity is a good starting point. Ensure pods can communicate with each other (if applicable) and that the cluster has some basic level of outbound access, even if limited. Referencing general guides on AKS outbound troubleshooting can provide a baseline.

Basic checks might include:
* Verifying that the pod attempting the connection is healthy and running.
* Checking if the application inside the pod is configured with the correct external endpoint address and port.
* Confirming that the external endpoint itself is reachable and operational from outside the Azure network (e.g., from your local machine or a test VM outside the VNet).

Step 2: Determine the Outbound Type for the AKS Cluster¶

The way your AKS cluster sends outbound traffic is determined by its configured outbound type. This is a critical piece of information as it dictates which Azure networking components are responsible for the traffic flow and where to look for potential blocks. AKS supports several outbound types: loadBalancer, userDefinedRouting, and managedNATGateway.

To identify the outbound type configured for your AKS cluster, use the Azure CLI command az aks show:

az aks show --resource-group <resource_group_name> --name <cluster_name> --query "networkProfile.outboundType" -o tsv

Replace <resource_group_name> and <cluster_name> with your actual resource group and cluster name. The -o tsv flag simplifies the output to just the value.

• If the outbound type is loadBalancer: This is the default type for AKS clusters using the Azure CNI network plugin. Outbound traffic uses a Standard Azure Load Balancer’s public IP address(es). There is no default route table (UDR) configured by AKS in this scenario, unless you are using the older kubenet network plugin.
  • If using kubenet: Ensure the default route table associated with the node subnet does not contain custom routes that incorrectly redirect or blackhole internet traffic (0.0.0.0/0).
  • If using Azure CNI (including Dynamic Allocation and Overlay): Outbound traffic is managed by the Load Balancer’s SNAT rules. Check the Network Security Group (NSG) applied to the AKS node subnet. While AKS manages a default NSG, custom rules might have been added that block outbound internet access (e.g., denying all outbound traffic or specific ports/protocols required by your application). Examine the outbound security rules in the NSG.

• If the outbound type is userDefinedRouting (UDR): This indicates that AKS is configured to use a custom route table. Outbound traffic from the cluster nodes is routed according to the rules in this table, typically directing 0.0.0.0/0 traffic to a next hop, which is often a Network Virtual Appliance (NVA) like a firewall (e.g., Azure Firewall, third-party firewall).

  • Verify that the route table associated with the AKS node subnet is correctly configured with a default route (0.0.0.0/0) pointing to your intended egress device (e.g., the private IP of the firewall).
  • Crucially, ensure the egress device (firewall or proxy) is reachable from the AKS nodes. Check the firewall’s network configuration and routing.
  • Confirm that the egress device is configured to allow the necessary outbound traffic from the AKS cluster’s source IPs/subnets to the external destination endpoint on the required ports and protocols. AKS requires access to specific FQDNs (Fully Qualified Domain Names) for control plane communication and other Azure services. Beyond these, your application’s specific external dependencies must also be allowed. To get the list of FQDNs required by AKS itself, you can use the Azure CLI:

  az aks egress-endpoints list --resource-group <resource_group_name> --name <cluster_name>
  

  
  Ensure your firewall rules permit traffic to these essential endpoints in addition to your application’s specific external destinations.
  * If the outbound type is managedNATGateway: This type uses an Azure NAT Gateway resource for outbound connectivity. All outbound traffic from the subnets associated with the NAT Gateway will use the NAT Gateway’s static public IP address(es).
  * Verify that the AKS subnet is correctly associated with the NAT Gateway resource. You can check this using the Azure CLI:

  az network nat gateway show --resource-group <resource_group_name> --name <nat_gateway_name> --query "subnets[].id"
  

  
  Replace <resource_group_name> and <nat_gateway_name> with your values. The output should show the resource ID of your AKS node subnet.
  * NAT Gateway provides SNAT. While it offers a large number of SNAT ports, potential issues could relate to TCP idle timeout (default 4 minutes) or resource limits if an extremely high number of connections is rapidly opened without reuse. Ensure your application handles idle connections appropriately.
  * Like other outbound types, check any NSGs applied to the AKS node subnet. Even with a NAT Gateway, an NSG could still block outbound flows based on port or destination.

Understanding the outbound type directs your focus to the relevant Azure resource (Load Balancer, Route Table/NVA, or NAT Gateway) and associated NSGs for further investigation.

Step 3: Examine the Curl Output When You Connect to the Application Pod¶

Once you have identified the outbound path, the next step is to test connectivity from within the AKS cluster. Deploy a simple test pod (e.g., using a standard Linux image) to the AKS node pool where your application pod is running. Use kubectl exec to run curl commands from inside this test pod targeting the external endpoint.

# First, find the name of your test pod
kubectl get pods

# Execute curl from inside the pod (replace <pod_name> and <external_endpoint_url>)
kubectl exec <pod_name> -- curl -vvv <external_endpoint_url>

The -vvv flag provides verbose output, showing the connection process, negotiation, and response headers, which is crucial for diagnosing issues.

Analyze the curl output:
* Connection Refused/Timeout: If curl reports a connection refused or timeout error, it strongly suggests a network path issue. The traffic is either not reaching the destination, or the destination is actively rejecting the connection at the network level. This could be due to incorrect routing (UDR), a blocking firewall/NSG rule, or the destination endpoint being unreachable from the internet.
* HTTP Status Codes: If the connection succeeds but you receive an HTTP status code in the response, the network path is likely functional, but the issue lies at the application level or with an application-aware network device (like an application firewall or proxy).
* 4xx Client Error codes: These indicate the server received the request but determined there was an issue with the request itself. Examples include 400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found. While 4xx are technically client errors, receiving them unexpectedly could indicate a network device (like a proxy or application firewall) intercepting the request and returning an error because it doesn’t like the request format, headers, or content, or due to access policies configured on the device. It could also simply mean the requested resource doesn’t exist on the external server.
* 5xx Server Error codes: These indicate the server failed to fulfill a valid request. Examples include 500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable. A 5xx error from the external endpoint suggests the issue is on their side. However, a 502 or 504 (Gateway Timeout) could indicate an issue with a gateway or proxy in the path between your AKS cluster and the external endpoint, possibly your own egress device (firewall/NVA) or an upstream proxy.

Refer to standard HTTP status code definitions for detailed meanings:

Information Source	Link
Internet Assigned Numbers Authority (IANA)	Hypertext Transfer Protocol (HTTP) status code registry
Mozilla	HTTP response status codes
Wikipedia	List of HTTP status codes

Analyzing the verbose curl output alongside the HTTP status code (if any) helps narrow down whether the problem is a network blockage or an application/server-side issue.

Step 4: Check What Happens If Outbound Traffic Bypasses the Virtual Appliance Temporarily (UDR specific)¶

If your AKS cluster uses userDefinedRouting and directs traffic through a Network Virtual Appliance (NVA) like a firewall, that appliance is the most likely place for a persistent block. To quickly test if the NVA is the culprit, you can temporarily modify the route table associated with the AKS subnet to bypass the NVA for the specific external endpoint you are trying to reach, or even for all internet traffic (0.0.0.0/0).

Caution: Temporarily changing routing can expose your AKS nodes directly to the internet or other networks. This should only be done in a controlled test environment or with extreme caution and for a very limited duration in production, and only if your security policies allow. Document the original configuration before making changes.

The steps would involve:
1. Go to the route table resource in the Azure portal or use Azure CLI.
2. Find the route that directs traffic to your NVA (likely for destination prefix 0.0.0.0/0).
3. You can either:
* Modify this route to change the “Next hop type” from “Virtual appliance” to “Internet”.
* Add a more specific route for the problematic external endpoint’s IP address or subnet, setting its “Next hop type” to “Internet”. Route lookups use the most specific match.
4. Save the changes to the route table.
5. Test connectivity from a pod in the AKS cluster again using curl.

If connectivity succeeds after bypassing the NVA, the issue is definitively within the NVA’s configuration (firewall rules, NAT rules, proxy settings, etc.). You should then revert the routing change immediately and investigate the NVA’s logs and configuration thoroughly. Look for denied connection attempts originating from your AKS node subnet’s IP range towards the external destination. Adjust the NVA rules to explicitly permit this traffic.

Is the Issue Intermittent?¶

Intermittent outbound connectivity problems are often harder to diagnose than persistent ones because they occur unpredictably. These issues are frequently related to resource exhaustion or transient network conditions rather than a permanent blockage.

Step 1: Check If the Pod or Node Resources Are Exhausted¶

Resource constraints on the AKS nodes or within the application pods themselves can lead to intermittent connectivity failures. When a node or pod is under heavy load (CPU, memory), network operations might time out or be delayed significantly, appearing as connectivity issues.

Use kubectl top to check current resource usage:

kubectl top pods --all-namespaces
kubectl top nodes

kubectl top pods shows CPU and memory usage for pods across all namespaces (or a specific namespace if you add -n <namespace>). Look for pods exhibiting consistently high resource usage, especially the pods experiencing connectivity issues. If a pod is hitting its configured resource limits (resources.limits in its YAML), it might be throttled, affecting its ability to make timely outbound connections.

kubectl top nodes shows the aggregate CPU and memory usage for each node. If nodes are consistently running at high utilization, it indicates the node pool is undersized, and resource contention could be impacting network performance for all pods on that node.

Consider setting appropriate resource requests and limits for your pods. If nodes are overutilized, scale up the node pool or add another node pool with larger VM sizes.

Step 2: Check If the Operating System Disk Is Used Heavily¶

High I/O operations on the node’s operating system (OS) disk can also contribute to intermittent performance issues, including network latency or timeouts. AKS nodes use Azure Virtual Machine Scale Sets (VMSS) for their underlying infrastructure. The OS disk is where the node’s operating system and container runtime data reside. Heavy logging, temporary file creation, or other disk-intensive tasks by pods running on the node can saturate the disk’s I/O capabilities.

To check OS disk usage for the VMSS backing your AKS node pool:
1. Go to the Azure portal.
2. Search for and select Virtual machine scale sets.
3. Find the scale set corresponding to your AKS node pool (its name usually contains the AKS cluster name). Select it.
4. In the left-hand menu, under Monitoring, select Metrics.
5. Configure the chart to view disk metrics:
* Scope: Select your VMSS name.
* Metric Namespace: Select Virtual Machine Host.
* Metric: Look for OS Disk Read Operations/sec, OS Disk Write Operations/sec, OS Disk Read Bytes/sec, OS Disk Write Bytes/sec. Analyze these metrics to see if the OS disk is experiencing high I/O.

Azure Advisor can also provide recommendations related to disk utilization for your AKS cluster:
1. Go to the Azure portal.
2. Search for and select Kubernetes services.
3. Select your AKS cluster name.
4. In the left-hand menu, under Monitoring, select Advisor recommendations.
5. Review any recommendations related to high disk usage or potential performance bottlenecks.

If high OS disk usage is identified, consider the following remedies:
* Increase OS disk size: A larger OS disk often comes with higher I/O limits. You might need to recreate the node pool to apply a different OS disk size.
* Switch to Ephemeral OS disks: Ephemeral OS disks are created on the node’s temporary storage or the VM’s cache. They offer lower latency and higher throughput compared to managed OS disks but are non-persistent (data is lost on reimage/rescale). This is often a good choice for AKS nodes as container images and cluster state are typically stored elsewhere (container registry, etcd).

If high OS disk I/O persists even after considering these remedies, investigate the applications/pods running on the nodes to identify which processes are performing heavy disk operations and optimize their behavior or redirect disk-intensive tasks to mounted data disks instead of relying solely on the OS disk.

Step 3: Check If the Source Network Address Translation (SNAT) Port Is Exhausted¶

This is a very common cause of intermittent outbound connection issues, especially with the loadBalancer outbound type. When pods initiate outbound connections to public IP addresses, Azure Load Balancer or NAT Gateway performs Source Network Address Translation (SNAT). This means the source IP and port of the outgoing connection are translated to a public IP and a source port owned by the Load Balancer or NAT Gateway. Each unique outbound connection requires a unique SNAT port.

Azure Load Balancer has a finite number of SNAT ports available per backend instance (node). If your application within AKS makes a very large number of concurrent or rapid outbound connections without reusing TCP connections, it can exhaust the available SNAT ports on the node’s assigned public IP. When a node runs out of SNAT ports, subsequent outbound connection attempts from that node will fail until ports become available.

Symptoms of SNAT port exhaustion include:
* Outbound connection failures occurring intermittently.
* Connections timing out.
* Error messages like “Connection refused” or “Address not available”.

You can monitor SNAT port usage for Standard Load Balancer using Azure Monitor metrics:
1. Go to the Azure portal.
2. Search for and select Load balancers.
3. Select the Load Balancer associated with your AKS cluster (if using loadBalancer outbound type).
4. In the left-hand menu, under Monitoring, select Metrics.
5. Configure the chart:
* Scope: Select your Load Balancer resource.
* Metric Namespace: Select Standard Load Balancer Probe and SNAT Metrics.
* Metric: Select SNAT connection count.
* Split by Frontend IP Address and/or Backend IP Address.
Analyze this metric to see if the SNAT connection count is approaching the limit (typically 64,000 outbound SNAT ports per frontend IP per backend instance, but affected by the outbound rules configuration).

If you are experiencing or at risk of SNAT port exhaustion:
* Optimize Application Behavior: The best long-term solution is often to design your application to use connections efficiently. Reuse existing connections for multiple requests (e.g., use HTTP connection pooling). Avoid creating a new connection for every single request. Configure appropriate TCP keep-alives if the protocol supports it.
* Increase Available SNAT Ports:
* Increase the number of public IPs on the egress device: If using loadBalancer outbound type, you can associate more public IP addresses with the Load Balancer’s frontend. Each additional public IP provides more SNAT ports. You can manage the number of outbound public IPs for a managed load balancer associated with AKS using az aks update:

```azurecli
az aks update --resource-group <resource_group_name> --name <cluster_name> --load-balancer-managed-outbound-ip-count <number_of_public_ips>
```
*   **Increase ports per node**: For `loadBalancer` outbound type, you can configure the outbound rules of the Load Balancer to allocate a higher number of SNAT ports per node. This is configured when creating or updating the cluster or node pool. For example, you can use `--load-balancer-managed-outbound-ports <number_of_ports>`. The default is 0, which allows Azure to dynamically allocate ports, up to ~64k per public IP, shared among nodes. Specifying a fixed number per node (`<number_of_ports>`) guarantees that many ports per node, potentially reducing the dynamic allocation risk but also limiting the total pool if not managed carefully across many nodes. Refer to AKS documentation for details on configuring allocated outbound ports.
*   **Switch to NAT Gateway**: If `loadBalancer` SNAT limits are consistently problematic despite tuning, consider changing the outbound type to `managedNATGateway`. NAT Gateway provides a much larger pool of SNAT ports (64,000 per public IP address associated with the gateway, up to 16 IPs) and is designed for high-volume outbound connections.

Addressing SNAT exhaustion requires either reducing the rate at which new outbound connections are established or increasing the pool of available SNAT ports.

Consider checking logs on your AKS nodes for any network-related errors if the above steps don’t pinpoint the issue. Syslog or dmesg might contain messages indicating connection problems at the OS level.

Here’s a simplified visual representation of different outbound types:

```mermaid
graph TD
A[Pod in AKS] → B[Node VM NIC]

B --> |OutboundType: loadBalancer| C[Standard Azure Load Balancer]
C --> |SNAT| D[Load Balancer Public IP(s)]
D --> E[Internet]

B --> |OutboundType: userDefinedRouting| F[Route Table (UDR)]
F --> |Next Hop: NVA| G[Network Virtual Appliance (Firewall/Proxy)]
G --> H[Internet]

B --> |OutboundType: managedNATGateway| I[Subnet (Associated with NAT GW)]
I --> J[Azure NAT Gateway]
J --> K[NAT Gateway Public IP(s)]
K --> L[Internet]

E --> M[External Endpoint]
H --> M
L --> M

Subgraph NSG
B -- Check --> O[Network Security Group]
end
O --> |Filter Outbound Traffic| E
O --> |Filter Outbound Traffic| H
O --> |Filter Outbound Traffic| L

```
This diagram illustrates how outbound traffic flows depending on the configured outbound type and where NSGs might apply filtering.

For a deeper dive into AKS networking concepts or troubleshooting common issues, relevant video resources can be helpful. Search platforms like YouTube for official Microsoft Azure videos or reputable community guides on AKS networking, outbound connectivity, or specific topics like SNAT exhaustion.

Example of a relevant search term for YouTube: “Azure AKS outbound connectivity troubleshooting” or “AKS SNAT exhaustion”.

It’s important to approach network troubleshooting methodically, starting from the basics and progressively investigating the components involved based on the AKS outbound configuration. Analyzing logs from pods, nodes, and relevant Azure network resources (firewall logs, load balancer metrics) is key to uncovering the root cause of persistent or intermittent connectivity issues.

Have you encountered similar outbound connectivity challenges with your AKS clusters? Share your experiences and the solutions you found effective in the comments below. What specific tools or techniques have helped you the most in diagnosing these issues?