Troubleshooting Kubelet Log Retrieval: Resolving TCP 10250 I/O Timeouts in Azure

Table of Contents

Retrieving logs from Kubernetes pods is a fundamental operation for debugging applications and understanding cluster behavior. In Azure Kubernetes Service (AKS), a common challenge encountered by administrators and developers is the dial tcp <IP-Address>:10250: i/o timeout error when attempting to fetch Kubelet logs. This specific error indicates a network communication problem, most frequently a blockage of internal traffic flowing between your AKS cluster’s components, particularly involving the Kubelet on your worker nodes.

The Kubelet is a crucial agent that runs on each node in your Kubernetes cluster, responsible for managing pods and their containers. It exposes an API on port 10250 for various operations, including the secure retrieval of container logs. When traffic to this port is impeded, functionalities like kubectl logs become dysfunctional, hindering vital diagnostic efforts. This guide will provide a comprehensive approach to understanding, diagnosing, and resolving these TCP 10250 timeouts within your Azure environment.

Understanding Kubelet and Log Retrieval Mechanics¶

The Kubelet acts as the primary node agent, ensuring containers are running in a pod and managing resource utilization. One of its key responsibilities is to expose an API endpoint, typically on port 10250 (secure HTTPS by default), which the Kubernetes API server uses to interact with the node. This interaction includes operations like fetching container logs, executing commands within containers, and attaching to running processes. When you execute kubectl logs <pod-name>, your command is not directly sent to the Kubelet.

Instead, the kubectl client first communicates with the Kubernetes API server. The API server then acts as a proxy, forwarding the log request to the Kubelet on the specific worker node where the target pod is scheduled. This secure communication between the API server and the Kubelet is crucial for the integrity and security of your cluster. Any interruption in this network path can manifest as the observed I/O timeout.

In Azure AKS, especially with private clusters or advanced networking configurations, additional “tunnel service pods” like Konnectivity agents might be deployed. These agents facilitate secure communication between the API server and Kubelets, particularly when worker nodes are in a private network segment. If these tunnel services are not operational or their communication is blocked, it can also contribute to the inability to reach Kubelets.

Identifying the Root Cause: Network Security Groups (NSGs)¶

TCP timeouts, particularly on a specific port like 10250, are almost invariably linked to network blockages. In Azure, Network Security Groups (NSGs) are the primary mechanism for filtering network traffic to and from Azure resources within a virtual network. An NSG contains a list of security rules that allow or deny inbound or outbound network traffic based on various criteria, such as source IP address, destination IP address, source port, destination port, and protocol.

These NSGs are typically associated with subnets where your AKS cluster nodes reside, or directly with the network interfaces of individual virtual machines in a Virtual Machine Scale Set (VMSS). While AKS automatically configures default NSG rules for basic cluster operation, custom NSG rules or modifications made by users or other automation tools can inadvertently block essential internal cluster communication. For instance, a restrictive NSG rule might explicitly deny inbound traffic on port 10250 to the node subnet, preventing the API server from reaching the Kubelet. It is essential to carefully review all custom NSG rules to identify any that might be interfering with necessary Kubernetes internal communications.

The Role of Azure Private Link and Konnectivity¶

For private AKS clusters, Azure Private Link is often utilized to secure the communication path between your virtual network and the AKS control plane. This creates a private endpoint within your VNet, eliminating exposure to the public internet. Furthermore, components like the Konnectivity agent (or tunnel service pods mentioned earlier) become paramount in private clusters.

These agents create a secure, bi-directional tunnel between the Kubernetes API server and the Kubelets on worker nodes. This tunnel ensures that the API server can communicate with the Kubelets even if the nodes are in an isolated private subnet without direct network routing to the API server’s private endpoint. If the network path for these Konnectivity agents (e.g., their own required ports or the Kubelet’s port 10250) is blocked by an NSG, the entire communication tunnel collapses, leading to connectivity failures and timeouts.

Symptoms of TCP 10250 I/O Timeouts¶

The most direct and common symptom of this issue is the failure of kubectl logs commands. When attempting to retrieve logs from a pod, you will receive an error message similar to the one provided in the original problem statement:

kubectl logs $POD_NAME

Results:

Error from server: Get "https://aks-agentpool-xxxxxxxxx-vmssxxxxxxxxx:10250/containerLogs/vsm-mba-prod/mba-api-app-xxxxxxxxxx/technosvc": dial tcp <IP-Address>:10250: i/o timeout

This error explicitly points to the inability of the API server (or the tunnel service acting on its behalf) to establish a TCP connection to the Kubelet on port 10250. The IP address shown will correspond to one of your AKS worker nodes. Beyond kubectl logs, you might also observe other related issues:

• kubectl exec failures: Commands like kubectl exec -it <pod-name> -- bash will also fail with similar timeout errors, as they rely on the same Kubelet API.
• kubectl attach failures: Attaching to a running container will similarly fail.
• Missing pod/node metrics: Monitoring tools that scrape metrics directly from the Kubelet’s /metrics endpoint might report missing data for affected nodes.
• Pod status stuck: In some severe cases, pods might remain in ContainerCreating or other transient states if the API server cannot effectively communicate with the Kubelet to fully manage them.

Diagnosing Tunnel Connectivity¶

If your AKS cluster utilizes tunnel service pods (like Konnectivity), it’s crucial to ensure they are running correctly on all nodes. You can inspect their status using kubectl:

kubectl get pods -n kube-system -o wide | grep konnectivity
# Or for generic tunnel services
kubectl get pods -n kube-system -o wide | grep tunnel

Examine the output to ensure that a Konnectivity agent pod is running and healthy on every worker node. If these pods are missing, crashing, or stuck in a pending state on certain nodes, it could indicate underlying issues preventing the secure tunnel from forming, which might also be network-related. A common scenario is that these tunnel pods themselves cannot communicate to the API server or cannot establish their necessary connections due to NSG blocks, leading to the broader log retrieval issue.

Connecting to Your Azure Kubernetes Service (AKS) Cluster¶

Before you can begin troubleshooting, you must establish a connection to your AKS cluster. This involves configuring your local kubectl context to authenticate with your specific AKS instance. Ensure you have the Azure CLI installed and logged in to the correct subscription.

First, define your resource group and cluster name as environment variables for convenience:

export RESOURCE_GROUP=<your-resource-group>
export CLUSTER_NAME=<your-cluster-name>

Next, use the Azure CLI to get the credentials for your cluster. This command automatically merges the cluster’s kubeconfig into your local ~/.kube/config file, allowing kubectl to interact with your AKS cluster.

az aks get-credentials --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME

Upon successful execution, you should see a message indicating that your kubeconfig has been updated. You can then verify your connection by running a simple kubectl get nodes command.

Command	Description
az aks get-credentials	Configures local kubectl to connect to a specific AKS cluster.
az aks show	Displays detailed information about an AKS cluster.
az aks nodepool list	Lists all node pools within an AKS cluster.
az aks upgrade	Upgrades the Kubernetes version of an AKS cluster.

Step-by-Step Resolution: Allowing Traffic on Port 10250¶

The primary solution for resolving TCP 10250 I/O timeouts is to ensure that network traffic to this port is explicitly allowed by the relevant Network Security Groups (NSGs). This typically involves modifying the NSG associated with the subnet where your AKS worker nodes are deployed.

Identifying the Affected NSG¶

The first step is to locate the NSG that is applied to your AKS worker nodes. AKS clusters create a separate resource group, often named MC_<resource-group-name>_<cluster-name>_<region>, which contains the cluster’s infrastructure resources, including the VM Scale Sets (VMSS) for your nodes.

You can find the NSG by inspecting the subnet your nodes are part of.

1. Retrieve Node Resource Group:
   

   az aks show --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME --query nodeResourceGroup -o tsv
   

   
   This will output the name of the managed resource group for your nodes (e.g., MC_myrg_myaks_eastus).

2. Find the Virtual Network and Subnet:
   You can then query the VM Scale Set within this node resource group to find its associated virtual network and subnet.
   

   NODE_RESOURCE_GROUP=$(az aks show --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME --query nodeResourceGroup -o tsv)
   VMSS_NAME=$(az vmss list --resource-group $NODE_RESOURCE_GROUP --query "[0].name" -o tsv) # Assumes one VMSS, adjust if multiple
   
   az vmss show --resource-group $NODE_RESOURCE_GROUP --name $VMSS_NAME --query "virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].ipConfigurations[0].subnet.id" -o tsv
   

   
   This command will provide the full resource ID of the subnet. From this ID, you can extract the VNet name and subnet name.

3. Identify the NSG associated with the Subnet:
   Once you have the VNet and subnet names, you can list the NSGs associated with that subnet:
   

   az network vnet subnet show --resource-group <VNet-Resource-Group> --vnet-name <VNet-Name> --name <Subnet-Name> --query networkSecurityGroup.id -o tsv
   

   
   This will give you the NSG ID. If no NSG is directly associated with the subnet, then NSGs might be attached to individual network interfaces of the VMs within the VMSS, which is less common for AKS managed nodes but still possible.

Modifying NSG Rules¶

Once you have identified the correct NSG, you need to add an inbound security rule to allow TCP traffic on port 10250. This rule should permit traffic from the Kubernetes API server to your worker nodes. For most AKS configurations, this means allowing traffic from the VirtualNetwork service tag (which includes the AKS API server’s private IP within the VNet) to the worker node subnet.

Here’s an example using the Azure CLI:

# Define your NSG details
NSG_NAME="<your-nsg-name>" # e.g., aks-agentpool-xxxx-nsg
NSG_RESOURCE_GROUP="<your-node-resource-group>" # e.g., MC_myrg_myaks_eastus

az network nsg rule create \
    --resource-group $NSG_RESOURCE_GROUP \
    --nsg-name $NSG_NAME \
    --name AllowKubelet10250 \
    --priority 100 \
    --direction Inbound \
    --access Allow \
    --protocol Tcp \
    --source-address-prefixes VirtualNetwork \
    --source-port-ranges '*' \
    --destination-address-prefixes '*' \
    --destination-port-ranges 10250 \
    --description "Allow Kubelet API traffic from AKS API Server to nodes"

Key considerations for this rule:

• --priority: Choose a priority number that is lower (higher precedence) than any existing Deny rules that might conflict. Priorities are evaluated from lowest to highest.
• --direction Inbound: The traffic is coming into the worker nodes.
• --source-address-prefixes VirtualNetwork: This is a safe choice, allowing traffic from any resource within the same virtual network, which includes the AKS control plane components. For private clusters, this is essential. For public clusters, the API server’s public IP range might need to be considered if it’s not handled by internal routing.
• --destination-port-ranges 10250: Specifically targets the Kubelet API port.
• --description: Add a clear description for future reference.

Visually, the network flow before and after the rule modification can be represented as follows:

Before NSG Rule Modification:

mermaid graph TD A[kubectl Client] --> B[Azure API Server] B -- Kubernetes API --> C{AKS Subnet NSG} C -- X[Blocked Port 10250] --> D[Kubelet on Worker Node] D -- Logs --> B

After NSG Rule Modification:

mermaid graph TD A[kubectl Client] --> B[Azure API Server] B -- Kubernetes API --> C{AKS Subnet NSG} C -- Port 10250 Allowed --> D[Kubelet on Worker Node] D -- Logs --> B

Verification Steps¶

After applying the NSG rule, it’s crucial to verify that the issue is resolved.

1. Re-run kubectl logs:
   Attempt to retrieve logs from a pod on an affected node:
   

   kubectl logs <pod-name>
   

   
   If the rule was correctly applied, you should now see the logs streaming from the container.

2. Check NSG Rules:
   Confirm that your new rule is present and correctly configured in the NSG:
   

   az network nsg rule list --resource-group $NSG_RESOURCE_GROUP --nsg-name $NSG_NAME --output table
   

   
   Look for the AllowKubelet10250 rule and verify its properties.

3. Test kubectl exec:
   As an additional verification, try executing a command inside a pod:
   

   kubectl exec -it <pod-name> -- ls /
   

   
   This command should now execute successfully, listing the root directory of your container.

Best Practices for AKS Networking¶

Proper network configuration is paramount for the stability and security of your AKS clusters. Adhering to best practices can help prevent future connectivity issues and ensure a robust environment.

• Principle of Least Privilege: Always apply the most restrictive network security rules necessary. Instead of opening wide port ranges or IP prefixes, target specific sources and destinations for essential services like Kubelet.
• Dedicated Node Resource Group NSG: Avoid making manual changes to the NSGs within the AKS-managed node resource group (MC_...) directly, unless absolutely necessary and documented. AKS manages these, and manual changes can sometimes be overwritten. If an NSG is associated with the subnet hosting your nodes, manage that subnet-level NSG.
• Leverage Azure Monitor and Network Watcher: Use Azure Monitor for logs and metrics to detect anomalies proactively. Network Watcher tools like IP Flow Verify and NSG Flow Logs can help diagnose network connectivity issues in real-time and trace which NSG rule is blocking traffic.
• Understand AKS Network Models: Familiarize yourself with Azure CNI and Kubenet networking models in AKS. Each has different implications for IP addressing, routing, and thus, NSG configurations.
• Document Network Changes: Maintain thorough documentation of all custom NSG rules, their purpose, and their impact on cluster communication. This is vital for troubleshooting and auditing.
• Regular Security Audits: Periodically review your NSG configurations to ensure they align with security policies and operational requirements, removing any outdated or overly permissive rules.

Conclusion¶

Resolving TCP 10250 I/O timeouts in Azure AKS is a common troubleshooting scenario that almost always points to an issue with network security rules, specifically Network Security Groups (NSGs). By understanding the role of the Kubelet, the flow of kubectl logs requests, and the function of NSGs in Azure, you can systematically diagnose and rectify these connectivity problems. Ensuring the correct inbound traffic rules are in place for port 10250 is crucial for maintaining the health and debuggability of your AKS cluster.

We hope this detailed guide helps you effectively resolve TCP 10250 I/O timeouts in your AKS environments. Share your insights, alternative solutions, or any challenges you faced during this troubleshooting process in the comments below! Your experiences can greatly benefit the wider community.