Resolve SubnetIsFull Errors During AKS Cluster Upgrades: A Troubleshooting Guide

Table of Contents

Managing Azure Kubernetes Service (AKS) clusters involves regular maintenance, including crucial upgrades to ensure security, stability, and access to the latest features. However, these upgrade operations can sometimes encounter unexpected hurdles. One such persistent issue that administrators might face is the “SubnetIsFull” error, which effectively halts the upgrade process and can lead to operational disruptions. This comprehensive guide aims to dissect this error, provide a deeper understanding of its root causes, and offer detailed, actionable solutions to navigate and resolve it efficiently.

When an AKS cluster upgrade fails due to a lack of available IP addresses, you will typically encounter an error message similar to the following:

Failed to scale node pool <AGENT POOL NAME>’ in Kubernetes service ‘<NAME>’. Error: VMSSAgentPoolReconciler retry failed: Code=’SubnetIsFull’ Message=’<SUBNET NAME> with address prefix <PREFIX> doesn’t have enough capacity for IP addresses.’ Details=[]

This message clearly indicates that the underlying subnet assigned to your node pool has run out of available IP addresses, preventing the creation of new nodes essential for the upgrade process. Understanding each component of this error message is the first step toward effective troubleshooting and resolution. The VMSSAgentPoolReconciler is the internal AKS component responsible for managing the Virtual Machine Scale Set (VMSS) that backs your node pools. Its failure to retry indicates a persistent underlying issue. The Code='SubnetIsFull' explicitly states the nature of the problem, while <SUBNET NAME> and <PREFIX> pinpoint the exact network resource at fault, making it easier to identify the specific subnet that needs attention.

Prerequisites for AKS Management and Troubleshooting¶

Before embarking on troubleshooting or applying solutions, it is imperative to ensure that your environment is properly configured with the necessary tools and permissions. Effective management of AKS clusters heavily relies on a robust set of utilities and a fundamental understanding of Azure networking. The following prerequisites are essential for a smooth troubleshooting experience:

• Azure CLI Version 2.0.65 or Later: This is the primary command-line interface for interacting with Azure resources, including AKS. To verify your installed version, execute az --version. If your version is outdated or you need to install it, refer to the official Azure documentation on “How to install the Azure CLI.” Newer versions often include crucial bug fixes and features that enhance management capabilities for AKS.
• Active Azure Subscription: You must have an active Azure subscription with sufficient permissions to manage AKS clusters, virtual networks, and subnets. Typically, roles like “Contributor” or “Owner” on the resource group containing the AKS cluster and its associated virtual network are required.
• Understanding of Azure Networking Concepts: A foundational knowledge of Azure virtual networks (VNets), subnets, IP addressing, Network Security Groups (NSGs), and User Defined Routes (UDRs) is critical. This understanding will enable you to grasp the implications of IP address exhaustion and the strategies involved in subnet resizing or creation.
• Familiarity with AKS Upgrade Process: For more detailed information about the AKS cluster upgrade process, including the mechanisms AKS uses to perform upgrades, it is highly recommended to review the “Upgrade an AKS cluster” section in the official documentation on “Upgrade an Azure Kubernetes Service (AKS) cluster.” This knowledge will provide context for why additional IP addresses are needed during an upgrade.
• Access to Cluster Logs and Metrics: While not strictly a prerequisite for starting troubleshooting, having access to AKS diagnostic logs, Azure Monitor, and Container Insights can provide valuable insights into cluster health and resource utilization, which might indirectly contribute to understanding IP address demands.

Ensuring these prerequisites are met will streamline the troubleshooting process and allow you to execute the proposed solutions effectively and confidently.

Symptoms of SubnetIsFull Errors¶

The most direct and undeniable symptom of a “SubnetIsFull” error is the failure of an AKS cluster upgrade operation. When you initiate an upgrade command, whether through the Azure portal, Azure CLI, or an automated pipeline, the process will eventually terminate with the aforementioned error message. However, there are other, more subtle symptoms and indicators that might precede this explicit error or offer additional context:

• Stuck or Prolonged Upgrade Process: An upgrade operation might appear to hang indefinitely or take an unusually long time before ultimately failing. This could indicate the cluster is attempting to provision new nodes but is continuously failing due to IP address limitations.
• Node Provisioning Failures: Even outside of a full cluster upgrade, attempts to scale up a node pool (adding more nodes) might also fail with the same “SubnetIsFull” error. This is a clear signal that the subnet is at its capacity limit.
• Reduced Cluster Performance: While not directly caused by IP exhaustion, a cluster struggling with upgrades or scaling issues might exhibit reduced performance due to an inability to scale out and handle increased workloads. This might push administrators to attempt upgrades or scaling, only to encounter the IP issue.
• Azure Portal Notifications and Activity Logs: The Azure portal will display failure notifications for the AKS upgrade operation. Detailed information, including the “SubnetIsFull” error code and message, will be logged in the activity logs for the AKS resource, providing an audit trail of the failure.
• Azure Monitor Alerts (if configured): If you have configured Azure Monitor alerts for subnet IP utilization, you might receive warnings or critical alerts before an upgrade attempt, indicating that your subnet is approaching its capacity. Proactive monitoring is a powerful tool to prevent such errors.

Recognizing these symptoms early can help in diagnosing the problem quickly and taking corrective action before it severely impacts your cluster’s operations.

Cause: Insufficient IP Addresses During Cluster Operations¶

The “SubnetIsFull” error fundamentally stems from a shortage of available IP addresses within the subnet assigned to your AKS node pools. This scarcity becomes particularly critical during operations that require the creation of new nodes, such as cluster upgrades or scaling events. Understanding the underlying mechanisms of AKS operations and IP address allocation is key to comprehending this cause.

The AKS Upgrade Mechanism: Surge Upgrades¶

AKS employs a “surge” upgrade mechanism to ensure high availability during cluster upgrades. Instead of performing an in-place upgrade of existing nodes, AKS provisions new nodes with the updated Kubernetes version alongside the old nodes. Once these new nodes are successfully brought online and are ready to host workloads, the existing (old) nodes are drained and then decommissioned. This “surge” model typically requires at least one extra node to be created before any old nodes are removed, effectively demanding additional IP addresses from the subnet during the upgrade window. If the subnet lacks the capacity for these temporary surge nodes, the upgrade will inevitably fail.

IP Address Planning and Consumption in AKS¶

The number of required IP addresses in an AKS cluster is not static and depends on several factors, including the chosen network plugin, the number of nodes, and the number of pods per node.

• Azure CNI (Container Network Interface): With Azure CNI, each node and each pod receive a distinct IP address from the subnet of the node pool. This is the most common cause of “SubnetIsFull” errors because pod IPs are directly consumed from the VNet subnet.
  • Node IPs: Each VM in the node pool requires one IP address.
  • Pod IPs: Each pod running on a node also requires an IP address. The maximum number of pods per node (configured via maxPods) directly influences the total IP demand. For instance, if a node can host 30 pods, it will consume 31 IPs (1 for the node, 30 for pods) from the subnet.
• Kubenet: With Kubenet, nodes receive an IP from the VNet subnet, but pods receive IPs from a logically separate, non-routable address space defined within the node itself. This means Kubenet consumes significantly fewer VNet IPs (primarily just for the nodes) and is less prone to “SubnetIsFull” errors related to pod IP exhaustion, though node IP exhaustion can still occur.

Illustrative Table: IP Address Consumption Comparison

Feature	Azure CNI	Kubenet
Node IP Source	VNet Subnet	VNet Subnet
Pod IP Source	VNet Subnet	Internal to Node (not from VNet Subnet)
IPs per Node	1 (for node) + maxPods (for pods)	1 (for node)
VNet IP Demand	High (scales with nodes and pods)	Low (scales primarily with nodes)
Routing	Direct Pod-to-Pod within VNet	NAT/Routing through Node
Use Case	Large clusters, direct VNet integration	Smaller clusters, simpler networking
Subnet Capacity	Critical for long-term growth and upgrades	Less critical for pod IPs, but still for node IPs

Calculation and Planning for IP Addresses¶

When planning your AKS cluster, it’s crucial to estimate the total number of IP addresses required, factoring in future growth and upgrade operations. A common formula for Azure CNI to estimate required IPs for a node pool is:

Total IPs Required = (Number of nodes + Number of surge nodes) * (maxPods per node + 1)

• Number of nodes: Your current desired node count.
• Number of surge nodes: Typically 1 to 2, depending on the maxSurge setting for your node pool (often a percentage or fixed number of nodes during upgrade).
• maxPods per node: The maximum number of pods that can run on a single node (default is 30, but configurable).
• +1: For the node’s own IP address.

If the IP address range configured for your subnet only supports a fixed number of nodes and their pods, and this number is exceeded during an upgrade (which temporarily requires more IPs), the operation will fail. This highlights the critical importance of careful IP address planning during the initial design phase of an AKS cluster and continuous monitoring of subnet utilization as the cluster grows.

Solution: Resolving SubnetIsFull Errors¶

Addressing the “SubnetIsFull” error requires either freeing up IP addresses or expanding the available IP space. The approach you choose will depend on the urgency, the current cluster state, and the available virtual network (VNet) CIDR space. Here are the primary solutions:

Option 1: Reduce Cluster Nodes to Reserve IP Addresses¶

This is often the quickest, though potentially disruptive, solution. By scaling down one or more node pools, you release their associated IP addresses back into the subnet, making them available for the upgrade’s surge nodes.

Steps:

1. Identify Node Pools: Determine which node pool is experiencing the issue and which ones can be temporarily scaled down without severe impact to your applications.
2. Assess Workload Impact: Before scaling down, ensure that your cluster has enough capacity to handle its current workloads on fewer nodes. Critical services might need to be temporarily scaled down or moved.
3. Scale Down Node Pools: Use the Azure CLI to reduce the node count of an affected or non-critical node pool.
   

   az aks nodepool scale \
       --resource-group <resource-group-name> \
       --cluster-name <aks-cluster-name> \
       --name <node-pool-name> \
       --node-count <reduced-count>
   

   
   For example, if your node pool has 5 nodes and you need 1 IP for surge, you might scale down to 4. Monitor the cluster to ensure workloads rebalance correctly.
4. Perform Cluster Upgrade: Once the IPs are freed, attempt the AKS cluster upgrade again.
   

   az aks upgrade \
       --resource-group <resource-group-name> \
       --name <aks-cluster-name> \
       --kubernetes-version <new-kubernetes-version>
   

5. Scale Up Node Pools (Optional): After the upgrade is successful, you can scale your node pools back up to their original size if needed.

Considerations: This method is effective for immediate relief but might cause temporary service degradation if not managed carefully. It’s best suited for clusters with ample spare capacity or during maintenance windows.

Option 2: Add a New Node Pool with a Unique and Larger Subnet¶

If scaling down isn’t an option due to high demand or if the virtual network’s CIDR has sufficient available space for a new, larger subnet, adding a new node pool with its own dedicated subnet is a robust long-term solution. This effectively provides a fresh, ample pool of IP addresses for new nodes.

This approach involves a multi-step migration process:

Step 1: Add a New User Node Pool in the Virtual Network on a Larger Subnet¶

This is the most critical step, where you provision new network space.

1. Create a New Subnet:
   • Ensure your existing VNet has a sufficiently large, unallocated CIDR block available.
   • Create a new subnet within your VNet with a generously sized IP range (e.g., /24, /23, or even /22, depending on your projected growth and maxPods settings). A /24 subnet provides 251 usable IPs, while a /23 offers 507.
   • Azure Portal: Navigate to your Virtual Network -> Subnets -> + Subnet.
   • Azure CLI:
     

     az network vnet subnet create \
         --resource-group <resource-group-name> \
         --vnet-name <vnet-name> \
         --name <new-subnet-name> \
         --address-prefixes <new-subnet-cidr> \
         --service-endpoints Microsoft.ContainerRegistry Microsoft.Storage # Add other required service endpoints
     

2. Add a New User Node Pool to AKS:
   • Create a user node pool associated with the newly created, larger subnet. Specify the subnet’s resource ID.
   • Azure CLI:
     

     az aks nodepool add \
         --resource-group <resource-group-name> \
         --cluster-name <aks-cluster-name> \
         --name <new-user-node-pool-name> \
         --node-count <initial-node-count> \
         --vnet-subnet-id /subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.Network/virtualNetworks/<vnet-name>/subnets/<new-subnet-name> \
         --mode User \
         --enable-cluster-autoscaler # Optional: If you use autoscaler
     

   • Start with a small initial-node-count (e.g., 1 or 2) to verify setup before scaling up.

Step 2: Switch the Original Node Pool to a System Node Pool Type¶

System node pools are designed to host critical system pods (like kube-proxy, coredns). By converting the original, IP-constrained node pool to a system pool, you ensure that these essential services have a dedicated home and prevent user workloads from consuming its limited IP space.

az aks nodepool update \
    --resource-group <resource-group-name> \
    --cluster-name <aks-cluster-name> \
    --name <original-node-pool-name> \
    --mode System

Step 3: Scale Up the New User Node Pool and Migrate Workloads¶

Now that you have a new node pool with ample IP space, you can gradually scale it up to accommodate your application workloads.

1. Scale Up New User Node Pool:
   

   az aks nodepool scale \
       --resource-group <resource-group-name> \
       --cluster-name <aks-cluster-name> \
       --name <new-user-node-pool-name> \
       --node-count <desired-count-for-workloads>
   

   
   Increase the node-count to match or exceed the capacity of your original node pool, allowing for seamless migration of pods.
2. Drain Pods from Original Node Pool (Optional but Recommended): To ensure a controlled migration, you can explicitly drain pods from the original node pool. This causes Kubernetes to reschedule those pods onto the new, larger node pool.
   

   kubectl drain <node-name-from-original-pool> --ignore-daemonsets --delete-emptydir-data
   

   
   Repeat for all nodes in the original pool. You can get node names with kubectl get nodes.
3. Verify Workload Migration: Monitor your applications and pods to ensure they have successfully moved to the new node pool and are running as expected. Use kubectl get pods -o wide to see which nodes your pods are running on.

Step 4: Scale Down or Remove the Original Node Pool¶

Once all user workloads have been safely migrated to the new user node pool and the original node pool is now solely hosting system pods (or is completely empty of user pods), you can safely scale it down.

1. Scale Down Original Node Pool:
   

   az aks nodepool scale \
       --resource-group <resource-group-name> \
       --cluster-name <aks-cluster-name> \
       --name <original-node-pool-name> \
       --node-count <minimum-system-nodes>
   

   
   Scale it down to the minimum number of nodes required for system pods (e.g., 1 or 2, depending on redundancy requirements).
2. Perform Cluster Upgrade: With ample IP space now available in the new user node pool and reduced demand from the original, you can now proceed with the AKS cluster upgrade.
3. Remove Original Node Pool (Optional): If you no longer need the original node pool even for system pods (e.g., if your new pool can also host system pods or you’ve created a new system pool), you can remove it entirely. Exercise caution: Ensure all critical pods have been migrated or are resilient to node removal.
   

   az aks nodepool delete \
       --resource-group <resource-group-name> \
       --cluster-name <aks-cluster-name> \
       --name <original-node-pool-name>
   

Proactive Measures and Best Practices¶

Preventing “SubnetIsFull” errors is always better than reacting to them. Implement these best practices:

• Generous IP Planning from the Start: Always allocate larger subnet CIDR blocks than initially required, accounting for growth in nodes, pods, and future surge upgrades. Err on the side of caution with subnet sizing.
• Monitor Subnet IP Utilization: Regularly monitor the IP address utilization of your AKS subnets using Azure Monitor. Set up alerts to notify you when utilization crosses a certain threshold (e.g., 70-80%).
• Understand maxPods Impact: Be aware that changing the maxPods setting on a node pool significantly impacts IP consumption. Carefully consider this setting.
• Use Multiple Node Pools: Distribute your workloads across multiple node pools, each potentially in its own subnet, to segment IP demands and improve isolation.
• Private AKS Clusters: If using private AKS, be mindful of the additional IP requirements for the private link service and other control plane components, though these typically use a separate dedicated subnet.
• Regular Review of Network Topology: Periodically review your VNet and subnet configurations, especially before major cluster changes or when anticipating significant growth.

More Information and Resources¶

For a deeper understanding of AKS networking and related issues, consider the following areas:

Understanding Network Topology with a Mermaid Diagram¶

Visualizing the network architecture can help in understanding IP distribution. Below is a simplified Mermaid diagram illustrating how new node pools with dedicated subnets fit into an existing VNet.

mermaid graph TD subgraph Azure Subscription subgraph Resource Group: myAKSResourceGroup subgraph Virtual Network: myVNet (/16) subgraph Subnet: original-subnet (/24) A[Original AKS Node Pool: System Mode] end subgraph Subnet: new-larger-subnet (/22) B[New AKS Node Pool: User Mode] end end C[AKS Control Plane] A -- uses --> C B -- uses --> C end end
This diagram illustrates how original-subnet now hosts the Original AKS Node Pool in System Mode, while a new, larger new-larger-subnet hosts the New AKS Node Pool in User Mode, both residing within myVNet and communicating with the AKS Control Plane.

AKS Networking Deep Dive¶

For those interested in exploring AKS networking in more detail, including discussions around Azure CNI vs. Kubenet, IP address planning, and advanced configurations, the following video provides an excellent visual and auditory explanation. While this video isn’t from the original article, it’s highly relevant for gaining a deeper understanding that can prevent “SubnetIsFull” errors.

YouTube Video: Azure AKS Networking Explained (Hypothetical)

<iframe width="560" height="315" src="https://www.youtube.com/embed/dQw4w9WgXcQ" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

Note: The YouTube video link above is a placeholder and links to Rick Astley’s “Never Gonna Give You Up.” In a real-world scenario, you would replace this with an actual, relevant educational video on AKS networking.

Related Error Codes and Documentation¶

• InsufficientSubnetSize error code: This error is closely related and often indicates a similar underlying issue of insufficient IP addresses. You might encounter this during initial cluster creation if the subnet is too small, rather than during an upgrade.
• Azure Kubernetes Service (AKS) documentation: The official Microsoft Azure documentation is an invaluable resource for all aspects of AKS, including networking, upgrades, and troubleshooting. Always refer to the latest documentation for up-to-date information and best practices.

By understanding the causes and implementing these detailed solutions and proactive measures, you can effectively resolve “SubnetIsFull” errors and ensure the smooth operation and upgrading of your Azure Kubernetes Service clusters.

We encourage you to share your experiences and insights in the comments section below. Have you encountered this error? What strategies did you find most effective in resolving it? Your contributions can help the broader AKS community navigate these challenges.