Azure Subnet Full? Troubleshoot & Resolve the SubnetIsFull Error Code

Table of Contents

Microsoft Azure Kubernetes Service (AKS) is a powerful container orchestration platform that simplifies the deployment and management of containerized applications. A fundamental aspect of any network design, including AKS deployments, is the allocation of IP addresses within virtual networks and subnets. When scaling your AKS cluster, the system requires additional IP addresses for the new nodes and potentially for the pods running on those nodes, depending on the networking configuration. Running out of available IP addresses within the subnet assigned to an AKS node pool is a common issue that prevents scaling operations and can manifest as the SubnetIsFull error.

This error specifically indicates that the subnet has exhausted its IP address capacity and cannot accommodate the required number of new IP addresses needed for the scaling operation. Understanding the cause and having a clear resolution path is crucial for maintaining the scalability and availability of your AKS workloads. This article details the symptoms, underlying cause, and a step-by-step solution to address the SubnetIsFull error, along with best practices to prevent it in the future.

Prerequisites¶

To effectively troubleshoot and resolve issues within your Azure environment, having the necessary tools readily available is essential. The primary tool for interacting with Azure resources via the command line is the Azure CLI.

You will need the Azure CLI installed on your local machine or accessible via the Azure Cloud Shell. Ensure you are using version 2.0.59 or a later version. This version and subsequent updates contain the necessary commands and features to manage AKS clusters and their associated network resources. Having the latest version helps ensure compatibility and access to the newest features and bug fixes related to AKS and networking management.

Symptoms¶

The most direct symptom of a full subnet preventing AKS scaling is the specific error message returned when attempting to increase the node count of a node pool using the Azure portal, Azure CLI, or other automation tools. This error is explicit about the problem.

You will encounter an error message similar to the following output:

{
  "code": "SubnetIsFull",
  "message": "Subnet <subnet-name> with address prefix <subnet-prefix> does not have enough capacity for <new-ip-count> IP addresses."
}

This message clearly identifies the code as SubnetIsFull, indicating the nature of the problem. The message provides crucial details: the name of the affected subnet (<subnet-name>), its current address range (<subnet-prefix>, e.g., 10.0.0.0/24), and the exact number of additional IP addresses (<new-ip-count>) that were requested but could not be allocated. Seeing this error confirms that the scaling operation failed specifically because the designated subnet has no more available IP addresses.

Cause¶

The root cause of the SubnetIsFull error is the inability to allocate new IP addresses from the subnet allocated to the AKS node pool. When you scale out an AKS cluster by adding more nodes, each new virtual machine (VM) node requires at least one IP address from the subnet. Additionally, depending on the network configuration chosen for your AKS cluster, pods running on these nodes may also require IP addresses from the same subnet or a related subnet.

AKS supports different networking models, primarily Kubenet and Azure Container Networking Interface (CNI). The IP address allocation strategy differs significantly between these two models, which impacts how quickly a subnet might become full.

Understanding AKS Networking Models and IP Allocation¶

Kubenet: In the Kubenet networking plugin, Azure provisions VMs into the subnet, and the VMs receive IP addresses from the VNet subnet. Pods on the nodes receive IP addresses from an internally managed IP address space, which is NAT’d (Network Address Translated) to the node’s IP address for external network communication. This model requires fewer IP addresses from the VNet subnet for the nodes themselves, as pod IPs are handled separately within the cluster. However, it introduces routing complexity and can have performance limitations for direct pod-to-pod communication across nodes. While less prone to the subnet full for pod IPs issue, it can still hit the limit for node IPs.

Azure Container Networking Interface (CNI): With Azure CNI, every pod receives an IP address directly from the VNet subnet assigned to the node pool. This allows pods to be directly addressable from the VNet and peered networks, which simplifies routing and improves performance. However, this model consumes IP addresses from the VNet subnet at a much higher rate. For each new node added, AKS typically pre-allocates a batch of IP addresses from the subnet for the pods that will run on that node. The number of IPs per node is configurable, but a default exists. When scaling out with Azure CNI, you need enough available IPs for the new nodes plus the pre-allocated IPs for pods on those new nodes.

Regardless of the model, scaling out requires IP allocation. When the total count of allocated IPs (for nodes and potentially pods) plus the requested new IPs exceeds the total capacity of the subnet (minus Azure’s reserved addresses), the SubnetIsFull error occurs.

Azure reserves the first four and last one IP address within every subnet for internal use (network address, default gateway, Azure DNS usage, network broadcast address). This means a /24 subnet, which mathematically contains 256 IP addresses (2^8), actually only has 251 usable IPs for your resources like AKS nodes and pods. This reservation must be factored into capacity planning.

Solution¶

Resolving the SubnetIsFull error requires migrating your AKS cluster or, more specifically, your node pools to a subnet with a larger address space. Directly expanding the CIDR range of an existing subnet that contains resources like AKS node pools is not currently supported in Azure. Therefore, the recommended approach involves creating a new, larger subnet and migrating your workloads to a new node pool deployed in that subnet.

This migration process ensures minimal disruption to your running applications by leveraging Kubernetes rolling updates and draining capabilities. It’s a multi-step process that requires careful planning and execution.

Here are the steps to migrate your workloads to a new node pool within a larger subnet:

1. Create a New Subnet with a Larger CIDR: The first step is to define and create a new subnet within the same virtual network as your existing AKS cluster. This new subnet must have a CIDR address range significantly larger than the current one to accommodate future scaling needs. Determining the appropriate size involves calculating the maximum expected number of nodes and pods (if using Azure CNI) and factoring in Azure’s reserved IPs and a buffer for future growth. Referencing documentation on planning IP addressing for AKS is crucial here to size the subnet correctly based on your anticipated maximum cluster size and chosen networking model. For example, if you anticipate needing up to 250 nodes in a CNI cluster, and each node requires 30 pod IPs (a common configuration), you’d need IPs for nodes + (nodes * pod IPs). Even with a separate pod subnet, you still need node IPs. A /24 (251 usable IPs) is clearly insufficient for 250 nodes. A /23 (507 usable IPs) might fit the nodes but not pods. A /22 (1019 usable IPs) or larger would be more appropriate depending on the detailed calculation.

2. Create a New Node Pool on the New Subnet: Once the new, larger subnet is ready, create a new node pool and deploy it into this subnet. You can add multiple node pools to an existing AKS cluster, allowing you to gradually shift workloads. Use the Azure CLI command az aks nodepool add and specify the --vnet-subnet-id parameter, providing the resource ID of the newly created subnet. This command adds a new set of nodes to your cluster, but these nodes will draw IP addresses from the larger subnet. Configure this new node pool with the desired VM size, node count (start with a small number initially), and other settings identical or similar to your old node pool. Adding this new pool gives your cluster nodes in the new IP space.

3. Migrate Workloads by Draining Old Nodes: With nodes available in the new node pool (and thus in the larger subnet), you can begin migrating your application workloads. The safest way to do this is by “draining” the nodes in the original, full subnet. Draining a node cordons it (marks it as unschedulable) and then gracefully terminates all pods running on it. Kubernetes will automatically reschedule these pods onto other available nodes, including the new nodes in the larger subnet. This process allows for a smooth transition of your applications without downtime, provided your applications are designed for graceful shutdown and your Pod Disruption Budgets (PDBs) are configured correctly. You can use kubectl drain <node-name> for individual nodes or script this process for all nodes in the old node pool. Monitor your applications closely during this phase to ensure they reschedule and start correctly on the new nodes.

4. Delete the Original Node Pool: After successfully migrating all workloads from the old node pool and confirming that your applications are running stably on the new node pool, you can safely delete the original node pool. Use the Azure CLI command az aks nodepool delete specifying the name of the node pool that was using the full subnet. Deleting the node pool removes the nodes and releases the IP addresses they were using back into the old subnet. This step cleans up your cluster resources and removes the node pool associated with the problematic subnet.

This four-step process effectively moves your cluster’s compute capacity to a network segment with ample IP address space, resolving the immediate SubnetIsFull error and providing room for future scaling.

Calculating Required IP Addresses for AKS Subnets (Azure CNI)¶

A crucial part of the solution is correctly sizing the new subnet. For Azure CNI, the calculation needs to account for both node IPs and pod IPs.

The formula is approximately:

Total Required IPs = (Expected Maximum Node Count * (Number of Pods per Node + 1)) + Buffer for Rolling Upgrades + Azure Reserved IPs

• Expected Maximum Node Count: The absolute largest number of nodes you ever anticipate needing in this node pool.
• Number of Pods per Node: This is configured during node pool creation (or defaults are used). It determines how many IPs are reserved per node for pods.
• + 1: This accounts for the IP address needed by the node VM itself.
• Buffer for Rolling Upgrades: AKS often requires extra IPs during node upgrades or scaling events (e.g., creating new nodes before deleting old ones). A common recommendation is to add enough capacity for at least one additional node or a percentage of the max nodes.
• Azure Reserved IPs: Always add 5 IPs for Azure’s internal use within the subnet.

Let’s simplify the calculation for CNI based on typical node pool settings. When you create an AKS node pool using Azure CNI without specifying a separate pod subnet, AKS assigns a block of IP addresses from the node pool subnet to each node for its pods. The number of IPs per node depends on the VM size, with a default maximum (e.g., 30 for many common sizes).

If Max Nodes is your expected peak node count, and Max Pods per Node is the configured or default setting:

Usable IPs Needed >= (Max Nodes * Max Pods per Node) + Max Nodes + Buffer + Azure Reserved

A simpler heuristic often used for initial sizing for Azure CNI with default pod density is to estimate total required IPs based on the maximum number of pods you expect to run concurrently plus the maximum number of nodes. Since each pod and each node needs an IP. However, the per-node pre-allocation strategy by AKS CNI is key.

A common approach is to size the subnet based on Max Nodes * (Max Pods per Node + 1). For example, if you expect a maximum of 100 nodes, and default pod density allows 30 pods per node: 100 * (30 + 1) = 3100 required IPs. Add buffer and 5 reserved IPs. A /21 subnet provides 2043 usable IPs. A /20 provides 4091 usable IPs. A /19 provides 8187 usable IPs. You would need at least a /20 in this example.

Using a separate subnet for pods (--pod-subnet-id) with Azure CNI changes the calculation; the node pool subnet only needs IPs for the nodes themselves (Max Nodes + Buffer + 5). The pod subnet needs capacity for all pods (Max Nodes * Max Pods per Node + Buffer + 5).

CIDR Block Sizes and Usable IPs¶

Understanding CIDR notation is vital for subnet sizing. Here’s a quick reference for common block sizes and their usable IP counts (after subtracting Azure’s 5 reserved IPs):

CIDR Block	Total IPs	Usable IPs (in Azure)
/29	8	3
/28	16	11
/27	32	27
/26	64	59
/25	128	123
/24	256	251
/23	512	507
/22	1024	1019
/21	2048	2043
/20	4096	4091
/19	8192	8187
/18	16384	16379
/17	32768	32763
/16	65536	65531

As you can see, increasing the CIDR prefix length by 1 (e.g., from /24 to /23) doubles the number of available IP addresses. Choosing a subnet that is sufficiently large initially, even if it seems excessive, is far easier and less disruptive than performing this migration process later.

Conceptual Diagram: Azure CNI IP Allocation¶

```mermaid
graph TD
A[Azure Virtual Network] → B(Subnet );
B → C1(AKS Node 1);
B → C2(AKS Node 2);
B → C3(AKS Node N);
C1 → P1_1(Pod 1);
C1 → P1_2(Pod 2);
C1 → P1_X(Pod X);
C2 → P2_1(Pod 1);
C2 → P2_Y(Pod Y);
C3 → P3_1(Pod 1);
C3 → P3_Z(Pod Z);

C1 -- IP from Subnet --> B;
C2 -- IP from Subnet --> B;
C3 -- IP from Subnet --> B;

P1_1 -- IP from Subnet --> B;
P1_2 -- IP from Subnet --> B;
P1_X -- IP from Subnet --> B;
P2_1 -- IP from Subnet --> B;
P2_Y -- IP from Subnet --> B;
P3_1 -- IP from Subnet --> B;
P3_Z -- IP from Subnet --> B;

style B fill:#f9f,stroke:#333,stroke-width:2px
style C1 fill:#ccf,stroke:#333,stroke-width:1px
style C2 fill:#ccf,stroke:#333,stroke-width:1px
style C3 fill:#ccf,stroke:#333,stroke-width:1px
style P1_1 fill:#eef,stroke:#999
style P1_2 fill:#eef,stroke:#999
style P1_X fill:#eef,stroke:#999
style P2_1 fill:#eef,stroke:#999
style P2_Y fill:#eef,stroke:#999
style P3_1 fill:#eef,stroke:#999
style P3_Z fill:#eef,stroke:#999

linkStyle 0 stroke:#666;
linkStyle 1 stroke:#666;
linkStyle 2 stroke:#666;
linkStyle 3 stroke:#666;

linkStyle 4 stroke:#00a,stroke-dasharray: 5 5;
linkStyle 5 stroke:#00a,stroke-dasharray: 5 5;
linkStyle 6 stroke:#00a,stroke-dasharray: 5 5;
linkStyle 7 stroke:#00a,stroke-dasharray: 5 5;
linkStyle 8 stroke:#00a,stroke-dasharray: 5 5;
linkStyle 9 stroke:#00a,stroke-dasharray: 5 5;
linkStyle 10 stroke:#00a,stroke-dasharray: 5 5;

subgraph "AKS Cluster"
    C1
    C2
    C3
    P1_1
    P1_2
    P1_X
    P2_1
    P2_Y
    P3_1
    P3_Z
end

classDef primaryNode fill:#ccf,stroke:#333,stroke-width:1px;
classDef pod fill:#eef,stroke:#999;
class C1,C2,C3 primaryNode;
class P1_1,P1_2,P1_X,P2_1,P2_Y,P3_1,P3_Z pod;

```
This diagram illustrates how, in the Azure CNI model without a separate pod subnet, both AKS nodes and the pods running on them consume IP addresses directly from the designated subnet.

Best Practices¶

Preventing the SubnetIsFull error is far more desirable than resolving it after it occurs. By following best practices related to subnet sizing, IP address management, and strategic use of node pools, you can significantly reduce the likelihood of encountering this issue.

Here are some key recommendations:

• Plan for Future Growth: Always provision subnets with future scaling in mind. Estimate the maximum number of nodes you anticipate needing for your cluster’s lifetime (or at least the next few years). Use the IP calculation methods discussed earlier, specific to your chosen networking model (Azure CNI is most sensitive to this), to determine the required subnet size. It’s better to allocate a slightly larger block than immediately necessary than to face the disruption of migration later. Reserving more IP addresses upfront is a prudent strategy to avoid hitting capacity limits unexpectedly.
• Use Larger Subnet CIDR Ranges: As demonstrated by the CIDR table, moving to a larger block size dramatically increases available IPs. When initially designing your VNet and subnets for AKS, choose a CIDR that provides ample IP space, perhaps a /22 or even /21 as a starting point for production clusters, depending on anticipated scale. While using very large blocks might seem wasteful, it buys you significant operational flexibility and avoids future network topology changes.
• Monitor IP Usage: Implement proactive monitoring of IP address usage within your AKS subnets. Tools like Azure Monitor and Log Analytics can track the number of allocated IP addresses. Setting up alerts based on IP usage thresholds (e.g., warning at 70% usage, critical at 90%) allows you to identify potential capacity issues well in advance of them becoming critical, giving you time to plan and implement a solution like adding a new node pool in a larger subnet before scaling is blocked.
• Optimize IP Allocation: While primarily applicable to Azure CNI, review the configured maximum pods per node setting if you customize it. Ensure it aligns with your application density needs. Avoid excessively high pod-per-node limits if not strictly necessary, as this consumes more IP space per node addition. For existing clusters approaching limits, explore if any resources are unnecessarily consuming IPs and can be optimized.
• Utilize Multiple Node Pools Strategically: Leverage the ability to create multiple node pools within your AKS cluster. You can associate different node pools with different subnets. This allows you to distribute IP address consumption across different network segments. For example, you might have a ‘system’ node pool in one subnet and ‘user’ or ‘application-specific’ node pools in other, potentially larger, subnets. This provides better isolation and allows for more granular IP planning per workload type. Multiple node pools also enable using different VM sizes or operating systems within the same cluster. If using Azure CNI, consider using the --pod-subnet-id parameter to place pod IPs in a separate subnet from node IPs, which simplifies node pool subnet sizing (only needs node IPs) and dedicates a large subnet solely for pod IPs.
• Factor in Rolling Upgrades and Scaling Events: Remember that scaling operations and cluster/node pool upgrades often temporarily require additional IPs. For example, during a rolling upgrade, new nodes are brought online before old ones are decommissioned, temporarily increasing the node count. Ensure your subnet size accounts for this temporary peak demand beyond your steady-state maximum node count.

By implementing these best practices, you can build a more resilient and scalable AKS infrastructure that is less susceptible to IP exhaustion issues.

Impact of SubnetIsFull on Cluster Operations¶

Beyond just preventing successful scaling out, hitting the SubnetIsFull error can have cascading negative impacts on your AKS cluster’s operational health and stability:

• Inability to Scale Out: As the error message directly states, you cannot add more nodes to the affected node pool. This prevents you from responding to increased application load by adding capacity.
• Upgrade Failures: Node pool upgrades often involve a rolling update strategy where new nodes are created before old ones are removed. If the subnet is full, the new nodes cannot be provisioned, causing the upgrade process to fail or stall. This can leave your cluster on an outdated version or in a partially upgraded state.
• Node Auto-Repair/Replacement Issues: If a node becomes unhealthy and AKS attempts to replace it automatically, this process involves provisioning a new node. A full subnet will prevent this replacement, potentially leading to a reduced cluster capacity or leaving unhealthy nodes active if replacement fails.
• Rescheduling Problems: While existing pods can typically be rescheduled onto existing healthy nodes if a node fails or is drained, if the cluster is already at its maximum capacity due to the subnet limit and nodes are lost, pods might not be able to reschedule if there isn’t sufficient IP space for new nodes to replace the failed ones.
• Deployment Issues: While existing deployments might run, deploying new applications or scaling up existing deployments that require new pods might indirectly be affected if the cluster cannot scale to accommodate the new demand, leading to resource starvation or scheduling failures.

Addressing the SubnetIsFull error promptly is therefore not just about scaling, but about maintaining the overall health, resilience, and ability to perform maintenance operations on your AKS cluster.

Automating Migration Steps¶

While the migration process involves several steps, it can be automated using scripting languages (like Bash or PowerShell) or infrastructure-as-code tools (like Terraform or Bicep).

• Scripting: You can chain together Azure CLI commands (az network vnet subnet create, az aks nodepool add, kubectl drain, az aks nodepool delete) into a script. This allows for repeatable execution and reduces the risk of manual errors. You would need to handle monitoring the draining process and confirming workload migration within the script.
• Infrastructure as Code (IaC): Tools like Terraform or Bicep can declare the desired state of your infrastructure. You can define the new subnet and the new node pool in your IaC code. Applying the changes would create these resources. Managing the workload migration (draining old nodes) might still require separate kubectl commands or potentially using Kubernetes-aware IaC providers, but the infrastructure provisioning part is automated. This approach is excellent for managing infrastructure changes and ensuring consistency.

Automating this process is highly recommended for production environments to minimize manual effort and potential downtime during the migration.

Conclusion¶

The SubnetIsFull error in Azure Kubernetes Service is a clear indicator that your subnet has run out of IP address capacity, preventing scaling and potentially impacting cluster operations. While resizing an existing subnet with active resources is not possible, the recommended solution involves creating a new subnet with a larger address space, adding a new node pool to that subnet, migrating your workloads gracefully, and finally deleting the old node pool.

Planning your IP address space generously from the outset, based on anticipated maximum scale and the AKS networking model (especially Azure CNI), is the most effective preventative measure. Implementing proactive monitoring of subnet IP usage allows you to identify potential issues before they block critical operations. By understanding the cause, knowing the resolution steps, and following best practices, you can ensure your AKS cluster remains scalable, resilient, and performs optimally.

What are your experiences dealing with the SubnetIsFull error in Azure AKS? Have you used automation to manage the migration process? Share your thoughts and strategies in the comments below!