Resolve Azure UpgradeFailed Errors: Fixing PDB-Caused Eviction Failures

Table of Contents

Upgrading an Azure Kubernetes Service (AKS) cluster is a critical operation for maintaining security, accessing new features, and ensuring compatibility with the latest Kubernetes ecosystem developments. However, this process can sometimes fail, leaving the cluster in an unhealthy state. One common cause for upgrade failures stems from issues during node drainage, specifically when Pod Disruption Budgets (PDBs) prevent the successful eviction of pods. This article delves into identifying and resolving these UpgradeFailed errors caused by PDB-related eviction failures. Understanding the interplay between AKS upgrades, node draining, and PDBs is essential for smooth cluster operations.

AKS upgrades involve cordoning and draining nodes within a node pool to safely move workloads before replacing or updating the underlying virtual machines. During the draining process, Kubernetes attempts to gracefully evict pods from the node being drained. Pod Disruption Budgets are a Kubernetes API object that application owners can define to limit the number of concurrent disruptions their applications can tolerate. These disruptions can be voluntary (like node drains for upgrades) or involuntary (like node failures). While crucial for maintaining application availability during disruptions, overly restrictive PDBs can inadvertently block voluntary disruptions like those required during a cluster upgrade, leading to failures. When a PDB prevents a pod from being evicted, the node drain process stalls, and eventually, the entire upgrade operation can time out and fail, resulting in the UpgradeFailed status.

Prerequisites¶

To effectively troubleshoot and resolve the UpgradeFailed errors discussed in this article, you need specific tools and access. The primary tool required is the Azure Command-Line Interface (Azure CLI). Ensure you have Azure CLI version 2.67.0 or a later version installed on your machine. This version contains necessary commands and potentially bug fixes relevant to AKS operations and troubleshooting.

You can verify your installed version by running the command az --version in your terminal. If your version is older, you will need to install or upgrade it. Additionally, you need appropriate permissions within Azure to manage AKS clusters and within the Kubernetes cluster itself to interact with pods, deployments, statefulsets, and Pod Disruption Budgets. Familiarity with basic Kubernetes concepts and kubectl commands is also assumed, as you will be inspecting cluster resources. Access to the cluster’s kubeconfig file is essential for using kubectl.

Symptoms¶

When an AKS cluster upgrade fails due to restrictive Pod Disruption Budgets, the error messages typically point to a failure during the node draining phase, specifically mentioning an inability to evict a pod. The error often includes phrases like “Drain node… failed when evicting pod… failed with Too Many Requests error.” This “Too Many Requests” often refers to the Kubernetes API rate limiting that can occur when drain operations repeatedly fail due to PDBs. The error message will explicitly state that the failure is “often caused by a restrictive Pod Disruption Budget (PDB) policy” and might provide specific debugging information about the pod, its namespace, the blocking PDB, and the current number of “unready” pods.

You might encounter error messages similar to these examples:

(UpgradeFailed) Drain node aks-<nodepool-name>-xxxxxxxx-vmssxxxxxx failed when evicting pod <pod-name> failed with Too Many Requests error. This is often caused by a restrictive Pod Disruption Budget (PDB) policy. Original error: Cannot evict pod as it would violate the pod's disruption budget.. PDB debug info: <namespace>/<pod-name> blocked by pdb <pdb-name> with 0 unready pods.

Or a variation with a more explicit error code structure:

Code: UpgradeFailed
Message: Drain node aks-<nodepool-name>-xxxxxxxx-vmssxxxxxx failed when evicting pod <pod-name> failed with Too Many Requests error. This is often caused by a restrictive Pod Disruption Budget (PDB) policy. Original error: Cannot evict pod as it would violate the pod's disruption budget.. PDB debug info: <namespace>/<pod-name> blocked by pdb <pdb-name> with 0 unready pods.

The key indicators within these messages are “UpgradeFailed,” “Drain node failed,” “evicting pod failed,” “Too Many Requests,” “restrictive Pod Disruption Budget (PDB) policy,” and “Cannot evict pod as it would violate the pod’s disruption budget.” If you see these elements, a PDB is the most likely culprit. The cluster or the specific node pool might enter a Failed provisioning state in the Azure portal or via Azure CLI commands.

Cause¶

The root cause of this specific UpgradeFailed error lies in the configuration of a Pod Disruption Budget (PDB) that is too restrictive to allow pods to be evicted during a voluntary disruption event like a node drain for an upgrade. A PDB aims to ensure a minimum number of replicas for an application remain running during disruptions. It does this by defining either minAvailable (minimum number of replicas that must be available) or maxUnavailable (maximum number of replicas that can be unavailable) for pods matching a specific label selector.

During a node drain, the Kubernetes control plane attempts to evict pods one by one. Before evicting a pod, it checks if the eviction request would violate any PDBs that apply to that pod. If evicting the pod would cause the number of available replicas for the application to drop below the minAvailable threshold or exceed the maxUnavailable threshold specified in the PDB, the eviction is blocked. The drain process cannot proceed until the PDB constraint is satisfied. If the PDB is configured such that no disruptions are allowed (e.g., minAvailable is equal to the total number of replicas, or maxUnavailable is 0 for an application with only one replica), and the pods needing to be drained are the ones necessary to meet the PDB’s requirements, the drain will be perpetually blocked.

To diagnose this, you can inspect the PDBs in your cluster using kubectl. The output of kubectl get pdb is particularly insightful. Look at the ALLOWED DISRUPTIONS column for PDBs associated with applications running on the nodes that are failing to drain. If ALLOWED DISRUPTIONS is 0, it means the PDB currently allows no further voluntary disruptions for the pods it protects. This configuration is often appropriate for critical singleton pods or stateful applications requiring strict availability, but it directly conflicts with the node draining process during an upgrade.

Consider the following example output from kubectl get pdb:

$ kubectl get pdb
NAME        MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
nginx-pdb   2               N/A               0                     24s

In this example, the nginx-pdb requires MIN AVAILABLE replicas to be 2. If the associated deployment (e.g., an Nginx deployment with 2 replicas) has both its pods running on the node being drained, and no other pods can take over immediately, the ALLOWED DISRUPTIONS will show 0. Evicting either of the two pods would drop the available count to 1, violating the PDB’s requirement of 2 available.

Another way to confirm a PDB-related drain failure is by examining Kubernetes events. Events provide a timeline of activities within the cluster. You can filter events related to node draining using kubectl get events | grep -i drain. Look for warning messages associated with the node that failed to drain. Messages like “Eviction blocked by Too Many Requests (usually a pdb): ” clearly indicate that a PDB is preventing the necessary pod evictions.

Here is an example output demonstrating these warning events:

$ kubectl get events | grep -i drain
LAST SEEN   TYPE      REASON                    OBJECT                                   MESSAGE
(... many other events ...)
32m         Normal    Drain                     node/aks-<nodepool-name>-xxxxxxxx-vmssxxxxxx   Draining node: aks-<nodepool-name>-xxxxxxxx-vmssxxxxxx
2m57s       Warning   Drain                     node/aks-<nodepool-name>-xxxxxxxx-vmssxxxxxx   Eviction blocked by Too Many Requests (usually a pdb): <pod-name-1>
12m         Warning   Drain                     node/aks-<nodepool-name>-xxxxxxxx-vmssxxxxxx   Eviction blocked by Too Many Requests (usually a pdb): <pod-name-2>
32m         Warning   Drain                     node/aks-<nodepool-name>-xxxxxxxx-vmssxxxxxx   Eviction blocked by Too Many Requests (usually a pdb): <pod-name-3>

Each warning entry signifies that a specific pod on the target node could not be evicted because a PDB blocked the operation. If multiple pods on the node are protected by restrictive PDBs, you will see multiple such warning events. This continuous blocking prevents the node from ever becoming empty and ready for removal or update, leading to the upgrade failure. Resolving this requires modifying the PDB configuration or temporarily bypassing the PDB’s constraints.

Solution 1: Enable Pods to Drain¶

The most recommended solution is to adjust the Pod Disruption Budget (PDB) configuration to allow for at least one disruption, thereby enabling pods to be drained during the upgrade process. This approach respects the intent of the PDB (protecting availability) while making it compatible with necessary maintenance operations like upgrades. The ALLOWED DISRUPTIONS value, as shown in the kubectl get pdb output, should be 1 or greater for the PDBs affecting pods on the nodes being upgraded.

You can influence the ALLOWED DISRUPTIONS value by modifying the minAvailable or maxUnavailable fields in the PDB definition, or by increasing the number of replicas for the workload protected by the PDB. For example, if a PDB uses minAvailable: 2 and the deployment only has 2 replicas, scaling the deployment up to 3 replicas would likely change ALLOWED DISRUPTIONS to 1 (allowing one replica to be unavailable while still meeting the minimum of 2 available). Alternatively, changing the PDB to minAvailable: 1 or maxUnavailable: 1 (if applicable based on replica count) would also allow disruptions.

To modify a PDB, you can use kubectl edit pdb <pdb-name> -n <pdb-namespace>. This opens the PDB’s YAML definition in your default editor. Carefully adjust the minAvailable or maxUnavailable value. For instance, if your PDB has minAvailable: 100% or minAvailable: <replica count>, consider changing it to maxUnavailable: 1 or a percentage that allows at least one pod to be evicted at a time. After saving the changes, Kubernetes updates the PDB configuration immediately. Verify the change using kubectl get pdb <pdb-name> -n <pdb-namespace> and confirm ALLOWED DISRUPTIONS is now greater than 0.

Once the PDB is adjusted to allow disruptions, you can retry the AKS cluster upgrade. Since the cluster is in a Failed state, retrying the upgrade to the same version often triggers a reconciliation process that attempts to complete the previously failed operation. Use the Azure CLI command az aks upgrade --name <aksName> --resource-group <resourceGroupName>. The CLI will recognize the cluster’s state and prompt you to proceed with the upgrade to the existing target version to resolve the failure. Confirm the prompts to start the reconciliation.

$ az aks upgrade --name <aksName> --resource-group <resourceGroupName>
Are you sure you want to perform this operation? (y/N): y
Cluster currently in failed state. Proceeding with upgrade to existing version 1.28.3 to attempt resolution of failed cluster state.
Since control-plane-only argument is not specified, this will upgrade the control plane AND all nodepools to version . Continue? (y/N): y

The upgrade process should now be able to successfully drain the nodes because the PDB allows for the necessary pod evictions. Monitor the upgrade progress using az aks show --name <aksName> --resource-group <resourceGroupName> --query provisioningState or by observing node status and events within the cluster using kubectl.

It is a best practice to review your PDB strategies regularly to ensure they balance high availability during disruptions with the need for planned maintenance activities like cluster upgrades. Configuring PDBs with reasonable maxUnavailable values (like 1 or a small percentage) is often a good compromise for typical applications.

Solution 2: Back Up, Delete, and Redeploy the PDB¶

Another approach to overcome a restrictive PDB blocking an upgrade is to temporarily remove the PDB’s constraints. This involves backing up the current PDB configuration, deleting the PDB, performing the cluster upgrade, and then redeploying the PDB from the backup. Be aware that deleting a PDB temporarily removes the availability protection it provides. During the period the PDB is deleted, voluntary disruptions could potentially make more replicas of your application unavailable than you typically tolerate. Use this solution with caution and ideally during a maintenance window or when you are confident your application can handle disruptions during the upgrade without the PDB.

First, back up the YAML definition of the problematic PDB using kubectl get pdb <pdb-name> -n <pdb-namespace> -o yaml > pdb-name-backup.yaml. This command fetches the PDB configuration and saves it to a local file. Make sure the file is saved securely, as you will need it later to restore the PDB. Review the generated backup file to ensure it contains the complete and correct PDB definition.

Next, delete the PDB using kubectl delete pdb <pdb-name> -n <pdb-namespace>. Confirm that the PDB is deleted by running kubectl get pdb <pdb-name> -n <pdb-namespace>; it should no longer appear in the list or an error indicating it’s not found should be returned. With the PDB removed, there is no object blocking pod evictions related to this specific PDB.

Now, retry the AKS cluster upgrade using the Azure CLI command as described in Solution 1: az aks upgrade --name <aksName> --resource-group <resourceGroupName>. The upgrade process, including node draining, should now proceed without the PDB blocking evictions. Monitor the upgrade until it completes successfully.

Once the AKS cluster upgrade is finished and the cluster is in a Succeeded provisioning state, you can redeploy the PDB from your backup file. Use the command kubectl apply -f pdb-name-backup.yaml. This will recreate the PDB with its original configuration, restoring the availability protection for your application. Verify the PDB is recreated using kubectl get pdb <pdb-name> -n <pdb-namespace>.

This method is effective for resolving the immediate upgrade failure caused by the PDB. However, it’s crucial to understand the temporary loss of protection. If frequent upgrades or planned maintenance are common, consider whether modifying the PDB as described in Solution 1 might be a better long-term strategy than repeatedly deleting and recreating it.

Solution 3: Delete the Pods That Can’t Be Drained or Scale the Workload Down to Zero (0)¶

A more direct, though potentially disruptive, approach is to manually intervene with the pods or workloads being blocked by the PDB. If you can identify the specific pods preventing the drain (from the error messages or kubectl get events), you can attempt to delete them. Kubernetes will typically recreate these pods based on their controller (Deployment, StatefulSet, etc.), potentially on a different node that is not undergoing drainage.

However, simply deleting pods controlled by a Deployment or StatefulSet might not always be straightforward or effective if the PDB is still in place and restrictive. The controller will try to reschedule the pod, but the PDB might still prevent its creation on a node if it violates the availability constraints across the cluster. A more reliable variation of this solution is to temporarily scale down the entire workload (Deployment or StatefulSet) protected by the problematic PDB to zero replicas before attempting the upgrade reconciliation.

Important Note: Scaling a workload down to zero will cause temporary downtime or unavailability for that specific application. Ensure you understand the impact on your services before proceeding. It is highly recommended to back up the workload definition before scaling it down. Use the command kubectl get <deployment.apps or statefulset.apps> <name> -n <namespace> -o yaml > backup.yaml. Replace <deployment.apps or statefulset.apps>, <name>, and <namespace> with the actual resource type, name, and namespace of your workload.

To scale down the workload, use the command:
kubectl scale --replicas=0 <deployment.apps or statefulset.apps> <name> -n <namespace>

This command instructs the controller to terminate all running replicas of the specified workload. Once scaled down to zero, there are no pods for the PDB to protect, and thus no pods blocking the node drain.

With the problematic pods or workload scaled down, retry the AKS cluster upgrade using the Azure CLI command: az aks upgrade --name <aksName> --resource-group <resourceGroupName>. As before, this triggers a reconciliation attempt. The node drain should now proceed successfully as the pods that were previously blocked by the PDB are no longer running.

$ az aks upgrade --name <aksName> --resource-group <resourceGroupName>
Are you sure you want to perform this operation? (y/N): y
Cluster currently in failed state. Proceeding with upgrade to existing version 1.28.3 to attempt resolution of failed cluster state.
Since control-plane-only argument is not specified, this will upgrade the control plane AND all nodepools to version . Continue? (y/N): y

After the AKS cluster upgrade completes successfully, scale the workload back up to its desired replica count using the backup YAML file or by specifying the previous replica count with kubectl scale. If you used the backup file, use kubectl apply -f backup.yaml. If you prefer scaling directly, use kubectl scale --replicas=<previous-replica-count> <deployment.apps or statefulset.apps> <name> -n <namespace>.

This solution is effective but comes at the cost of temporary application downtime. It’s suitable for situations where the application can tolerate a brief outage or during scheduled maintenance windows. It highlights the importance of designing applications and PDBs with maintenance events in mind.

Additional Considerations¶

Understanding Pod Disruption Budgets is crucial for operating Kubernetes effectively, especially in managed services like AKS where node lifecycle events (like upgrades, scaling, or maintenance) are handled by the platform. PDBs are powerful tools for ensuring application availability, but misconfigurations can lead to operational roadblocks.

Here’s a summary of key PDB fields and their interaction:

Field	Description	How it affects Drains
selector	Defines which pods the PDB applies to (must match workload labels).	Determines which pods’ evictions are checked against this PDB.
minAvailable	Minimum number or percentage of replicas that must remain available.	If evicting a pod would drop available count below this, eviction is blocked.
maxUnavailable	Maximum number or percentage of replicas that can be unavailable at any time.	If evicting a pod would exceed the allowed unavailable count, eviction is blocked.
allowedDisruptions	Calculated field showing how many more disruptions are currently allowed.	Value of 0 means no more voluntary evictions are permitted by this PDB currently.

When designing PDBs, strive for a balance. A PDB that requires 100% availability (minAvailable: 100% or maxUnavailable: 0 for a single replica) effectively prevents any voluntary eviction, making upgrades or node maintenance difficult without manual intervention. For most applications, allowing a small number of pods to be unavailable (maxUnavailable: 1 or a small percentage like maxUnavailable: 10%) is sufficient to maintain high availability during rolling upgrades or maintenance events while still permitting node drains.

You can test the potential impact of your PDBs during hypothetical disruptions using the Kubernetes API directly, though this is complex. A simpler approach is to use tools that simulate drain scenarios or to test your PDB configurations in staging environments that mimic your production setup.

Implementing PDBs should be part of your application deployment pipeline. Defining sensible PDBs alongside Deployments and StatefulSets ensures that applications are designed with resilience and maintainability in mind from the start. Regularly auditing existing PDBs, especially before major cluster operations like upgrades, is a good practice to prevent unexpected failures.

If you encounter persistent issues with node draining that you suspect are PDB-related, and the above solutions do not fully resolve the problem, consider checking for other potential factors. These might include custom admission controllers, finalizers on pods that prevent deletion, or issues with the application itself that prevent graceful termination. However, the “Eviction blocked by PDB” message is a strong indicator pointing directly to the PDB configuration as the primary obstacle.

Have you encountered UpgradeFailed errors in your AKS clusters? How have you debugged and resolved issues related to Pod Disruption Budgets blocking node drains? Share your experiences and tips in the comments below!