Fixing Azure Kubernetes Fleet Manager: A Troubleshooting Guide

Table of Contents

Welcome to the comprehensive troubleshooting guide for Azure Kubernetes Fleet Manager (Fleet). This document serves as a central resource to assist you in identifying, diagnosing, and resolving issues that may arise while you are utilizing Azure Kubernetes Fleet Manager to manage your collection of Kubernetes clusters.

Managing distributed systems like multiple Kubernetes clusters can introduce complexities. Understanding the common failure points and the appropriate diagnostic steps is crucial for maintaining a healthy and functional fleet. This guide aims to provide the necessary information to navigate these challenges effectively.

Understanding Azure Kubernetes Fleet Manager¶

Before diving into troubleshooting, it’s helpful to have a foundational understanding of what Azure Kubernetes Fleet Manager does. Fleet is designed to provide a unified control plane and management experience for multiple Azure Kubernetes Service (AKS) clusters. Key functionalities typically include:

• Cluster Registration and Joining: Adding existing AKS clusters to a Fleet resource.
• Resource Propagation: Synchronizing Kubernetes resources (like namespaces, deployments, configmaps) from the Fleet control plane to member clusters.
• Update Orchestration: Managing and orchestrating updates across member clusters in a controlled manner.
• Policy Enforcement: Applying policies uniformly across the fleet.

Issues can occur at any of these stages or in the underlying infrastructure dependencies.

General Troubleshooting Flow¶

When you encounter an issue with Azure Kubernetes Fleet Manager, following a structured approach can help pinpoint the root cause more quickly. Consider these general steps:

1. Identify the Scope: Is the issue affecting a single member cluster, multiple clusters, or the Fleet resource itself? Is it related to a specific operation (e.g., joining a cluster, applying a resource)?
2. Check Status and Logs: Examine the status of the Fleet resource and affected member clusters in the Azure portal or via Azure CLI. Check logs from relevant components, including the Fleet agent deployed on member clusters and any related Azure Activity Logs.
3. Verify Permissions: Ensure the identity used by Fleet (typically a managed identity) has the necessary permissions on the member clusters and any other Azure resources it interacts with.
4. Review Configuration: Double-check the configuration of the Fleet resource, member cluster join parameters, and any resources being propagated.
5. Check Network Connectivity: Verify that the Fleet control plane can communicate with the member clusters and vice-versa, depending on the specific operation.
6. Consult Documentation and Community Resources: Refer to official Azure documentation, troubleshooting guides (like this one!), and community forums or support channels.

Let’s explore common troubleshooting scenarios in more detail.

Troubleshooting Cluster Joining Issues¶

One of the first hurdles you might face is successfully adding an AKS cluster to your Fleet. Issues during the joining process can stem from various factors.

Common Symptoms¶

• The member cluster status in the Fleet resource remains “Joining” or transitions to a failed state.
• Errors reported during the az fleet member create or similar commands.
• The Fleet agent pods do not start or crash in the member cluster.

Diagnostic Steps¶

1. Verify Member Cluster Prerequisites: Ensure the AKS cluster you are trying to join meets the requirements for Fleet membership (e.g., supported Kubernetes version, network configuration).
2. Check Azure Permissions: The identity performing the join operation and the Fleet’s managed identity need specific permissions. The joining identity requires permissions to create roles and role assignments on the member cluster. The Fleet managed identity requires permissions (typically Microsoft.ContainerService/managedClusters/read and permissions to create RoleAssignment on the cluster) to interact with the member cluster’s Kubernetes API.
   • Use Azure CLI commands like az role assignment list to check assignments.
   • Ensure the Fleet identity has Reader access at minimum, and potentially contributor or custom roles depending on future operations like resource propagation.
3. Examine Fleet Agent Deployment: Once a join operation is initiated, Fleet attempts to deploy an agent (often referred to as the Fleet agent or related components) onto the member cluster.
   • Connect to the member cluster using kubectl.
   • Check the status of pods in the system namespace where the Fleet agent is deployed (e.g., kubectl get pods -n kube-system or the namespace specified by Fleet).
   • Look for pods stuck in Pending, Init:Error, CrashLoopBackOff, or Error states.
   • View the logs of these pods: kubectl logs <pod-name> -n <namespace>. The logs might indicate connectivity issues, permission errors, or configuration problems.
4. Check Network Connectivity: The member cluster needs to be able to reach the Fleet control plane endpoint in Azure. This communication typically happens over the public internet or potentially via private links if configured.
   • From within a pod in the member cluster (if possible, e.g., by exec’ing into a running pod or deploying a temporary diagnostic pod), attempt to reach the Azure control plane endpoints.
   • Check any Network Security Groups (NSGs), firewalls, or other network policies that might be blocking outbound connections from the member cluster to Azure services.
5. Review Activity Logs: Check the Azure Activity Log for the Fleet resource or the resource group containing the member cluster. Look for failed operations related to creating member resources or assigning roles.

Potential Fixes¶

• Grant the necessary Azure RBAC permissions to the identities involved.
• Ensure the member cluster’s network configuration allows outbound connections to Azure endpoints.
• Investigate specific errors found in the Fleet agent pod logs (e.g., if it’s a pull image error, check container registry access; if it’s an API error, double-check permissions).
• If a transient error occurred, try initiating the join operation again.

Troubleshooting Resource Propagation Issues¶

One of Fleet’s core functions is propagating Kubernetes resources defined in the Fleet resource to member clusters. Failures here mean your desired state isn’t reaching your clusters.

Common Symptoms¶

• Resources (e.g., Deployments, Namespaces, ConfigMaps) defined within the Fleet manifest are not created or updated on member clusters.
• The status of the propagated resource in the Fleet resource shows an error or pending state for one or more member clusters.
• Errors related to resource application appear in Fleet agent logs.

Diagnostic Steps¶

1. Inspect the Fleet Resource Manifest: Double-check the YAML manifest defined within your Fleet resource for the resources you intend to propagate. Ensure it is valid Kubernetes YAML and refers to the correct resources.
2. Check Member Cluster Connectivity: The Fleet control plane (via its agent on the member cluster) needs to be able to apply these resources to the member cluster’s Kubernetes API server. Revisit the network connectivity checks from the joining phase.
3. Verify Fleet Identity Permissions on Member Clusters: The Fleet managed identity must have sufficient Kubernetes RBAC permissions within the member cluster to create, update, or delete the specific types of resources you are propagating in the target namespaces.
   • For example, to propagate a Deployment in the default namespace, the Fleet identity needs permissions to manage deployments in the default namespace on the member cluster.
   • Use kubectl auth can-i <verb> <resource-type> -n <namespace> --as=<fleet-managed-identity-object-id> (you might need to impersonate the identity or check its role bindings directly).
4. Examine Fleet Agent Logs: Just like with joining, the Fleet agent on the member cluster is responsible for applying the resources. Check its logs for errors related to apply, create, or update operations. These logs will often contain specific Kubernetes API error messages (e.g., “forbidden,” “resource quota exceeded,” “validation error”).
5. Check Member Cluster Resource Quotas: If you are propagating resources that consume cluster resources (like Deployments, StatefulSets, PersistentVolumeClaims), check if resource quotas on the member cluster or in the target namespace are preventing their creation.
6. Look for Conflicts: Ensure the resources being propagated don’t conflict with existing resources on the member cluster (unless intentional, e.g., patching).

Potential Fixes¶

• Grant the necessary Kubernetes RBAC permissions to the Fleet managed identity on the member clusters.
• Correct any errors in the resource YAML manifest in the Fleet resource.
• Adjust resource quotas on member clusters if they are being hit.
• Resolve conflicts with existing resources.
• Ensure network connectivity between the Fleet agent and the member cluster’s API server.

Troubleshooting Update Orchestration Issues¶

Fleet can orchestrate rolling updates across your member clusters. Issues can occur if the update process gets stuck or fails on one or more clusters.

Common Symptoms¶

• Cluster updates initiated via Fleet do not start or complete.
• An update rollout plan gets stuck at a specific stage or on a particular cluster.
• Member clusters fail to update to the target version.

Diagnostic Steps¶

1. Check Update Run Status: Monitor the status of the update run resource in Fleet. It should show which clusters are pending, updating, or have failed.
2. Examine Individual Member Cluster Status: If a specific cluster is failing, check its status and logs directly in AKS. Is the cluster itself reporting any issues preventing the update?
3. Review Fleet Agent Logs: The Fleet agent might log issues if it’s failing to coordinate the update request with the member cluster’s AKS control plane.
4. Check AKS Update Prerequisites: Ensure the member cluster meets AKS’s own prerequisites for the update (e.g., sufficient resources for control plane upgrade, no pending operations).
5. Review Azure Activity Logs: Check the Activity Log for the specific AKS cluster that is failing to update. Look for errors reported by the AKS update operation itself.

Potential Fixes¶

• Resolve any underlying issues reported by AKS on the failing member cluster.
• Ensure the Fleet identity has permissions to trigger AKS updates on member clusters.
• Check for network issues that might prevent communication between Fleet and the AKS control plane during the update process.
• If the rollout is stuck, you might need to manually intervene on the failing cluster or potentially cancel and restart the Fleet update run.

Troubleshooting Connectivity and Networking¶

Fleet relies on communication between the Fleet control plane (managed service) and the Fleet agents running in member clusters, as well as between the Fleet agent and the member cluster’s Kubernetes API server. Networking issues can manifest in many ways.

Common Symptoms¶

• Cluster joining fails with timeout errors.
• Resource propagation is slow or fails intermittently.
• Update orchestration fails to start or monitor progress.
• Fleet UI or CLI commands show stale information for member clusters.

Diagnostic Steps¶

1. Verify Outbound Connectivity from Member Clusters: The Fleet agent needs to communicate with the Azure control plane. This typically involves outbound HTTPS connections.
   • Identify the required endpoints for Azure Fleet and related services (refer to official Azure documentation for the most current list).
   • From a diagnostic pod within the member cluster’s network, attempt to reach these endpoints using tools like curl or telnet/nc.
   • Check NSGs, firewalls (Azure Firewall, network virtual appliances), and any on-premises firewalls if using ExpressRoute or VPN.
2. Verify Connectivity within the Member Cluster: The Fleet agent pod needs to communicate with the member cluster’s Kubernetes API server (typically on port 443).
   • From the Fleet agent pod itself (if possible), attempt to reach the cluster’s internal API endpoint.
   • Check Network Policies within the member cluster that might be blocking communication between namespaces or pods.
3. Check DNS Resolution: Ensure the member cluster’s network can resolve Azure service endpoints and the cluster’s internal API endpoint correctly.
4. Review Fleet Agent Logs for Network Errors: Look for specific error messages indicating connection refused, timeouts, or unreachable hosts.

Potential Fixes¶

• Update firewall rules and NSGs to allow necessary outbound traffic from member clusters to Azure endpoints.
• Review and adjust Network Policies within member clusters if they are blocking the Fleet agent’s access to the API server.
• Ensure DNS is configured correctly within the VNet or on-premises network.

Troubleshooting Permission Issues¶

Incorrect or insufficient permissions are a frequent cause of failure in Azure services, including Fleet. Both Azure RBAC and Kubernetes RBAC are relevant.

Common Symptoms¶

• Operations fail with “AuthorizationFailed” or “Forbidden” errors in Azure Activity Logs or API responses.
• Fleet agent logs show “403 Forbidden” or similar errors when trying to interact with the Kubernetes API.
• Specific operations (like joining, resource propagation, updating) fail while others might succeed.

Diagnostic Steps¶

1. Identify the Identity: Determine which identity is performing the failing action. Is it your user identity running an Azure CLI command? Is it the Fleet managed identity attempting an action on a member cluster?
2. Check Azure RBAC Assignments:
   • For actions failing at the Azure level (e.g., creating the Fleet resource, adding a member), check the Azure RBAC role assignments for your user identity on the resource group or subscription.
   • For actions Fleet performs on member clusters (e.g., initiating AKS update, assigning Kubernetes roles), check the Azure RBAC assignments for the Fleet managed identity on the member cluster resource.
3. Check Kubernetes RBAC Assignments:
   • For actions the Fleet agent performs within the member cluster (e.g., creating deployments, namespaces), check the Kubernetes RBAC RoleBindings or ClusterRoleBindings that grant permissions to the Fleet managed identity within the member cluster’s API.
   • You’ll need to map the Azure managed identity’s object ID to a Kubernetes ServiceAccount, User, or Group (depending on how AKS RBAC integration is configured and how Fleet utilizes it) and check the bindings for that principal.
4. Least Privilege Principle: Ensure the identity has enough permissions, but also be mindful of granting excessive permissions, especially for the Fleet managed identity on member clusters.

Potential Fixes¶

• Grant the required Azure RBAC roles (e.g., Contributor, Reader, specific custom roles) to the relevant identities on the appropriate scope (subscription, resource group, member cluster resource).
• Create or update Kubernetes RoleBindings or ClusterRoleBindings within the member clusters to grant the Fleet identity the necessary permissions to manage resources.
• Use az role assignment create for Azure RBAC and kubectl apply -f <rbac-yaml> for Kubernetes RBAC.

Utilizing Logs and Metrics¶

Logs and metrics are your eyes into the system. Knowing where to look and what to look for is critical.

• Azure Activity Log: Provides a history of Azure resource operations. Filter by the Fleet resource, member cluster resources, or relevant resource groups to see failed operations and associated error details.
• Azure Monitor Logs (Log Analytics): Configure AKS to send control plane logs and kube-system pod logs (which include the Fleet agent logs) to a Log Analytics workspace. This allows for powerful querying of pod logs, events, and other cluster diagnostics. Look for logs from pods associated with the Fleet agent or Fleet system components.
• Fleet Resource Status: The status fields of the Fleet resource and its member resources in Azure provide a high-level view of health and ongoing operations.
• Kubernetes Events: Within the member clusters, check Kubernetes events (kubectl get events -A) for relevant information related to pod scheduling, image pulling, resource creation/update failures, or controller errors in the namespace where the Fleet agent runs or where propagated resources should appear.

Consider setting up alerts in Azure Monitor based on key error signals from these logs to proactively identify issues.

Table of Common Errors and Causes¶

Symptom/Error Message	Potential Cause	Diagnostic Steps
Cluster join status stuck/failed	Insufficient Azure RBAC permissions for identity	Check az role assignment list for joining identity and Fleet identity.
Cluster join status stuck/failed	Network connectivity issues	Verify outbound connectivity from member cluster to Azure endpoints. Check NSGs.
Fleet agent pods crashing/error	Image pull failure	Check container registry access from member cluster nodes. Check pod logs.
Fleet agent pods crashing/error	Insufficient resources on member cluster node	Check node resource utilization. Check pod requests/limits. View pod events.
Resource not propagating	Insufficient Kubernetes RBAC for Fleet identity	Check RoleBindings/ClusterRoleBindings for Fleet identity on member cluster.
Resource not propagating	Malformed resource YAML in Fleet manifest	Validate the YAML syntax and content.
Resource propagation failed	Resource Quota exceeded on member cluster	Check ResourceQuota objects in the target namespace on the member cluster.
Update run stuck/failed	Underlying AKS cluster update issue	Check AKS cluster status and Activity Log for update errors.
API call forbidden (403)	Insufficient permissions (Azure or K8s RBAC)	Determine which identity made the call and check its permissions.
Timeout connecting to endpoint	Network firewall or routing issue	Trace network path, check firewall rules (NSG, Azure Firewall, etc.).

Simple Troubleshooting Flowchart (Conceptual)¶

mermaid graph TD A[Issue Detected] --> B{Is it impacting<br>one or multiple clusters?}; B -- One --> C[Focus on specific<br>member cluster]; B -- Multiple --> D[Focus on Fleet resource<br>and common factors]; C --> E[Check Member Cluster Status<br>and K8s Events]; D --> F[Check Fleet Resource Status<br>and Azure Activity Logs]; E --> G{Is Fleet Agent running<br>on member?}; F --> H{Is issue related to<br>join, sync, or update?}; G -- No --> I[Troubleshoot Agent Deployment<br>- Permissions, Network, Resources]; G -- Yes --> J[Check Fleet Agent Pod Logs]; H -- Join --> K[Troubleshoot Join Process<br>- Azure/K8s Permissions, Network]; H -- Sync --> L[Troubleshoot Resource Propagation<br>- K8s Permissions, Manifest, Quotas]; H -- Update --> M[Troubleshoot Update Orchestration<br>- AKS Update Status, Fleet Permissions]; I --> N[Implement Fix]; J --> N; K --> N; L --> N; M --> N; N --> O[Verify Resolution];

This diagram illustrates a basic flow, starting with identifying the scope and leading to specific checks based on the issue type and affected components.

Conclusion¶

Troubleshooting Azure Kubernetes Fleet Manager involves understanding the interactions between the Fleet service, the Fleet agent, member AKS clusters, and underlying Azure infrastructure. By systematically checking status, logs, permissions, and network connectivity, you can effectively diagnose and resolve most issues.

This guide provides a starting point, covering common scenarios. Always refer to the most current official Azure documentation for detailed information and specific error messages.

Did this guide help you resolve your Fleet issue? Do you have a troubleshooting tip that isn’t covered here? Share your experiences and questions in the comments below!