Mastering ClusterResourcePlacement API: Troubleshooting Guide for Azure Kubernetes Fleet Manager

Table of Contents

Azure Kubernetes Fleet Manager simplifies the management of multiple Kubernetes clusters, enabling consistent configuration and resource propagation across a fleet of clusters. At the heart of this capability lies the ClusterResourcePlacement (CRP) API, a powerful tool for distributing Kubernetes resources from a central hub cluster to various member clusters. However, like any complex distributed system, issues can arise during resource placement. This guide provides a detailed approach to troubleshooting common ClusterResourcePlacement API object-related problems, ensuring your fleet operates smoothly.

To effectively diagnose and resolve errors within the Azure Kubernetes Fleet Manager environment, a fundamental understanding of several interconnected API objects is essential. These include ClusterResourceSnapshot, ClusterSchedulingPolicySnapshot, ClusterResourceBinding, and Work. Each of these objects plays a distinct role in the lifecycle of resource propagation, from defining the resources to be distributed to tracking their application on individual member clusters. Grasping their purpose and interdependencies is key to pinpointing the root cause of any ClusterResourcePlacement failures.

Understanding the ClusterResourcePlacement Lifecycle¶

The ClusterResourcePlacement custom resource orchestrates the entire process of distributing resources across your fleet. Its status fields provide critical insights into the current state of resource propagation, making it the primary starting point for any troubleshooting endeavor. You can inspect the detailed status of any ClusterResourcePlacement resource using the Kubernetes command-line tool.

To view the current status of your ClusterResourcePlacement custom resource, execute the following command:

kubectl describe clusterresourceplacement <name>

Replace <name> with the actual name of your ClusterResourcePlacement object. The output will provide a comprehensive overview of its conditions, events, and placement statuses across your member clusters. Analyzing this output carefully is often the quickest way to identify where the propagation process might be stalled or failing.

The journey of a resource through ClusterResourcePlacement involves a series of sequential conditions, each signifying a crucial stage in its deployment lifecycle. Understanding the complete progression of these conditions is paramount for efficient diagnosis and identification of failures. Each condition transitions from False to True as the resource moves through the propagation pipeline.

Here is the complete progression of a ClusterResourcePlacement deployment, along with initial troubleshooting directions if a condition is not met as expected:

1. ClusterResourcePlacementScheduled: This condition indicates that the Fleet scheduler has successfully evaluated the ClusterResourcePlacement policy and has selected a set of target member clusters where the resources will be placed.

   • Troubleshooting False status: If this condition is False, it suggests an issue with the scheduling phase. This could be due to an inability to find suitable clusters matching the specified placement.policy criteria, or there might be issues with the Fleet scheduler itself. Investigate the ClusterSchedulingPolicySnapshot and relevant Fleet controller logs for scheduler-related messages.

2. ClusterResourcePlacementRolloutStarted: This condition signifies that the rollout process for the defined resources has officially begun. It marks the transition from scheduling to the actual preparation and distribution of resources.

   • Troubleshooting False status: A False status here might indicate a problem after scheduling but before the actual resource processing starts. This could be an internal Fleet Manager error, or perhaps there are misconfigurations preventing the initiation of the rollout. Check the ClusterResourcePlacement object’s events for any error messages during this phase.

3. ClusterResourcePlacementOverridden: This condition tracks whether any resources specified in the ClusterResourcePlacement have been modified or overridden by policies defined in ResourceOverride objects. This allows for dynamic adjustments to resources before propagation.

   • Troubleshooting False status: If False when overrides are expected, it means the specified override policies were not applied or were not found. Verify the existence and correctness of your ResourceOverride custom resources. Ensure their selectors match the resources intended for modification.

4. ClusterResourcePlacementWorkSynchronized: This critical condition confirms that the Work objects, which encapsulate the resources to be applied, have been successfully synchronized with the selected member clusters. Work objects are the actual deployment units sent to individual member clusters.

   • Troubleshooting False status: A False status here often points to network connectivity issues between the hub and member clusters, or problems with the Fleet agent on the member cluster. Verify the member cluster’s connectivity to the hub and check the logs of the Fleet agent running on the member cluster for errors related to receiving or processing Work objects.

5. ClusterResourcePlacementApplied: This condition indicates that the resources contained within the Work objects have been successfully applied to the target member clusters. This is a crucial checkpoint, showing that the resources are now instantiated on the fleet members.

   • Troubleshooting False status: If this condition is False, it implies that the application of resources on one or more member clusters failed. This could be due to various reasons, such as insufficient permissions on the member cluster, resource conflicts, invalid resource manifests, or resource quotas being exceeded. Examine the placementStatuses section of the ClusterResourcePlacement status for specific error messages from the affected member clusters.

6. ClusterResourcePlacementAvailable: The final condition, ClusterResourcePlacementAvailable, signifies that the deployed resources are not only applied but also in a ready or available state on the member clusters. This might mean pods are running, services are accessible, or other defined readiness criteria are met.

   • Troubleshooting False status: A False status here suggests that while the resources might have been applied, they are not functioning as expected. This typically requires debugging the deployed resources directly on the member cluster, just as you would any other Kubernetes application. Check the status of the deployed pods, deployments, or services on the member cluster using kubectl get/describe commands.

Visualizing ClusterResourcePlacement Progression¶

To better visualize the flow, consider the following state diagram for ClusterResourcePlacement progression:

mermaid graph TD A[ClusterResourcePlacement Created] --> B{Scheduling Policy Evaluation?}; B -- True --> C(ClusterResourcePlacementScheduled); B -- False --> D[Troubleshoot Scheduling Policy/Scheduler]; C --> E{Rollout Initiated?}; E -- True --> F(ClusterResourcePlacementRolloutStarted); E -- False --> G[Troubleshoot Rollout Initiation]; F --> H{Resource Overrides Applied?}; H -- True --> I(ClusterResourcePlacementOverridden); H -- False --> J[Troubleshoot ResourceOverride]; I --> K{Work Objects Synchronized?}; K -- True --> L(ClusterResourcePlacementWorkSynchronized); K -- False --> M[Troubleshoot Network/Fleet Agent]; L --> N{Resources Applied on Member Clusters?}; N -- True --> O(ClusterResourcePlacementApplied); N -- False --> P[Troubleshoot Resource Application Errors]; O --> Q{Resources Available?}; Q -- True --> R(ClusterResourcePlacementAvailable); Q -- False --> S[Troubleshoot Deployed Resource Status];

Frequently Asked Questions and Advanced Debugging¶

This section delves into specific scenarios and provides detailed steps for diagnosing and resolving issues when ClusterResourcePlacement behaves unexpectedly.

How can I debug if some clusters aren’t selected as expected?¶

One of the most common issues in resource propagation involves clusters not being selected according to the ClusterResourcePlacement policy. The ClusterSchedulingPolicySnapshot object holds crucial information about which clusters were considered, which were selected, and importantly, the reasons behind their selection or non-selection. This snapshot is essentially a record of the scheduler’s decision-making process.

To debug selection issues, first, retrieve the relevant ClusterSchedulingPolicySnapshot related to your ClusterResourcePlacement object. This snapshot will detail the outcome of the scheduling policy evaluation. Examine its status and spec to understand the scheduler’s interpretation of your policy and the characteristics of available member clusters. Look for discrepancies between your intended cluster labels or properties and what the scheduler actually found. Common reasons for non-selection include label mismatches, policy conflicts, or insufficient available clusters meeting specific criteria like resource availability or geographic location. Carefully review the selectedClusters and unselectedClusters fields, paying close attention to any associated reasons for non-selection.

How can I debug if a selected cluster doesn’t have the expected resources or if ClusterResourcePlacement doesn’t pick up the latest changes?¶

This scenario often indicates a problem in the resource application pipeline on the member clusters or a failure to detect updates to the source resources. Follow these steps for a structured debugging approach:

1. Check ClusterResourcePlacementRolloutStarted condition:

   • Inspect the status of your ClusterResourcePlacement object to verify the ClusterResourcePlacementRolloutStarted condition. This condition confirms if the propagation process has even begun.
   • If False: The rollout hasn’t started. This could be due to issues earlier in the pipeline, such as scheduling failures (e.g., no clusters selected), or an internal error preventing the rollout initiation. Review Fleet controller logs on the hub cluster and the events of the ClusterResourcePlacement for clues.
   • If True: The rollout has initiated, so proceed to the next step.

2. Examine ClusterResourcePlacementApplied condition:

   • Next, check the ClusterResourcePlacementApplied condition within the ClusterResourcePlacement status. This condition directly reports on the success of applying resources to member clusters.
   • If Unknown: This state typically means the application process is still underway, or it might be stuck. Wait for a reasonable period for the process to complete. If the status remains Unknown for an extended duration, it suggests an unusual hang within the Fleet agent on the member cluster or a communication breakdown. In such cases, checking the Fleet agent’s logs on the specific member cluster is crucial. If it persists, it might warrant filing an issue with the Azure Kubernetes Fleet Manager team.
   • If False: This indicates that the resources failed to apply on one or more member clusters. This is a common point of failure. You must delve deeper into the specific application errors. Proceed to step 3.
   • If True: The resources were successfully applied. If you still don’t see the expected resources or changes, it might mean the source resources on the hub cluster either weren’t correctly defined, or ClusterResourcePlacement hasn’t yet picked up the latest version. Verify the existence and content of the original resources on the hub cluster to ensure they are as expected. Then, check the ClusterResourceSnapshot to see what version of the resources Fleet Manager captured.

3. Inspect placementStatuses for specific errors:

   • If ClusterResourcePlacementApplied was False, scroll down to the placementStatuses section within the ClusterResourcePlacement status output. This section provides per-cluster details regarding resource application.
   • Look for the FailedPlacements array or similar error fields for the particular cluster in question. This section often contains detailed error messages directly from the Kubernetes API server on the member cluster, explaining why a resource failed to apply (e.g., “Forbidden,” “already exists,” “invalid value,” “quota exceeded”). These messages are invaluable for diagnosing the precise cause of the failure.

How can I find and verify the latest ClusterSchedulingPolicySnapshot for a ClusterResourcePlacement deployment?¶

The ClusterSchedulingPolicySnapshot captures the scheduling decisions made for a ClusterResourcePlacement. It’s vital for understanding which clusters were considered and selected based on your policy. To retrieve the latest snapshot associated with your deployment, use the following kubectl command:

kubectl get clusterschedulingpolicysnapshot -l kubernetes-fleet.io/is-latest-snapshot=true,kubernetes-fleet.io/parent-CRP={CRPName}

Replace {CRPName} with the actual name of your ClusterResourcePlacement resource. This command will return the most recent ClusterSchedulingPolicySnapshot that reflects the current scheduling decisions.

Once you have the snapshot, compare its spec with the placement.policy section of your ClusterResourcePlacement resource. Ensure that the cluster selectors, affinity rules, and other policy definitions match your expectations. A crucial distinction to note is that the numberOfClusters field from the ClusterResourcePlacement spec is typically processed and not directly mirrored in the snapshot’s policy. If your placement type is PickN (meaning “pick N clusters”), confirm that the number of clusters requested in the ClusterResourcePlacement policy aligns with the value specified in the number-of-clusters label on the ClusterSchedulingPolicySnapshot. Any discrepancies here could explain unexpected cluster selections.

How can I find the latest ClusterResourceBinding resource?¶

ClusterResourceBinding objects act as the intermediary between a ClusterResourcePlacement and the Work objects destined for individual member clusters. Each selected member cluster will have a corresponding ClusterResourceBinding that tracks the status of resources being applied to it. To list all ClusterResourceBinding instances associated with a specific ClusterResourcePlacement, run:

kubectl get clusterresourcebinding -l kubernetes-fleet.io/parent-CRP={CRPName}

Let’s walk through an example to illustrate this:

1. View ClusterResourcePlacement status: First, you would describe your ClusterResourcePlacement (e.g., test-crp) to understand its overall status and the member clusters it targets.

   kubectl describe clusterresourceplacement test-crp
   

2. Interpret the placementStatuses: The output from the describe command will include a placementStatuses section. For instance, if test-crp has successfully distributed resources to kind-cluster-1 and kind-cluster-2, this section would look something like this:

   status:
     conditions:
     - lastTransitionTime: "2023-11-23T00:49:29Z"
       # ... other conditions ...
     placementStatuses:
     - clusterName: kind-cluster-1
       conditions:
         # ... conditions specific to kind-cluster-1 ...
         type: ResourceApplied
         status: "True"
         reason: ApplySucceeded
     - clusterName: kind-cluster-2
       conditions:
         # ... conditions specific to kind-cluster-2 ...
         reason: ApplySucceeded
         status: "True"
         type: ResourceApplied
   

   This output clearly indicates that test-crp has bindings for two member clusters, kind-cluster-1 and kind-cluster-2.

3. Retrieve ClusterResourceBindings: Now, use the command from above to fetch the actual ClusterResourceBinding objects:

   kubectl get clusterresourcebinding -l kubernetes-fleet.io/parent-CRP=test-crp
   

   The resulting output will list the specific ClusterResourceBinding resources:

   NAME                               WORKCREATED   RESOURCESAPPLIED   AGE
   test-crp-kind-cluster-1-be990c3e   True          True               33s
   test-crp-kind-cluster-2-ec4d953c   True          True               33s
   

   The naming convention for ClusterResourceBinding resources follows a predictable pattern: {CRPName}-{clusterName}-{suffix}. This naming makes it easy to associate a binding with its parent ClusterResourcePlacement and the specific member cluster it targets. Examining the WORKCREATED and RESOURCESAPPLIED columns provides a quick glance at the status of Work object creation and resource application for each binding.

How can I find the latest ClusterResourceSnapshot resource?¶

The ClusterResourceSnapshot object captures the precise version of the Kubernetes resources that are intended for propagation by a ClusterResourcePlacement. It acts as a static copy of the resources at a given point in time, ensuring consistency during the distribution process. If you suspect that the wrong version of a resource is being propagated, or if updates are not being picked up, checking the ClusterResourceSnapshot is crucial.

To retrieve the latest ClusterResourceSnapshot associated with a ClusterResourcePlacement (e.g., my-crp), execute the following command:

kubectl get clusterresourcesnapshot -l kubernetes-fleet.io/is-latest-snapshot=true,kubernetes-fleet.io/parent-CRP={CRPName}

This command filters for snapshots marked as the latest and linked to your specific ClusterResourcePlacement. Once retrieved, you can inspect the snapshot’s content (kubectl get clusterresourcesnapshot <name> -o yaml) to verify that it accurately reflects the resources and their desired state. If the snapshot does not contain the expected resource definitions or their latest changes, it suggests an issue earlier in the Fleet Manager pipeline, preventing the ClusterResourcePlacement from correctly capturing the updated resources.

How can I find the correct Work resource that’s associated with ClusterResourcePlacement?¶

The Work resource is the final payload that is delivered to a member cluster, containing the actual Kubernetes manifests to be applied. It resides in a special namespace on the member cluster itself, specifically designed for Fleet Manager operations. If resources are not appearing or are misbehaving on a specific member cluster, inspecting the Work resource on that cluster is a critical step.

Follow these steps to locate and inspect the relevant Work resource:

1. Identify the Member Cluster Namespace and CRP Name:

   • The member cluster namespace where Work objects reside follows the format: fleet-member-{clusterName}. For example, if your member cluster is named kind-cluster-1, the namespace will be fleet-member-kind-cluster-1.
   • Note down the name of your ClusterResourcePlacement (e.g., my-crp).

2. Retrieve the Work resource:

   • Using the identified clusterName and CRPName, execute the following kubectl command directly on the hub cluster, targeting the member cluster’s virtual namespace:

   kubectl get work -n fleet-member-{clusterName} -l kubernetes-fleet.io/parent-CRP={CRPName}
   

   Note: In this command, replace {clusterName} with the actual name of your member cluster and {CRPName} with the name of your ClusterResourcePlacement object. For example, kubectl get work -n fleet-member-kind-cluster-1 -l kubernetes-fleet.io/parent-CRP=test-crp.

   The output will list the Work resource(s) that Fleet Manager has attempted to apply to that specific member cluster for your ClusterResourcePlacement. Inspect the Work resource’s YAML (kubectl get work <work-name> -n fleet-member-{clusterName} -o yaml). The workload.manifests section will contain the actual Kubernetes resource manifests that were sent. Check the status section for any errors reported by the member cluster’s Fleet agent during the application process. This will provide direct evidence of what was sent and whether its application was successful or encountered an error.

General Best Practices for Troubleshooting Fleet Manager¶

Beyond specific API object inspection, adopting a few general troubleshooting practices can significantly expedite problem resolution within Azure Kubernetes Fleet Manager:

• Check Controller Logs: The Fleet controllers running on your hub cluster are responsible for managing ClusterResourcePlacement and related objects. Review their logs for any errors, warnings, or detailed messages that might indicate internal processing failures or misconfigurations. You can usually find these logs by inspecting the pods in the fleet-system namespace.
• Verify Member Cluster Connectivity: Ensure that your member clusters are properly connected to the hub cluster and that the Fleet agents on them are running and healthy. Network connectivity issues can prevent Work objects from being synchronized and applied.
• Examine RBAC Permissions: Resource application failures on member clusters are often due to insufficient Role-Based Access Control (RBAC) permissions. The service account used by the Fleet agent on the member cluster must have the necessary permissions to create, update, or delete the resources defined in your ClusterResourcePlacement.
• Validate Resource Manifests: Before propagation, ensure that the Kubernetes resource manifests themselves are syntactically correct and semantically valid for the target Kubernetes version on the member clusters. Invalid manifests will consistently fail during the application phase.
• Monitor Resource Quotas: If you are deploying many resources or large applications, ensure that the target namespaces on your member clusters have sufficient resource quotas (CPU, memory, storage) to accommodate the new resources. Exceeding quotas will lead to application failures.
• Version Compatibility: Verify that your Fleet Manager version and the Kubernetes versions on your hub and member clusters are compatible. Incompatibilities can lead to unexpected behavior.

Example: Troubleshooting a Pod Deployment Failure on a Member Cluster¶

Imagine you’ve deployed a Deployment via ClusterResourcePlacement, but the pods aren’t coming up on member-cluster-prod.

1. kubectl describe clusterresourceplacement my-app-crp: You see ClusterResourcePlacementApplied is False for member-cluster-prod in placementStatuses. The reason might be “Forbidden” or “ResourceQuotaExceeded”.
2. Identify the specific error:
   • If “Forbidden”, check the RBAC permissions for the Fleet agent’s service account (fleet-member-agent) in the fleet-member-prod namespace on member-cluster-prod. It might lack permissions to create Deployments or Pods.
   • If “ResourceQuotaExceeded”, inspect the ResourceQuota objects in the target namespace on member-cluster-prod to see if your deployment is requesting more resources than allowed.
3. Inspect Work resource on member cluster:
   • Run kubectl get work -n fleet-member-member-cluster-prod -l kubernetes-fleet.io/parent-CRP=my-app-crp on the hub cluster.
   • Then, kubectl get work <work-name> -n fleet-member-member-cluster-prod -o yaml.
   • In the status section of the Work resource, you’ll likely find the exact API server error message reported from member-cluster-prod, confirming the RBAC or quota issue.
4. Rectify the issue: Adjust RBAC roles, increase quotas, or modify the deployment’s resource requests. The ClusterResourcePlacement will then attempt to re-apply the resources.

Conclusion¶

Mastering the ClusterResourcePlacement API is fundamental to effectively managing your Kubernetes fleet with Azure Kubernetes Fleet Manager. By systematically examining the progression conditions, understanding the roles of associated API objects, and leveraging kubectl to inspect their statuses and logs, you can efficiently diagnose and resolve most resource propagation issues. Remember to approach troubleshooting methodically, starting from the ClusterResourcePlacement status and progressively drilling down into the ClusterSchedulingPolicySnapshot, ClusterResourceBinding, and Work resources, all while keeping an eye on controller and agent logs.

What challenges have you encountered with ClusterResourcePlacement in Azure Kubernetes Fleet Manager? Share your experiences and tips in the comments below!