Troubleshooting Azure VM Scale Set: Why Instances Aren't Being Repaired?

Table of Contents

Azure Virtual Machine Scale Set (VMSS) instances are designed to maintain a healthy state automatically when the automatic repair policy is enabled. However, sometimes instances may remain in an “Unhealthy” state and are not repaired as expected. This indicates that the automated repair process is encountering an issue preventing it from taking action. Several potential causes can lead to this situation, and understanding each one is crucial for effective troubleshooting. This article delves into the common reasons why your VMSS instances might not be undergoing automatic repairs and provides detailed steps to diagnose and resolve these issues. We will explore configuration problems, health monitoring misconfigurations, instance provisioning issues, policy suspensions, and the concept of the grace period.

When you observe that instances in your VMSS are consistently reported as “Unhealthy” and no repair actions are being initiated by the Azure platform, the first step is to methodically check the potential failure points. Automatic instance repair relies on a combination of scale set configuration, proper health signaling from the application running on the instances, and the overall state of the scale set’s repair policy. A failure in any of these areas can halt the automated repair process. It is important to approach troubleshooting systematically, starting with the most basic configuration checks and moving towards more complex scenarios.

Automatic Repairs Policy Isn’t Correctly Enabled¶

The automatic repairs feature in Azure VMSS is an opt-in capability. If the policy is not correctly enabled or is in a state other than Running, the platform will not attempt to repair unhealthy instances. It is essential to confirm the current state of the automatic repairs policy for your specific scale set.

You can check the service state of the automatic instance repairs policy using various Azure tools, including the Azure portal, Azure CLI, Azure PowerShell, or the REST API. In the Azure portal, navigate to your Virtual Machine Scale Set and look for the “Orchestration services” section. Within the Orchestration services properties, you should find an entry related to “Automatic repairs”.

Alternatively, using Azure CLI, you can retrieve the scale set details with a command like az vmss show --resource-group <your-rg> --name <your-vmss-name>. The output will include an orchestrationServices property. Within this property, look for the entry where serviceName is “AutomaticRepairs”. The corresponding serviceState indicates whether the policy is active.

Similarly, using Azure PowerShell, the command Get-AzVmss -ResourceGroupName <your-rg> -VMScaleSetName <your-vmss-name> will provide similar information. Examine the OrchestrationServices property in the returned object.

If the serviceState for automatic repairs is listed as NotRunning, it means the feature is not enabled for this scale set. If the orchestrationServices property or the “AutomaticRepairs” entry is missing entirely, the policy has never been explicitly configured or enabled. In either case, you must enable the automatic repairs policy. This typically involves updating the scale set configuration to explicitly set the serviceState for automatic repairs to Running.

Enabling the policy for an existing scale set requires an update operation. Ensure you follow the correct procedure for updating an existing scale set’s orchestration services configuration. After updating, verify the serviceState is indeed Running.

If the serviceState is found to be Suspended, this indicates a different issue. A Suspended state usually implies that the platform has temporarily halted repairs due to repeated failures. If you encounter this state, proceed to the section discussing suspension due to too many failed repairs. Ensuring the policy is correctly enabled and in the Running state is the foundational step for enabling automatic instance repairs.

Health Monitoring Isn’t Correctly Configured¶

Automatic instance repairs rely on accurate health signals emitted by the applications running within your VMSS instances. These health signals are typically provided through health probes configured either via the Application Health Extension or a Load Balancer probe associated with the scale set. If all instances in the scale set are showing an “Unhealthy” status, a common culprit is a misconfiguration in the health monitoring setup itself, rather than an actual issue with every single instance simultaneously.

The health probe is designed to check the availability and responsiveness of your application. For HTTP or HTTPS probes, the application is expected to return a 2xx series HTTP status code (e.g., 200 OK) from the configured endpoint path (e.g., /health, /status). For TCP probes, a successful TCP handshake on the specified port is required. If the probe does not receive the expected response within the configured timeout or frequency, the instance is marked as “Unhealthy” by the health monitoring system.

Common configuration issues include:
* Incorrect Protocol/Port: The probe might be configured for the wrong protocol (HTTP vs HTTPS vs TCP) or port number that your application is listening on.
* Wrong Endpoint Path: For HTTP/HTTPS probes, the specified path might be incorrect or the application is not configured to respond at that specific path.
* Firewall Issues: Network security groups (NSGs) or host-based firewalls on the VM instances might be blocking the incoming probe requests. The probe traffic originates from specific Azure infrastructure IP ranges or services tags (like AzureLoadBalancer or the virtual network’s address space for Application Health Extension) and must be allowed.
* Application Not Running/Responding Correctly: Even if the configuration is correct, the application might not be running or might be consistently failing to return the expected health signal due to internal errors.

It is crucial to ensure that your application is configured to emit the correct health signals and that the health probe configuration in Azure matches your application’s setup. Verify the protocol, port, and path (for HTTP/S) configured in the health probe settings. Check the network configuration (NSGs, firewalls) to ensure probe traffic is permitted. You can test the health endpoint directly from within the virtual network if possible to confirm the application’s response. If the health monitoring is misconfigured, the system will incorrectly mark all instances as unhealthy, and thus automatic repairs will not resolve the underlying configuration problem. Correcting the health probe configuration is necessary to get instances accurately reported as healthy.

The Instance Is Marked Unhealthy Due to a Provisioning Failure¶

Sometimes, an instance within a VMSS might fail during its initial provisioning process. This could happen due to various reasons, such as issues with the custom image used, problems during extension installation, or underlying infrastructure transient errors during creation. When an instance fails provisioning, it might end up in a state where it cannot run the application correctly or install the health agent/extension, leading to it being reported as “Unhealthy”.

You can determine if an instance failed provisioning by checking its detailed status, specifically the provisioning state. This information is available via the Azure portal, Azure CLI, Azure PowerShell, or the REST API. Using the REST API with API version 2019-12-01 or higher for the VMSS allows you to view a summary of provisioning states under the virtualMachine property, specifically in the statusesSummary.

For example, a successful deployment might show:

"virtualMachine": {
        "statusesSummary": [
            {
                "code": "ProvisioningState/succeeded",
                "count": 2
            }
        ]
}

If any instance failed provisioning, you would see an entry like "code": "ProvisioningState/failed" within this summary.

Alternatively, in the Azure portal, navigating to the VMSS instance details might show a provisioning status other than Succeeded. Using Azure CLI, you can use az vmss instance show --resource-group <your-rg> --vmss-name <your-vmss-name> --instance-id <instance-id> --query provisioningState. In PowerShell, Get-AzVmssVM -ResourceGroupName <your-rg> -VMScaleSetName <your-vmss-name> -InstanceId <instance-id> | Select-Object ProvisioningState will show this.

The crucial point here is that automatic instance repairs currently do not support scenarios where a virtual machine is marked “Unhealthy” specifically because of a provisioning failure. The repair mechanism is designed to replace or reimage instances that become unhealthy after being successfully provisioned and running. An instance that failed provisioning never reached a healthy operational state to begin with.

If you identify instances marked with a ProvisioningState/failed, the recommended solution is not to wait for automatic repair (which won’t happen) but to manually intervene. You should remove the failed instance from the scale set. Removing the instance stops billing for that specific VM and cleans up its associated resources. Once the failed instance is removed, you can then increase the capacity of the scale set by one to provision a new, hopefully successful, instance to replace the failed one. This manual process ensures that you replace the fundamentally flawed instance with a fresh attempt at provisioning.

Automatic Repairs Have Been Suspended in the Scale Set Due to Too Many Failed Repairs¶

Azure’s automatic repair mechanism includes safeguards to prevent a repair loop that could exacerbate problems or lead to a cascade of failures. If the platform repeatedly attempts to repair an instance (e.g., by reimaging or deleting/recreating it), but the instance continues to report an “Unhealthy” status after each repair attempt, the automatic repair policy for the entire scale set might be automatically suspended. This happens when the system detects a pattern of unsuccessful repairs, suggesting an underlying issue that cannot be resolved by simple instance replacement.

This suspension is a safety measure. It prevents the scale set from potentially destroying and recreating instances endlessly if, for example, the “Unhealthy” state is caused by a persistent issue like a critical application bug deployed to all instances, a fundamental configuration error in the image, or an external dependency failure that affects all VMs.

When automatic repairs are suspended, the serviceState for the automatic repairs policy within the scale set’s orchestrationServices property changes to Suspended. As mentioned earlier, you can check this state using the Azure portal, CLI, PowerShell, or REST API as described in the section on checking if the policy is enabled.

If you find the serviceState is Suspended, the first step is to identify and resolve the root cause of why the repair attempts were failing. This often points back to issues discussed in the “Health monitoring isn’t correctly configured” section. If the application consistently reports “Unhealthy” due to misconfiguration, a bug, or an environment issue, simply replacing the instance won’t fix it, and the new instance will likely also be marked “Unhealthy”. You must address the underlying problem that is causing instances to be unhealthy after repair or provisioning.

Once you have investigated and fixed the root cause (e.g., corrected the health probe configuration, deployed a fixed application version, resolved external dependency issues), you can then resume automatic repairs. Resuming the policy involves updating the scale set configuration to change the serviceState for automatic repairs back from Suspended to Running. This can be done using the setOrchestrationServiceState API call or equivalent commands in Azure CLI or PowerShell.

After resuming the policy, the platform will begin monitoring instance health again and attempt repairs on any instances still reporting as “Unhealthy”, provided the underlying issue has been resolved and they can now reach a healthy state.

The Instance Is in Its Grace Period¶

If you have checked all the previous potential causes – confirmed the automatic repairs policy is enabled and running, validated the health monitoring configuration, ensured the instance wasn’t a provisioning failure, and verified the policy isn’t suspended – but an instance is still “Unhealthy” and not being repaired, it might simply be within its configured grace period.

The grace period is a built-in mechanism within the automatic repairs feature designed to prevent premature or accidental repairs. When an instance undergoes a state transition – such as starting up, being updated, or restarting – it needs some time to become fully operational. This includes time for the operating system to boot, for the VM agent and extensions (like the Application Health Extension) to initialize, and for the application itself to start and begin emitting health signals. The grace period provides this crucial window.

During the grace period, even if the health probe initially reports the instance as “Unhealthy” (because the application hasn’t started yet or the agent isn’t ready), the automatic repair mechanism will wait. It will only consider initiating a repair action after the grace period has elapsed and if the instance is still reporting as “Unhealthy” at that point.

The default grace period is 10 minutes, but it is configurable. You can set a longer grace period if your application takes more time to start up and become healthy. Conversely, you might set a shorter one if your instances become operational very quickly.

If an instance has recently started, restarted, or been updated, check its timeline of events. It is possible that the “Unhealthy” status was detected, but the instance is simply waiting for its grace period to finish before the repair is evaluated or attempted. The repair action should occur shortly after the grace period concludes, assuming the instance is still unhealthy and all other conditions for repair are met. If the instance becomes healthy during the grace period, no repair is needed.

Understanding the configured grace period for your scale set is important when troubleshooting delays in repair actions. You can find the grace period setting within the automatic repair policy configuration of your VMSS.

In summary, troubleshooting why VMSS instances aren’t being repaired requires a systematic approach. Check the policy state, validate health monitoring, rule out provisioning failures, verify policy suspension status, and consider the grace period. By methodically investigating these areas, you can identify the specific cause and take the appropriate steps to restore the automatic repair functionality and maintain the health of your scale set instances.

Have you encountered other reasons for VMSS instances not being repaired automatically? Share your experiences and solutions in the comments below!