Troubleshooting VM Extension Provisioning Failures in Azure Virtual Machine Scale Sets

Table of Contents

Azure Virtual Machine Scale Sets (VMSS) allow you to deploy and manage a set of identical virtual machines as a single unit. They are designed for scalability and resilience, making it easy to deploy large-scale workloads. A crucial part of configuring and managing these instances is the use of VM extensions, which provide capabilities like configuration management, monitoring, security, and application deployment. However, issues during the provisioning of these extensions can halt deployments, updates, or scaling operations within your Scale Set, leading to frustrating errors.

This article focuses on resolving common errors related to VM extension provisioning in Azure Virtual Machine Scale Sets. Specifically, we address VMExtensionProvisioningError, VMExtensionHandlerNonTransientError, and VMExtensionProvisioningTimeout. Understanding the nature of these errors and having a systematic troubleshooting approach is key to maintaining the health and availability of your VMSS instances. Note that when these errors appear in the context of a Virtual Machine Scale Set, the term “VM” refers to an individual instance within that specific Scale Set.

Understanding VM Extension Provisioning¶

When an extension is applied to a VMSS instance, the Azure fabric instructs the Azure VM Agent running inside the virtual machine to download, install, and execute the extension’s payload. The agent reports the status of this process back to the Azure platform. The provisioning state reflects the current phase of this operation. It typically progresses through states like ‘Creating’, ‘Transitioning’, and ideally ends in ‘Succeeded’.

Failures can occur at various stages of this process. An extension might fail to download, encounter issues during installation, or the script/application it runs might exit with an error or take too long to complete. These failures prevent the provisioning state from reaching ‘Succeeded’, leaving the instance or the entire Scale Set in a failed state. Diagnosing the exact point of failure is essential for effective remediation.

Recognizing the Symptoms¶

The presence of VMExtensionProvisioningError, VMExtensionHandlerNonTransientError, or VMExtensionProvisioningTimeout errors is the primary symptom indicating issues with extension deployment on VMSS instances. These errors are typically visible in the provisioning state of the VMSS instance itself or in the details of a failed deployment operation. They block the instance from becoming fully operational according to the Scale Set model.

Here are examples of how these errors might manifest in status messages or activity logs:

'statusMessage': '{\\'status\\\\':\\\\'Failed\\\\',\\\\'error\\\\':{\\\\'code\\\\':\\\\'ResourceOperationFailure\\\\',\\\\'message\\\\':\\\\'The resource operation completed with terminal provisioning state 'Failed'.\\\\',\\\\'details\\\\':[{\\\\'code\\\\':\\\\'VMExtensionProvisioningError\\\\',\\\\'message\\\\':\\\\'Multiple VM extensions failed to be provisioned on the VM. Please see the VM extension instance view for other failures.'}]}}

This VMExtensionProvisioningError indicates that one or more extensions failed. It’s a general error suggesting you need to investigate the specific extension failures on the instance.

{'status': 'Failed','error': {'code':'VMExtensionHandlerNonTransientError','message': 'The handler for VM extension type 'Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux' has reported terminal failure for VM extension 'OmsAgentForLinux' with error message: '[ExtensionOperationError] Non-zero exit code: 10'}}

A VMExtensionHandlerNonTransientError suggests the extension handler itself reported a fatal error. This often comes with a specific error message or exit code from the extension’s execution, like the ‘Non-zero exit code: 10’ example shown here, pointing to an issue within the extension’s script or application. This is considered non-transient, meaning it’s unlikely to resolve without intervention.

'statusMessage': '{\\'status\\\\':\\\\'Failed\\\\',\\\\'error\\\\':{\\\\'code\\\\':\\\\'ResourceOperationFailure\\\\',\\\\'message\\\\':\\\\'The resource operation completed with terminal provisioning state 'Failed'.\\\\',\\\\'details\\\\':[{\\\\'code\\\\':\\\\'VMExtensionProvisioningTimeout\\\\',\\\\'message\\\\':\\\\'Provisioning of VM extension configure-settings has timed out. Extension provisioning has taken too long to complete.'}]}}

The VMExtensionProvisioningTimeout error occurs when the extension provisioning process exceeds the maximum allowed time. This could be due to the extension’s task taking too long to complete, network issues preventing necessary downloads, or the VM agent being unresponsive. It signifies that the operation did not finish within the expected timeframe.

These errors collectively point to a problem where a VM extension is stuck or has failed during its execution lifecycle on one or more instances within the Virtual Machine Scale Set.

Common Causes of Failure¶

VM extension provisioning failures can stem from a variety of issues. Identifying the root cause is critical for a permanent solution. Some common causes include:

• Network Connectivity Issues: The VM instance might be unable to reach necessary endpoints, such as Azure storage (to download extension files), package repositories (for installing dependencies), or external services required by the extension script. Firewall rules, Network Security Groups (NSGs), or custom routing can block this traffic.
• Dependencies Not Met: The extension’s script or application might require specific software, libraries, or configurations that are not present on the VM instance. If these prerequisites are not met before the extension runs, it will likely fail.
• Errors in Extension Script or Configuration: For extensions like the Custom Script Extension (CSE) or Desired State Configuration (DSC), errors within the provided script or configuration are frequent causes of failure. This could include syntax errors, incorrect file paths, invalid commands, or issues with permissions within the script context.
• Long Running Tasks: Scripts or operations initiated by the extension might take an excessive amount of time to complete, exceeding the provisioning timeout limit. This is common if the script involves downloading large files, compiling software, or waiting for external processes without proper handling.
• VM Agent Issues: The Azure VM Agent is responsible for processing extension requests. If the agent is not running, is outdated, or is corrupted, it cannot execute the extension properly, leading to failures or timeouts.
• Resource Constraints: The VM instance might lack sufficient resources (CPU, memory, disk space) to successfully run the extension’s tasks. For example, a script requiring significant memory might fail if the VM is undersized or already under heavy load.
• Permission Issues: The context under which the extension runs might not have the necessary permissions on the operating system to perform its tasks, such as installing software, modifying system files, or accessing network resources.
• Configuration Conflicts: Multiple extensions or existing configurations on the VM might conflict with the requirements or actions of the failing extension.

Understanding these potential causes helps narrow down the investigation when encountering provisioning errors.

Initial Troubleshooting Steps¶

When faced with VM extension provisioning errors, a structured approach is best. The first step is to gather more detailed information about the failure.

Get More Information About Extension Failure¶

The Azure CLI is a powerful tool for diagnosing VMSS issues. You can use the az vmss list-instances command to check the provisioning state of extensions across all instances in your Scale Set. This command helps identify which specific instances and extensions are affected.

Run the following command, replacing MyResourceGroup and MyVmss with your actual resource group name and Scale Set name:

az vmss list-instances --resource-group MyResourceGroup --name MyVmss --query "[].{instanceId:instanceId, extension:resources[].id, extProvisioningState:resources[].provisioningState}"

The output of this command provides a list of your VMSS instances, each showing the extensions applied and their current provisioning states. It’s formatted to show the instanceId, a list of extension IDs (names), and a corresponding list of extProvisioningState values.

For example, the output might look something like this (simplified):

[
  {
    "extProvisioningState": [
      "Failed",
      "Succeeded",
      "Succeeded"
    ],
    "extension": [
      "/subscriptions/subid/resourceGroups/rg/providers/Microsoft.Compute/virtualMachineScaleSets/vmss/virtualMachines/0/extensions/customScript",
      "/subscriptions/subid/resourceGroups/rg/providers/Microsoft.Compute/virtualMachineScaleSets/vmss/virtualMachines/0/extensions/AzureMonitorAgent",
      "/subscriptions/subid/resourceGroups/rg/providers/Microsoft.Compute/virtualMachineScaleSets/vmss/virtualMachines/0/extensions/DependencyAgent"
    ],
    "instanceId": "0"
  },
  {
    "extProvisioningState": [
      "Succeeded",
      "Succeeded",
      "Succeeded"
    ],
    "extension": [
      "/subscriptions/subid/resourceGroups/rg/providers/Microsoft.Compute/virtualMachineScaleSets/vmss/virtualMachines/1/extensions/customScript",
      "/subscriptions/subid/resourceGroups/rg/providers/Microsoft.Compute/virtualMachineScaleSets/vmss/virtualMachines/1/extensions/AzureMonitorAgent",
      "/subscriptions/subid/resourceGroups/rg/providers/Microsoft.Compute/virtualMachineScaleSets/vmss/virtualMachines/1/extensions/DependencyAgent"
    ],
    "instanceId": "1"
  }
]

Carefully examine the output for instances where extProvisioningState shows ‘Failed’ or is stuck in a ‘Transitioning’ state for an extended period. Match the position in the extProvisioningState list to the corresponding extension ID in the extension list for that specific instance. This will tell you exactly which extension failed on which instance(s). In the example above, instance ‘0’ shows ‘customScript’ as failed, while the other extensions and instance ‘1’ are successful.

Attempt to Scale Out the Virtual Machine Scale Set¶

If the failure appears to be isolated to only a few instances rather than all of them, a quick diagnostic step is to try scaling out the Scale Set. Add one or more new instances.

If the newly created instances provision successfully, including all extensions, it suggests the issue might be specific to the original failing instances or a transient problem that has since resolved. In this scenario, you can delete the instances where the extension provisioning failed. This is often the fastest way to restore the desired instance count with healthy VMs if the number of affected instances is small.

However, if the new instances also experience the same extension provisioning failures, the problem is likely systemic. This indicates an issue with the Scale Set model configuration, the extension configuration itself, or a dependency accessible to all instances.

Deep Dive: Analyzing Logs on Impacted Instances¶

Once you’ve identified the failing instance(s) and extension(s), the next crucial step is to examine the logs directly on the affected virtual machines. The Azure VM Agent and each extension maintain detailed logs about their operations. These logs often contain specific error messages, warnings, or diagnostic information that pinpoint the exact cause of the failure.

Connect to one of the impacted instances using RDP (for Windows) or SSH (for Linux). Once connected, navigate to the appropriate directory to find the extension logs.

• For Windows Virtual Machine Scale Sets:
  Navigate to C:\WindowsAzure\logs\plugins\ExtensionName\. Inside this directory, you will find log files, typically named Extension.log or similar, along with version-specific subdirectories. Check the logs corresponding to the version of the extension being deployed.
  Also check the Azure VM Agent logs located at C:\WindowsAzure\logs\WaAppAgent.log. This log records the agent’s communication with the Azure fabric and its attempts to download and launch extensions.
• For Linux Virtual Machine Scale Sets:
  Navigate to /var/log/plugins/ExtensionName/. Similar to Windows, you’ll find log files like Extension.log and potentially versioned directories. Examine the relevant log files for the failing extension.
  Review the main Azure VM Agent log file at /var/log/waagent.log. This log provides insights into the agent’s activities, including extension handling.

When reviewing the logs, pay close attention to:
* Timestamps: Correlate log entries with the time you attempted to deploy or update the Scale Set.
* Error Messages: Look for explicit error messages, exceptions, or failed command executions.
* Exit Codes: Non-zero exit codes from scripts or processes usually indicate a failure. The specific code might correspond to an operating system error or an application-specific error.
* Network Errors: Look for messages related to connection refused, timeouts, or unreachable hosts if you suspect network issues.
* Dependency Errors: Check for messages indicating missing files, libraries, or failed package installations.
* Permissions Errors: Look for “permission denied” or similar messages.

Analyzing these logs is often the most direct way to understand why the extension failed to provision.

Verifying Extension Best Practices¶

For highly customizable extensions like the Custom Script Extension (CSE) or the Desired State Configuration (DSC) extension, configuration errors are very common. Ensure that the script or configuration you are providing adheres to best practices and meets all necessary prerequisites.

• Custom Script Extension (CSE):
  • Ensure the script is idempotent: Running it multiple times should have the same result as running it once.
  • Scripts should handle errors gracefully and provide informative output/exit codes.
  • Avoid very long-running scripts. Break down complex tasks if necessary.
  • Ensure any files or dependencies the script needs are accessible (e.g., stored in Azure Storage).
  • Consider the execution context – does the user running the script (typically Local System/root) have the necessary permissions?
• Desired State Configuration (DSC):
  • Verify the DSC configuration syntax is correct.
  • Ensure all required DSC modules are available to the VM. These might need to be staged or pulled from a repository.
  • Check pull server configuration if using that method.
  • Validate that the resources defined in the configuration are applicable and can be applied successfully to the target OS version.

Many other extensions, such as monitoring agents or security extensions, also have specific prerequisites (e.g., outbound connectivity to certain endpoints, specific OS versions, sufficient disk space). Always consult the official documentation for the specific extension you are troubleshooting.

Advanced Troubleshooting Techniques¶

If basic log analysis doesn’t reveal the cause, consider these more advanced steps:

• Check VM Agent Status: On the affected instance, verify that the Azure VM Agent service is running and healthy. Restarting the agent can sometimes resolve transient communication issues.
• Test Network Connectivity: From the failing instance, use tools like ping, telnet, curl, or wget to test connectivity to any external endpoints required by the extension (e.g., storage accounts, package repositories, licensing servers).
• Manually Execute Extension Logic: For script-based extensions (CSE, DSC), try executing the core logic or script manually on an instance while logged in. This allows you to debug interactively, see error messages directly, and test individual commands.
• Review Azure Activity Logs: Check the Azure Activity Log for the VMSS resource and the individual VM instances. Look for any related operations that might have failed or provided additional clues around the time the extension provisioning failed. Resource provider errors might show up here.
• Check Azure Service Health: Occasionally, regional Azure service incidents can affect VM or extension functionality. Review the Azure Service Health dashboard for any known issues in your region.

By systematically testing dependencies and execution manually, you can often isolate the exact point of failure within the extension’s logic or environment.

Reinstalling the Extension¶

If you’ve identified a potential issue (e.g., corrected a script error, fixed network access, or suspect a corrupted extension state), or if other troubleshooting steps fail, reinstalling the extension can force a clean deployment attempt.

You can do this through the Azure portal:
1. Navigate to your Virtual Machine Scale Set resource in the Azure portal.
2. Select the Extensions blade.
3. Find the extension that is showing provisioning errors or is stuck.
4. Select the extension and click the Uninstall button. Confirm the uninstallation.
5. Wait a few minutes for the uninstallation to process.
6. On the Extensions blade, click Add.
7. Find and select the same extension from the list.
8. Configure the extension settings as required and click Apply or OK to initiate the installation again.

Alternatively, you can remove and re-add the extension definition in your VMSS model using Azure CLI, PowerShell, or by redeploying the ARM template defining the Scale Set. This ensures the Scale Set model pushes a fresh installation command to the affected instances.

# Example using Azure CLI to remove and then add an extension (Conceptual steps)
# First, get the current model and remove the extension part
# Then update the model to remove the extension
# az vmss update --resource-group MyResourceGroup --name MyVmss --remove virtualMachineProfile.extensionProfile.extensions {extensionName}

# Then add the extension back in your desired configuration
# az vmss extension set --resource-group MyResourceGroup --vmss-name MyVmss --name {extensionName} --publisher {Publisher} --version {Version} --settings '{...}' --protected-settings '{...}'

Note: The exact commands to modify the VMSS model can be complex. Exporting the existing template or using az vmss extension commands are common approaches.

This process essentially clears the failed state and prompts the VM agent to download and execute the extension from scratch.

Supporting Media for Deeper Understanding¶

To better visualize the process and common failure points, consider the following supporting resources:

VM Extension Provisioning Lifecycle (Mermaid Diagram)¶

This diagram illustrates the typical flow of an extension being applied to a VMSS instance and where failures can occur.

mermaid graph TD A[Azure Scale Set Model Update] --> B{Extension Definition Added/Updated}; B --> C[Azure Fabric Initiates Extension Deployment]; C --> D[Azure VM Agent Receives Command]; D --> E{Agent Downloads Extension Payload}; E --> |Success| F{Agent Verifies Payload}; F --> |Success| G{Agent Installs Extension}; G --> |Success| H{Agent Executes Extension Logic}; H --> I{Extension Reports Status}; I --> |Success| J[Provisioning State: Succeeded]; I --> |Failure| K[Provisioning State: Failed]; E --> |Failure: Network/Storage| K; F --> |Failure: Signature/Corruption| K; G --> |Failure: OS Issues/Dependencies| K; H --> |Failure: Script Errors/Timeouts| K; D --> |Failure: Agent Unresponsive| K; C --> |Failure: Platform Issues| K;

Explanation: The process starts with a change in the VMSS model. The Azure fabric instructs the agent. The agent downloads, verifies, installs, and runs the extension. Failures can happen at any step, leading to a ‘Failed’ provisioning state. Timeouts can occur during download, installation, or execution if they take too long.

Table of Common Extensions and Log Locations¶

Extension Name (Publisher)	Windows Log Path	Linux Log Path	Key things to look for in logs
Custom Script Extension (Microsoft.Compute)	C:\WindowsAzure\logs\plugins\Microsoft.Compute.CustomScriptExtension\	/var/log/plugins/Microsoft.Compute.CustomScriptExtension/	Script output (stdout/stderr), exit code, download errors
Desired State Configuration (Microsoft.Powershell.DSC)	C:\WindowsAzure\logs\plugins\Microsoft.Powershell.DSC\	/var/log/plugins/Microsoft.Powershell.DSC/	Configuration application details, syntax errors, dependency issues
Azure Monitor Agent (Microsoft.Azure.Monitor)	C:\Packages\Plugins\Microsoft.Azure.Monitor.AzureMonitorAgent\	/var/lib/waagent/Microsoft.Azure.Monitor.AzureMonitorAgent/	Agent connectivity, data collection errors, configuration issues
Dependency Agent (Microsoft.Azure.Monitoring.DependencyAgent)	C:\Packages\Plugins\Microsoft.Azure.Monitoring.DependencyAgent\	/var/lib/waagent/Microsoft.Azure.Monitoring.DependencyAgent/	Connectivity to Map service, process monitoring details
Azure Security Agent (Microsoft.Azure.Security)	C:\WindowsAzure\logs\plugins\Microsoft.Azure.Security.SecurityOperations\	/var/log/plugins/Microsoft.Azure.Security.SecurityOperations/	Security scan results, policy application, connectivity

Note: Log paths can vary slightly based on extension version and OS configuration. Always check the directories listed.

Relevant YouTube Video Topic¶

Searching platforms like YouTube for videos covering “Troubleshooting Azure VMSS Extension Failures” or “Debugging Azure Custom Script Extension” can provide visual walkthroughs of connecting to VMs, navigating file systems, and interpreting log files. Look for videos demonstrating the use of the Azure portal, Azure CLI, and connecting via RDP/SSH to perform diagnostics on VMSS instances. These resources can complement the steps outlined in this article.

Preventing Future Failures¶

Preventing extension provisioning failures is better than troubleshooting them. Consider implementing these practices:

• Thorough Testing: Test your extension configurations and scripts on individual virtual machines or a small, separate VMSS before deploying to production Scale Sets.
• Use Azure Monitor: Configure Azure Monitor to track VMSS instance provisioning states. Set up alerts for instances entering a ‘Failed’ state.
• Review Dependencies: Clearly document and ensure all prerequisites (software, network access, permissions) are met before applying extensions. Use custom VM images with pre-installed dependencies if possible.
• Implement Idempotency: Design custom scripts and configurations to be idempotent so running them multiple times doesn’t cause issues.
• Manage Credentials Securely: Use Managed Identities or Key Vault for accessing resources instead of embedding credentials directly in scripts or configurations.
• Monitor VM Agent Health: Ensure the Azure VM Agent is running and updated on your VM images.

By adopting robust deployment practices and leveraging Azure monitoring capabilities, you can significantly reduce the occurrence and impact of VM extension provisioning failures.

Successfully troubleshooting VM extension provisioning errors in Azure VM Scale Sets requires a combination of understanding the error messages, using Azure tools to pinpoint the affected resources, and diving into the operating system logs on the individual instances. By systematically following the steps outlined above – identifying affected instances, analyzing logs, verifying configurations, and leveraging advanced techniques – you can diagnose and resolve most common extension provisioning issues. Remember to verify best practices for extensions like CSE and DSC and consider reinstalling the extension if necessary. Proactive monitoring and thorough testing of extensions before deployment are key to preventing these issues in the future.

What has been your experience troubleshooting VM extension provisioning failures in Azure VM Scale Sets? Share your common causes or effective techniques in the comments below!