Troubleshooting PowerShell Scripts in Azure VMs: Debugging Custom Script Extension and Run Command

Table of Contents

This guide provides a comprehensive approach to testing and resolving failures in PowerShell scripts executed within Azure Windows Virtual Machines (VMs). It specifically focuses on troubleshooting issues encountered when deploying scripts using the Azure Custom Script Extension or the Run Command feature. Understanding how to effectively diagnose these problems is crucial for maintaining the operational health and automation capabilities of your Azure infrastructure.

The techniques discussed here will empower administrators and developers to quickly identify the root cause of script failures, ranging from simple syntax errors to complex permission-related issues. By leveraging built-in Azure tools and advanced logging capabilities, you can streamline your troubleshooting process and ensure your automation scripts run smoothly. This article applies exclusively to Windows VMs.

Prerequisites¶

Before delving into the debugging methods, ensure you have the following tools and knowledge at your disposal. These prerequisites are essential for performing the manual tests and advanced logging configurations discussed later.

• Azure PowerShell: This command-line shell and scripting language is used for managing and administering Azure resources. Having it installed and configured on your local machine allows you to interact with your Azure VMs, including deploying and managing extensions or invoking commands.
  
• PsExec tool: Part of the Sysinternals Suite, PsExec is a powerful command-line utility that enables you to execute processes on remote systems. For troubleshooting, it’s particularly useful for running scripts under the Local System account directly on your Azure VM, simulating the execution environment of the Custom Script Extension and Run Command.
  

Understanding Script Execution and Output Streams¶

When a PowerShell script is executed on an Azure VM using either the Custom Script Extension or the Run Command feature, it produces various output streams. These streams are fundamental to understanding the script’s behavior and diagnosing any failures. PowerShell typically uses several output streams, but for troubleshooting, the Success stream and the Error stream are the most relevant.

The logs generated by both the Custom Script Extension and Run Command scripts channel the Success stream to the StdOut substatus and the Error stream to the StdErr substatus. These substatuses are integral components of the specific extension that initiated the script’s execution. By examining these substatuses, you can often gain immediate insight into whether a script completed successfully or encountered an error.

The StdOut and StdErr substatuses are accessible through the certificate registration point (CRP) instance view of the virtual machine. This means they are visible across multiple Azure interfaces, providing flexible access for administrators. The table below outlines the various ways you can view these crucial substatuses to begin your troubleshooting journey.

| Interface | How to View the Substatus Azure portal Search for and select Virtual machines. Select your VM from the list. On the VM overview page, select Extensions + applications > Extensions. Select the extension that was used to run the command. (It will be named either CustomScriptExtension or RunCommand.) Select View detailed status.
| Azure PowerShell | Enter the Get-AzVM cmdlet to get the properties of an Azure VM, as follows:
Get-AzVM -ResourceGroupName <resource-group-name> -Name <vm-name> -Status |

The error that typically leads to script failure appears in the StdErr substatus. However, it’s worth noting that scripts can sometimes fail without logging a fatal error entry within this specific substatus. In such cases, deeper investigative methods become necessary.

Testing the Script Manually and by Using PsExec¶

When a script fails, the initial troubleshooting step should always involve manual verification. This helps determine if the issue lies with the script itself or with the execution environment provided by the Custom Script Extension or Run Command.

Manual Verification on the VM¶

Log directly into your Azure VM and attempt to run the problematic PowerShell script from an administrative PowerShell console. This manual execution helps you identify basic issues like syntax errors, missing dependencies, or incorrect paths that might not be immediately obvious from the extension logs. If the script executes successfully under these conditions, it suggests that the problem is not with the script’s core functionality but rather with how the Azure extension is attempting to run it.

Understanding the Local System Account¶

For both the Custom Script Extension and the Run Command feature, scripts are executed using the Local System account on the VM. This is a highly privileged account with extensive permissions, but it operates differently from a logged-in user account. Issues can arise if the script relies on user profile settings, network drives mapped under a specific user, or environment variables that are not present for the Local System account. This distinction is critical for accurate debugging.

Testing by Using PsExec¶

If your script functions correctly when run manually from an administrative console, the next logical step is to simulate the execution environment of the Azure extensions more closely. The PsExec tool allows you to run the script using the Local System account, effectively bypassing the Custom Script Extension or Run Command mechanism while retaining the same user context. This helps isolate whether the failure is reproducible under the Local System account or if it’s directly related to the extension’s interaction.

By entering psexec -s along with your script, you can test its behavior under the Local System account. If the script failure is reproduced using this method, it strongly indicates that the issue is within the script’s compatibility with the Local System account’s environment, rather than a problem with the Custom Script Extension or Run Command themselves. This isolation significantly narrows down the scope of your investigation.

To run a PowerShell test script remotely using PsExec, open an administrative Command Prompt window. Replace the placeholder with the fully qualified path and name of your PowerShell script, then execute the following command:

psexec -accepteula -nobanner -s powershell.exe -NoLogo -NoProfile -File <C:\path\script-name.ps1>

Alternatively, you can use PsExec interactively to confirm the current user context. This allows you to open a PowerShell session under the Local System account and execute commands directly. In the example below, the whoami command demonstrates that PowerShell is indeed running under the Local System (NT AUTHORITY\SYSTEM) account:

C:\>psexec -accepteula -nobanner -s powershell.exe -NoLogo -NoProfile

PS C:\Windows\system32> whoami
nt authority\system

Enhancing PowerShell Script Execution Logging for Deeper Insight¶

When the StdErr substatus does not provide sufficient detail to diagnose a script failure, enabling more granular logging becomes essential. By activating various types of logging, you can collectively capture a comprehensive view of the script’s content, its execution flow, and the resulting output. This detailed record is invaluable for understanding exactly what the script attempted to achieve and why it might have failed.

Turning on these logging mechanisms creates a much richer dataset for analysis. It allows you to trace command invocations, examine script block contents, and review process creation events, all of which contribute to a clearer picture of the script’s journey through the system. Follow the steps in the subsequent sections to configure these advanced logging options on your Azure VM.

Increasing Event Log Maximum Sizes¶

Before enabling extensive logging, it is prudent to increase the maximum size of relevant event logs. The Security log and the Microsoft-Windows-PowerShell/Operational event log can generate a substantial number of events, especially when detailed auditing is turned on. Increasing their capacity prevents the loss of crucial log entries due to log wrapping, ensuring you have a complete record of events during your troubleshooting period.

However, if either of these logs is already at its maximum size of 100 MB or larger (indicated by a maxSize value of 104,857,600 or more), there is no need to make further adjustments. To check the current maximum size of these logs, you can use the wevtutil command with the get-log option. This command retrieves detailed information about the specified event logs.

wevtutil get-log "Security"
wevtutil get-log "Microsoft-Windows-PowerShell/Operational"

You will observe output similar to the following examples, illustrating the current maxSize setting. In these typical scenarios, the maximum log size is significantly less than the recommended 100 MB, necessitating an increase for effective troubleshooting.

name: Security
enabled: true
type: Admin
owningPublisher:
isolation: Custom
channelAccess: O:BAG:SYD:(A;;CCLCSDRCWDWO;;;SY)(A;;CCLC;;;BA)(A;;CC;;;ER)(A;;CC;;;NS)
logging:
  logFileName: %SystemRoot%\System32\Winevt\Logs\Security.evtx
  retention: false
  autoBackup: false
  maxSize: 20971520
publishing:
  fileMax: 1

name: Microsoft-Windows-PowerShell/Operational
enabled: true
type: Operational
owningPublisher: Microsoft-Windows-PowerShell
isolation: Application
channelAccess: O:BAG:SYD:(A;;0x2;;;S-1-15-2-1)(A;;0x2;;;S-1-15-3-1024-3153509613-960666767-3724611135-2725662640-12138253-543910227-1950414635-4190290187)(A;;0xf0007;;;SY)(A;;0x7;;;BA)(A;;0x7;;;SO)(A;;0x3;;;IU)(A;;0x3;;;SU)(A;;0x3;;;S-1-5-3)(A;;0x3;;;S-1-5-33)(A;;0x1;;;S-1-5-32-573)
logging:
  logFileName: %SystemRoot%\System32\Winevt\Logs\Microsoft-Windows-PowerShell%4Operational.evtx
  retention: false
  autoBackup: false
  maxSize: 15728640
publishing:
  fileMax: 1

To increase the maximum size of both the Security log and the Microsoft-Windows-PowerShell/Operational event log to 100 MB, execute the wevtutil command along with the set-log option. This adjustment provides ample space for capturing detailed events during your troubleshooting session.

wevtutil set-log "Security" /ms:104857600
wevtutil set-log "Microsoft-Windows-PowerShell/Operational" /ms:104857600

Enabling Process Creation Auditing¶

Process creation auditing is a critical logging feature that records every new process initiated on the system, including the command-line arguments used to launch it. This level of detail is invaluable when debugging scripts, as it allows you to see exactly how PowerShell was invoked and with what parameters. By enabling this auditing, you can verify if your script is being called with the correct arguments or if any unexpected modifications are occurring during its execution.

Use the following registry command to enable process creation auditing. This modification ensures that the system begins recording comprehensive information about process starts, which is then logged to the Security event log for your review.

reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System\Audit" /v "ProcessCreationIncludeCmdLine_Enabled" /t REG_DWORD /d 1 /f

Enabling PowerShell Transcription¶

PowerShell transcription captures a detailed record of all input and output from PowerShell sessions, including commands executed and the results they produced. This feature creates a plaintext log of your PowerShell activity, providing an exact reproduction of what occurred during script execution. It’s particularly useful for understanding the exact flow of a script and any interactive elements that might be at play.

To enable PowerShell transcription, you need to modify specific registry keys. These changes configure PowerShell to automatically generate transcription logs in a designated directory. Ensure the specified OutputDirectory exists or will be created, as this is where your transcription files will be stored.

reg add "HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\Windows\PowerShell\Transcription" /v "EnableTranscripting" /t REG_DWORD /d 1 /f
reg add "HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\Windows\PowerShell\Transcription" /v "EnableInvocationHeader" /t REG_DWORD /d 1 /f
reg add "HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\Windows\PowerShell\Transcription" /v "OutputDirectory" /t REG_SZ /d C:\Transcripts /f

Enabling PowerShell Module Logging¶

PowerShell module logging records pipeline execution details for specific PowerShell modules. When enabled, it captures information such as the name of the cmdlet executed, its parameters, and the output generated. This is incredibly useful for pinpointing exactly which commands within your script are causing issues or producing unexpected results.

By logging these details, you can trace the execution of individual cmdlets and functions. To activate PowerShell module logging, you need to adjust specific registry settings. The following commands enable logging for all modules, providing a comprehensive overview of module-related activity.

reg add "HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\Windows\PowerShell\ModuleLogging" /v "EnableModuleLogging" /t REG_DWORD /d 1 /f
reg add "HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\Windows\PowerShell\ModuleLogging\ModuleNames" /v "*" /t REG_SZ /d *

Enabling PowerShell Script Block Logging¶

PowerShell script block logging captures the full content of script blocks as they are executed. This includes scripts, functions, and commands, providing a literal record of the code that ran. This feature is particularly valuable for identifying obfuscated scripts, analyzing complex logic, or simply verifying the exact version of a script that was processed.

Enabling script block logging adds another layer of forensic detail to your troubleshooting efforts. The following registry modification will activate this logging, ensuring that the contents of executed script blocks are recorded in the appropriate event log.

reg add "HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\Windows\PowerShell\ScriptBlockLogging" /v "EnableScriptBlockLogging" /t REG_DWORD /d 1 /f

Interpreting Enhanced Logging Output¶

Once you have enabled the various logging mechanisms and reproduced the script failure, you will have a wealth of information at your disposal. Understanding where to look and what to expect in these logs is crucial for effective debugging. Each type of logging provides unique insights into different aspects of the script’s execution.

By combining the data from process creation audits, transcriptions, module logs, and script block logs, you can construct a detailed timeline of events. This integrated view allows for a holistic understanding of the script’s behavior, helping you to pinpoint the exact moment and cause of failure. The following sections explain how to interpret the output from each enhanced logging feature.

Process Creation Auditing Events¶

Process creation auditing records vital information in the Security event log. Specifically, Event ID 4688 signifies the creation of a new process, and it includes the critical Process Command Line. This detail is essential for verifying how your script was invoked and any arguments passed. Event ID 4689, on the other hand, logs process termination. By examining these events, you can confirm that your script process started as expected and track its lifecycle.

Transcription Files¶

PowerShell transcription creates detailed text files that capture the entire session. These files are typically found in the C:\Transcripts\<output-date> directory, which will be automatically created if it doesn’t already exist. For example, a transcription file might be named C:\transcripts\20201211\PowerShell_transcript.<vm-name>.a+BWp8CT.20201211034929.txt. These logs provide a faithful reproduction of all input and output during the script’s execution, making them invaluable for step-by-step analysis.

Module Logging Events¶

Module logging writes Event ID 4103 entries to the Microsoft-Windows-PowerShell/Operational event log. Each “4103” event contains comprehensive details about cmdlet execution, including the cmdlet name and its output. For instance, if you run Write-Host $Env:ComputerName, the top of Event ID 4103 will show information indicating that the command Write-Host was invoked, and the output, such as value="<vm-name>", reveals the name of your VM. This level of detail allows you to track individual command results within your script.

CommandInvocation(Write-Host): "Write-Host"
ParameterBinding(Write-Host): name="Object"; value="<vm-name>"

Script Block Logging Events¶

Script block logging also logs Event ID 4104 to the Microsoft-Windows-PowerShell/Operational event log. These “4104” events contain the actual contents of the script blocks that were executed. This is incredibly powerful for verifying the code that ran. For larger scripts, the content might exceed the maximum message size for a single event, resulting in the script being logged across multiple “4104” events.

Exporting Event Logs for Analysis¶

After enabling comprehensive logging and reproducing the script failure, the next step is to consolidate and analyze the collected event data. Manually sifting through thousands of event log entries can be tedious and time-consuming. To streamline this process, you can use a PowerShell script to export the relevant events to a single, easy-to-read comma-separated value (CSV) file.

The following script queries both the Security log and the Microsoft-Windows-PowerShell/Operational log for specific event IDs related to process creation and PowerShell execution, focusing on events generated by the Local System account. The script is configured to look back a specified number of hours (defaulting to the previous 24 hours, or 86,400,000 milliseconds). You can adjust the $hours variable to retrieve data for a different timeframe, for example, 3600000 for the most recent hour or 604800000 for the past week, enabling focused analysis.

$path = "PSEvents_$($env:COMPUTERNAME)_$(Get-Date ((Get-Date).ToUniversalTime()) -Format yyyyMMddHHmmss).csv"

$hours = 1 # Increase this to have it query more than just the last 1 hour
$now = Get-Date
$startTimeUtc = Get-Date ($now.AddHours(-$hours).ToUniversalTime()) -Format 'yyyy-MM-ddTHH:mm:ssZ'
$endTimeUtc = Get-Date ($now.ToUniversalTime()) -Format 'yyyy-MM-ddTHH:mm:ssZ'

$filterXML = @"
<QueryList>
    <Query Id="0" Path="Security">
        <Select Path="Security">
            Event
            [
                System
                [
                    (EventID = '4688' or EventID = '4689')
                    and
                    TimeCreated
                    [
                        @SystemTime >= '$startTimeUtc'
                        and
                        @SystemTime <= '$endTimeUtc'
                    ]
                ]
                and
                EventData
                [
                    Data[@Name="SubjectUserSid"] = "S-1-5-18"
                ]
            ]
        </Select>
    </Query>
    <Query Id="1" Path="Microsoft-Windows-PowerShell/Operational">
        <Select Path="Microsoft-Windows-PowerShell/Operational">
            Event
            [
                System
                [
                    (EventID ='4103' or EventID ='4104')
                    and
                    Security
                    [
                        @UserID ='S-1-5-18'
                    ]
                    and
                    TimeCreated
                    [
                        @SystemTime >= '$startTimeUtc'
                        and
                        @SystemTime <= '$endTimeUtc'
                    ]
                ]
            ]
        </Select>
    </Query>
</QueryList>
"@

$events = Get-WinEvent -FilterXml $filterXML | Sort-Object -Property RecordId
$events = $events | Select-Object -Property RecordId, `
    TimeCreated, Id, MachineName, LogName, TaskDisplayName, Message
$events | Export-Csv -Path $path -NoTypeInformation

Disabling PowerShell Script Execution Logging¶

After successfully troubleshooting your PowerShell scripts and resolving any failures, it is important to revert the advanced logging settings to their default states. Maintaining overly verbose logging can consume significant disk space and potentially impact VM performance, especially in production environments. Disabling these settings ensures that your VM returns to its normal operational configuration.

This section outlines the steps to undo the changes made to enable PowerShell script execution logging. It covers resetting event log sizes, turning off auditing and transcription, and cleaning up any generated log files. Follow these instructions carefully to restore your VM’s logging configuration efficiently.

1. Revert Event Log Maximum Sizes: If you previously increased the maximum size of the Security log or the Microsoft-Windows-PowerShell/Operational log, restore these values to their default maximum sizes. This action reduces the disk space allocated for these logs.

   wevtutil set-log "Security" /ms:20971520
   wevtutil set-log "Microsoft-Windows-PowerShell/Operational" /ms:15728640
   

2. Turn off Process Creation Auditing: Disable process creation auditing to stop recording detailed information about every new process on the system. This reduces the number of events written to the Security log.

   reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System\Audit" /v "ProcessCreationIncludeCmdLine_Enabled" /t REG_DWORD /d 0 /f
   

3. Turn off Transcription: Disable PowerShell transcription to prevent further recording of session input and output. This stops the creation of plaintext log files in your transcription directory. The output directory registry entry will also be removed.

   reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\PowerShell\Transcription" /v "EnableTranscripting" /t REG_DWORD /d 0 /f
   reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\PowerShell\Transcription" /v "EnableInvocationHeader" /t REG_DWORD /d 0 /f
   reg delete "HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\PowerShell\Transcription" /v "OutputDirectory"
   

4. Turn off Module Logging: Disable PowerShell module logging to stop tracking pipeline execution details for specific modules. This reduces the number of Event ID 4103 entries in the Operational log. The wildcard entry for module names should also be removed.

   reg add "HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\Windows\PowerShell\ModuleLogging" /v "EnableModuleLogging" /t REG_DWORD /d 0 /f
   reg delete "HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\PowerShell\ModuleLogging\ModuleNames" /v "*"
   

5. Turn off Script Block Logging: Disable PowerShell script block logging to discontinue capturing the full content of executed script blocks. This reduces the generation of Event ID 4104 entries.

   reg add "HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\Windows\PowerShell\ScriptBlockLogging" /v "EnableScriptBlockLogging" /t REG_DWORD /d 0 /f
   

6. Remove the Transcription Folder: Delete the directory where transcription files were stored. This cleans up any accumulated transcription logs, freeing up disk space.

   Remove-Item -Path 'C:\Transcripts' -Force -Recurse
   

7. Clear Event Logs: Finally, either back up and then clear the Security log and the Microsoft-Windows-PowerShell/Operational log, or clear them directly without a backup. This action ensures that the logs are clean for future operational events.

   To back up and clear the logs:

   wevtutil clear-log Security /bu:Security.evtx
   wevtutil clear-log Microsoft-Windows-PowerShell/Operational /bu:Microsoft-Windows-PowerShell_Operational.evtx
   

   Or, to clear the logs without backing them up:

   wevtutil clear-log Security
   wevtutil clear-log Microsoft-Windows-PowerShell/Operational
   

Testing Run Command and Custom Script Extension Logging on your VM¶

To validate that your logging configurations are working as expected, or to simulate a script failure for demonstration, you can use a test script. The Test-CustomScriptExtension.ps1 script is specifically designed to generate output and a non-zero exit code, which helps in observing how errors are captured by the enhanced logging.

Testing Run Command Logging on your VM¶

First, download the Test-CustomScriptExtension.ps1 test script to your current local directory. This script simulates a simple failure condition, making it ideal for verifying your logging setup. Once downloaded, you can run this script on your Azure VM using the Invoke-AzVMRunCommand cmdlet. Remember to replace the placeholders for your resource group name and VM name with the actual properties of your virtual machine. This execution will allow you to observe how Run Command handles the script output and errors under your current logging configurations.

$scriptUri = 'https://raw.githubusercontent.com/Azure/azure-support-scripts/blob/users/GitHubPolicyService/6294a303-e34d-4bad-b6cd-5ed54245f020/Images_Extensions/PowerShell/Test-CustomScriptExtension.ps1'
$localFileLocation = "$PWD\Test-CustomScriptExtension.ps1"
(New-Object System.Net.WebClient).DownloadFile($scriptUri, $localFileLocation)

$commandSettings = @{
    ResourceGroupName = '<resource-group-name>'
    VMName = '<vm-name>'
    CommandId = 'RunPowerShellScript'
    ScriptPath = $localFileLocation
}
Invoke-AzVMRunCommand @commandSettings

Testing Custom Script Extension Logging on your VM¶

Similarly, you can run the Test-CustomScriptExtension.ps1 script on your VM using the Set-AzVMCustomScriptExtension cmdlet. This method tests the logging behavior when scripts are deployed via the Custom Script Extension. As before, ensure you replace the placeholders for the resource group name, VM name, and location with your VM’s specific details. The ForceRerun parameter, set to (Get-Date).Ticks, ensures that the extension attempts to run the script every time, even if the configuration hasn’t changed, which is useful for repeated testing.

$commandSettings = @{
    ResourceGroupName = '<resource-group-name>'
    VMName = '<vm-name>'
    Name = 'CustomScriptExtension'
    FileUri = 'https://raw.githubusercontent.com/Azure/azure-support-scripts/blob/users/GitHubPolicyService/6294a303-e34d-4bad-b6cd-5ed54245f020/Images_Extensions/PowerShell/Test-CustomScriptExtension.ps1'
    Run = 'Test-CustomScriptExtension.ps1'
    Location = '<azure-region-name-or-code>'
    ForceRerun = (Get-Date).Ticks
}
Set-AzVMCustomScriptExtension @commandSettings

Alternatively, you can deploy the Custom Script Extension using the more generic Set-AzVMExtension cmdlet. This method offers greater flexibility for various extension types. When using Set-AzVMExtension, you must specify ExtensionType as CustomScriptExtension and configure the script URI and command to execute within the Settings parameter. Again, replace the placeholders with your VM’s actual resource group name, VM name, and location to accurately target your virtual machine for testing.

$publicConfigSettings = @{
    'fileUris' = @('https://raw.githubusercontent.com/Azure/azure-support-scripts/blob/users/GitHubPolicyService/6294a303-e34d-4bad-b6cd-5ed54245f020/Images_Extensions/PowerShell/Test-CustomScriptExtension.ps1')
    'commandToExecute' = 'powershell -File Test-CustomScriptExtension.ps1 -ExecutionPolicy Unrestricted'
}
$commandSettings = @{
    Publisher = 'Microsoft.Compute'
    ExtensionType = 'CustomScriptExtension'
    Settings = $publicConfigSettings
    ResourceGroupName = '<resource-group-name>'
    VMName = '<vm-name>'
    Name = 'CustomScriptExtension'
    TypeHandlerVersion = '1.10'
    Location = '<azure-region-name-or-code>'
}
Set-AzVMExtension @commandSettings

Common Troubleshooting Scenarios and Solutions¶

During the deployment and execution of PowerShell scripts via Azure’s Custom Script Extension, you might encounter several common errors. Understanding these errors and knowing how to address them is crucial for efficient troubleshooting. This section details some frequent issues and provides practical solutions to help you overcome them.

Common Errors in the Custom Script Extension Test Script¶

Using the Test-CustomScriptExtension.ps1 script can help simulate and understand typical failure scenarios. Here are some errors you might encounter when working with the Custom Script Extension:

• Nonzero Exit Code:
  After executing the Test-CustomScriptExtension.ps1 script, you should anticipate receiving an error message indicating a non-zero exit code. This behavior is by design, as the test script intentionally runs the Exit 2 command. The Custom Script Extension is programmed to report a failure whenever a script returns any exit code other than zero. In this specific instance, 2 represents the non-zero exit code. This particular example serves as a clear demonstration of how a failure is registered within the additional PowerShell logging you have enabled, providing valuable insight into error reporting mechanisms.

  Long running operation failed with status 'Failed'.
  Additional Info: VM has reported a failure when processing extension 'CustomScriptExtension'.
  Error message: "Command execution finished, but failed because it returned a non-zero exit code of: '2'"
  

• Change is in Conflict:
  This error typically arises when an Azure VM already has the Custom Script Extension installed under a specific resource name, such as Microsoft.Compute.CustomScriptExtension, but your current execution attempt specifies a different resource name, like CustomScriptExtension. Azure prevents updating an extension if the new request conflicts with an existing one’s handler version or auto-upgrade settings under the same handler type but a different instance name. This conflict occurs because you’re essentially trying to deploy a new extension instance with a different identifier while an existing one of the same type is already present and managed.

  Cannot update handlerVersion or autoUpgradeMinorVersion for VM extension 'CustomScriptExtension'.
  Change is in conflict with other extensions under handler 'Microsoft.Compute.CustomScriptExtension', with typeHandler version '1.10' and autoUpgradeMinorVersion 'True'.
  ErrorCode: OperationNotAllowed
  ErrorMessage: Cannot update handlerVersion or autoUpgradeMinorVersion for VM extension 'CustomScriptExtension'.
  Change is in conflict with other extensions under handler 'Microsoft.Compute.CustomScriptExtension', with typeHandler version '1.10' and autoUpgradeMinorVersion 'True'.
  ErrorTarget:
  StatusCode: 409
  ReasonPhrase: Conflict
  

  You have flexibility in choosing a resource name for your Custom Script Extension deployment. However, if the Custom Script Extension is already present on the VM, you must either consistently use the same name for all subsequent deployments targeting that extension or first remove the existing extension resource before attempting to deploy a new one with a different name. This ensures that Azure can correctly manage and update the extension without conflicts.

• Invalid File URI Configuration:
  This error indicates a misconfiguration regarding where the script’s file URIs are specified within the Custom Script Extension settings. The Custom Script Extension allows you to provide file URIs in either the protected settings or the public settings, but not both simultaneously. This error occurs if the extension was originally installed with file URIs defined in one configuration section (e.g., protected) and a subsequent update or re-deployment attempt specifies them in the other section (e.g., public), or if both sections contain URI definitions.

  Long running operation failed with status 'Failed'.
  Additional Info: VM has reported a failure when processing extension 'CustomScriptExtension'.
  Error message: "Invalid Configuration - FileUris is present in both the protected and public configuration sections; it must be specified in only one section."
  

Solutions to Common Errors in the Custom Script Extension Test Script¶

Effectively resolving common errors when using the Custom Script Extension often involves understanding the extension’s state on your VM. Here are targeted solutions for the issues discussed above:

• To fix the “Change is in conflict” error:
  When faced with the “Change is in conflict” error, the primary solution involves rerunning Set-AzVMCustomScriptExtension or Set-AzVMExtension while ensuring that the -Name parameter correctly matches the resource name of the Custom Script Extension that is already installed on your VM. The error message itself often provides a strong hint for this name; for example, if the sample error indicated a conflict with handler 'Microsoft.Compute.CustomScriptExtension', then Microsoft.Compute.CustomScriptExtension is the name you should use for your -Name parameter when attempting to modify or update the extension.

  You can also programmatically verify the correct resource name of an existing Custom Script Extension by using the Get-AzVM cmdlet. This command retrieves the properties of your Azure VM, including details about its installed extensions. By filtering for extensions of type Microsoft.Compute.CustomScriptExtension, you can accurately identify the name currently assigned to it, ensuring your subsequent commands target the correct resource.

  $status = Get-AzVM -ResourceGroupName <resource-group> -Name <vm-name> -Status
  $status.Extensions | `
  Where-Object Type -EQ 'Microsoft.Compute.CustomScriptExtension' | `
  Select-Object Name
  

• To mitigate the “Change is in conflict” error and the “FileUris” error:
  A robust way to resolve both the “Change is in conflict” error and the “Invalid file URI configuration” error is to remove the existing Custom Script Extension entirely before attempting a new deployment or re-configuration. This clears any previous conflicting states or misconfigurations, allowing you to start fresh. You can achieve this by using the Remove-AzVMCustomScriptExtension cmdlet.

  $params = @{
      ResourceGroupName = '<resourceGroupName>'
      VMName = '<vm-name>'
      Name = '<extension-resource-name>' # Use the correct resource name found via Get-AzVM
      Force = $True
  }
  Remove-AzVMCustomScriptExtension @params
  

  It is important to provide the correct extension resource name to the Remove-AzVMCustomScriptExtension cmdlet. If you specify an incorrect resource name, the command will execute without error but return a StatusCode of NoContent, indicating that no matching extension was found to remove.

  RequestId IsSuccessStatusCode StatusCode ReasonPhrase
  --------- ------------------- ---------- ------------
                           True  NoContent No Content
  

  Conversely, if you provide the correct resource name for the extension, the Remove-AzVMCustomScriptExtension cmdlet will successfully delete it and return a StatusCode of OK, confirming that the operation was completed as intended.

  RequestId IsSuccessStatusCode StatusCode ReasonPhrase
  --------- ------------------- ---------- ------------
                           True         OK OK
  

Conclusion¶

Troubleshooting PowerShell scripts in Azure VMs, especially those deployed via the Custom Script Extension or Run Command, can be a complex but manageable task. By systematically approaching the problem—starting with manual verification, understanding the Local System account, and leveraging advanced logging—you can gain deep insights into script execution failures. The enhanced logging features, including process creation auditing, transcription, module logging, and script block logging, provide an invaluable forensic trail to pinpoint root causes that might otherwise remain hidden.

Remember to revert logging configurations once troubleshooting is complete to maintain optimal VM performance and security. We hope this comprehensive guide empowers you to debug your Azure PowerShell automation with confidence and efficiency.

Do you have any experiences with these troubleshooting techniques or specific script failures in Azure VMs? Share your thoughts, questions, or alternative strategies in the comments below! Your insights are valuable to the community.