Solve Azure App Insights Snapshot Debugger Issues: A Troubleshooting Guide

Table of Contents

If you have enabled Application Insights Snapshot Debugger for your application and are encountering difficulties in viewing snapshots for exceptions, this guide is designed to assist you in troubleshooting common issues. Snapshot generation can fail due to a variety of reasons. A recommended first step is to utilize the Snapshot Health Check tool, which can automatically identify and diagnose several frequently encountered problems. This proactive approach can often streamline the troubleshooting process and quickly pinpoint the root cause of the issue.

Unsupported Scenarios¶

There are specific scenarios where the Snapshot Collector is not designed to function and is therefore unsupported. Understanding these limitations is crucial to avoid misdiagnoses and wasted troubleshooting efforts. It’s important to review these scenarios to ensure your application environment is compatible with Snapshot Debugger. Using Snapshot Collector in unsupported scenarios can lead to unexpected behavior or a complete lack of snapshot availability.

| Scenario REPLACED WITH:

If you’ve enabled Application Insights Snapshot Debugger for your application, but are not seeing snapshots for exceptions, this guide will help you troubleshoot common issues. Snapshot generation failures can stem from a multitude of causes. A good starting point is to run the Snapshot Health Check, a tool designed to identify and diagnose common problems automatically.

Unsupported Scenarios¶

It’s important to understand that Snapshot Collector has limitations and is not supported in all scenarios. Using it in these unsupported scenarios can lead to unexpected behavior and the absence of snapshots. Review the table below to ensure your application’s configuration is compatible with Snapshot Debugger.

| Scenario Side Effects | Side Effects | Recommendation

Snapshot Debugger, while beneficial, can introduce overhead in certain situations. Understanding these overhead scenarios is crucial for diagnosing potential performance issues and optimizing performance. The default settings are designed to minimize the impact on production applications, rate limiting snapshot capture to reduce resource consumption.

Exception Overhead¶

When an exception occurs within your application, the Snapshot Debugger initiates a series of actions that can introduce a small overhead. These actions include:

• Signature Creation: Generating a unique signature for the exception type to categorize and track problem patterns. This process involves minimal CPU and memory usage.
• Snapshot Decision: Determining whether a snapshot should be created for the specific exception based on configured rules and limits. This decision-making process also adds a slight overhead.
• Deoptimization (If Enabled): If deoptimization is enabled, the method that threw the exception needs to be re-JITted (Just-In-Time compiled) to allow for snapshot capture. This re-JITting incurs overhead the next time the method is executed. The overhead can range from 1 ms to 100 ms of CPU time, depending on the method’s complexity.

Snapshot Creation Overhead¶

If the exception handler decides to create a snapshot, further overhead is introduced during the snapshot creation and upload process:

• Process Snapshot Creation: Capturing a snapshot of the application process typically takes around half a second. During this brief period, the thread that triggered the exception is paused to ensure data consistency. However, it’s important to note that other threads within the application remain unaffected and are not blocked. The typical duration for snapshot creation is characterized by the following percentiles: P50 = 0.3 seconds, P90 = 1.2 seconds, and P95 = 1.9 seconds.

• Minidump Conversion and Upload: The process snapshot is then converted into a minidump file, a smaller representation of the process memory, and subsequently uploaded to Application Insights. This operation takes a longer duration, typically spanning several minutes. It’s important to understand the breakdown of this process:

  • Conversion to Minidump: P50 = 63 seconds, P90 = 187 seconds, P95 = 275 seconds.
  • Upload to Application Insights: P50 = 31 seconds, P90 = 75 seconds, P95 = 98 seconds.

  This entire conversion and upload process is handled by a separate process called Snapshot Uploader. To minimize application impact, the Snapshot Uploader runs with below-normal CPU priority and utilizes low-priority I/O operations. The minidump file is initially written to disk, requiring disk space roughly equivalent to the working set of the original process. The process of writing the minidump to disk can result in page faults as memory data is read. During upload, the minidump is compressed, which consumes CPU and memory within the Snapshot Uploader process. The overall overhead in terms of CPU, memory, and disk I/O is directly proportional to the size of the process snapshot. The Snapshot Uploader processes snapshots sequentially, one at a time.

TrackException Call Overhead¶

When the `TrackException` method is called within your application code, the Snapshot Debugger performs a quick check to determine if the exception is new or if a snapshot has already been created for it. This check introduces a minimal CPU overhead, ensuring efficient snapshot management.

It’s crucial to understand these overhead scenarios to effectively monitor and manage the performance impact of Snapshot Debugger in production environments. While designed to be minimally invasive, awareness of these potential impacts allows for informed decisions regarding configuration and resource allocation.

Preview Versions of .NET Core¶

If you are utilizing a preview version of .NET Core, or if your application references the Application Insights SDK, either directly or indirectly through dependent assemblies, specific configuration steps are required to ensure Snapshot Debugger functions correctly. For these environments, it is essential to follow the dedicated instructions provided for enabling Snapshot Debugger in other environments. These instructions typically involve manual configuration steps to ensure compatibility and proper operation with preview .NET Core versions or specific SDK configurations.

Check the Diagnostic Services Site Extension’s Status Page¶

When Snapshot Debugger is enabled through the Azure portal’s Application Insights pane, it leverages the Diagnostic Services site extension for deployment and management. This extension simplifies the process of enabling and configuring Snapshot Debugger within Azure App Service environments. To troubleshoot potential issues, the Status Page of this extension provides valuable insights into the health and installation state of the Snapshot Debugger components.

You can access the Status Page by navigating to a specific URL structure: https://{site-name}.scm.azurewebsites.net/DiagnosticServices. Replace {site-name} with the name of your Azure App Service web app.

Note: The domain of the Status Page link may vary depending on the specific Azure cloud environment you are using (e.g., Azure Government, Azure China).

This URL leads to the same Kudu management site used for App Service diagnostics and administration. The Status Page displays the installation status of both the .NET Profiler and Snapshot Collector agents, which are essential components for Snapshot Debugger functionality. If any unexpected errors have occurred during installation or operation, the Status Page will often provide guidance on how to resolve them, making it a valuable resource for self-diagnosis and troubleshooting.

Accessing the Status Page via Kudu:

1. Navigate to your App Service application within the Azure portal.
2. Locate and select Advanced Tools (or search for “Kudu” in the search bar).
3. Click Go to open the Kudu management site in a new browser tab.
4. Once on the Kudu site, append /DiagnosticServices to the end of the URL in your browser’s address bar and press Enter. The final URL should resemble: https://<kudu-url>/DiagnosticServices.

This will directly access the Diagnostic Services Status Page for your App Service application, providing a quick overview of the Snapshot Debugger’s health and potential issues.

Upgrade to the Latest Version of the NuGet Package¶

Maintaining the most recent version of the Microsoft.ApplicationInsights.SnapshotCollector NuGet package is crucial for optimal performance, bug fixes, and access to the latest features of Snapshot Debugger. The upgrade process depends on how Snapshot Debugger was initially enabled for your application.

• Portal Enabled: If Snapshot Debugger was activated through the Application Insights pane in the Azure portal, your application should already be running the latest compatible NuGet package. Azure platform updates typically ensure that the most recent and stable versions are deployed automatically in this scenario.

• NuGet Package Inclusion: If you manually enabled Snapshot Debugger by directly including the Microsoft.ApplicationInsights.SnapshotCollector NuGet package in your project, you must use Visual Studio’s NuGet Package Manager to verify and update to the latest version.

Using Visual Studio’s NuGet Package Manager:

1. Open your solution in Visual Studio.
2. Go to Tools > NuGet Package Manager > Manage NuGet Packages for Solution…
3. Select the Updates tab.
4. Search for Microsoft.ApplicationInsights.SnapshotCollector.
5. If an update is available, select the package and click Update.

For detailed information about the latest updates, bug fixes, and new features, consult the official release notes available in the ApplicationInsights-SnapshotCollector GitHub repository. Regularly reviewing these release notes ensures you are aware of any important changes or improvements and helps keep your Snapshot Debugger implementation up-to-date.

Check the Uploader Logs¶

After a snapshot is successfully created, a minidump file (.dmp) is generated on disk. A separate background process, the Snapshot Uploader, is responsible for processing this minidump file. The uploader performs several key tasks: it reads the minidump, identifies and collects any associated Program Database (PDB) files necessary for symbol resolution, and then uploads both the minidump and PDBs to Application Insights Snapshot Debugger storage. Once the upload is complete and successful, the minidump file is automatically deleted from the local disk to conserve space.

Detailed log files for the uploader process are maintained on disk, providing valuable insights into the uploader’s operations and any potential issues encountered during snapshot processing and upload. The location of these log files depends on the application’s hosting environment.

For App Service Environments:

In Azure App Service environments, uploader logs are typically located in the D:\Home\LogFiles directory. You can access these logs using the Kudu management site for your App Service.

Accessing Uploader Logs via Kudu in App Service:

1. Open your App Service application in the Azure portal.
2. Select Advanced Tools or search for Kudu.
3. Click Go.
4. In the Kudu interface, navigate to the Debug console dropdown menu and select CMD.
5. In the command prompt, navigate to the LogFiles directory.

You should find one or more files with names starting with Uploader_ or SnapshotUploader_ and a .log extension. These files contain the detailed logs. You can download these log files using the download icon or open them directly in the browser for review. The filename often includes a unique suffix that identifies the specific App Service instance. If your App Service is running on multiple machines, there will be separate log files for each instance.

Log File Content:

The uploader log files record various events, including the detection of new minidump files, the progress of minidump creation, upload status, PDB file processing, and cleanup operations. A successful snapshot and upload sequence in the log would resemble the example provided in the original documentation, showing details like request IDs, timestamps, file paths, and upload sizes. Crucially, the connection string mentioned in the logs should match the connection string configured for your Application Insights resource to ensure the snapshots are associated with the correct application. The request ID mentioned in the logs can be used to correlate the snapshot with the corresponding exception record in Application Insights Analytics for further investigation.

For Non-App Service Environments:

For applications hosted outside of App Service environments (e.g., on virtual machines, on-premises servers), the uploader logs are stored in the same directory as the minidump files. This directory is typically located at %TEMP%\Dumps\<string>, where <string> is your Application Insights connection string.

Analyzing these uploader logs is a crucial step in troubleshooting snapshot debugger issues. They can provide detailed error messages, warnings, and status updates that help pinpoint problems during the snapshot capture and upload process.

Troubleshooting Cloud Services¶

For applications running in Azure Cloud Services, a common issue that can lead to lost snapshots is insufficient disk space in the default temporary folder. Cloud Services environments may have limited temporary storage, and if the minidump files cannot be written due to lack of space, snapshots will be lost. The required disk space depends on the working set size of your application and the anticipated number of concurrent snapshots being generated.

Disk Space Considerations:

For a typical 32-bit ASP.NET web role, the working set size can range from 200 MB to 500 MB. It’s recommended to account for at least two concurrent snapshots to handle potential spikes in exceptions. Therefore, if your application’s total working set is around 1 GB, ensure that you have at least 2 GB of dedicated disk space available for storing snapshots.

Configuring Dedicated Local Resource for Snapshots:

To address potential disk space limitations in Cloud Services, it is best practice to configure a dedicated local resource specifically for snapshot storage. This involves modifying your Cloud Service definition file (.csdef) and updating your role’s startup code.

Steps to Configure a Local Resource:

1. Modify Cloud Service Definition (.csdef): Add a new local resource definition to your .csdef file. The example below defines a local resource named SnapshotStore with a size of 5 GB.

   <LocalResources>
       <LocalStorage name="SnapshotStore" cleanOnRoleRecycle="false" sizeInMB="5120" />
   </LocalResources>
   

2. Modify Role Startup Code: Update your role’s startup code to set an environment variable that points to the SnapshotStore local resource.

   • Worker Roles: Add the following code to your worker role’s OnStart method:

     public override bool OnStart()
     {
         Environment.SetEnvironmentVariable("SNAPSHOTSTORE", RoleEnvironment.GetLocalResource("SnapshotStore").RootPath);
         return base.OnStart();
     }
     

   • Web Roles (ASP.NET): Add the code to your web application’s Application_Start method within the Global.asax.cs file:

     using Microsoft.WindowsAzure.ServiceRuntime;
     using System;
     namespace MyWebRoleApp
     {
         public class MyMvcApplication : System.Web.HttpApplication
         {
             protected void Application_Start()
             {
                 Environment.SetEnvironmentVariable("SNAPSHOTSTORE", RoleEnvironment.GetLocalResource("SnapshotStore").RootPath);
                 // TODO: The rest of your application startup code
             }
         }
     }
     

3. Update ApplicationInsights.config: Modify your role’s ApplicationInsights.config file to override the default temporary folder location used by SnapshotCollector. Add the following within the <TelemetryProcessors> section:

   <TelemetryProcessors>
       <Add Type="Microsoft.ApplicationInsights.SnapshotCollector.SnapshotCollectorTelemetryProcessor, Microsoft.ApplicationInsights.SnapshotCollector">
           <!-- Use the SnapshotStore local resource for snapshots -->
           <TempFolder>%SNAPSHOTSTORE%</TempFolder>
           <!-- Other SnapshotCollector configuration options -->
       </Add>
   </TelemetryProcessors>
   

By implementing these steps, you ensure that Snapshot Debugger utilizes a dedicated and sufficiently sized local resource for storing minidump files in your Cloud Services environment, significantly reducing the risk of lost snapshots due to disk space constraints.

Overriding the Shadow Copy Folder¶

When the Snapshot Collector service starts, it needs to find a suitable location on disk to operate and execute the Snapshot Uploader process. This location is referred to as the Shadow Copy folder. The Snapshot Collector automatically searches through a predefined list of well-known system folders to find a folder that meets the necessary criteria, including write and execute permissions. The following environment variables are checked in order:

• Fabric_Folder_App_Temp
• LOCALAPPDATA
• APPDATA
• TEMP

Troubleshooting Shadow Copy Folder Issues:

• “Couldn’t find a suitable shadow copy folder” Error: If the Snapshot Collector cannot identify a suitable folder from the list above, it will report this error. This typically indicates a restrictive environment where standard temporary folders are not accessible or lack the required permissions.

• ShadowCopyFailed Error: This error occurs if the Snapshot Collector successfully finds a folder but fails to copy the Snapshot Uploader binaries to it. This is usually due to insufficient write permissions in the selected shadow copy folder.

• UploaderCannotStartFromShadowCopy Error: This error indicates that while the uploader binaries were successfully copied to the shadow copy folder, the Snapshot Collector is unable to launch the uploader process from that location. The error message often includes System.UnauthorizedAccessException, which typically means the application is running under a security context with restricted permissions. The account might have write access to the folder but lacks execute permissions, preventing the uploader process from starting.

These shadow copy related errors often occur during application startup, and are frequently followed by an ExceptionDuringConnect error, indicating that the uploader process failed to start or connect.

Manually Specifying the Shadow Copy Folder:

To resolve these shadow copy folder issues, you can explicitly specify a custom shadow copy folder location using the ShadowCopyFolder configuration option. This allows you to direct the Snapshot Collector to use a folder that you know has the necessary permissions.

Configuration Examples:

• ApplicationInsights.config (for .NET Framework applications):

  <TelemetryProcessors>
      <Add Type="Microsoft.ApplicationInsights.SnapshotCollector.SnapshotCollectorTelemetryProcessor, Microsoft.ApplicationInsights.SnapshotCollector">
          <!-- Override the default shadow copy folder. -->
          <ShadowCopyFolder>D:\SnapshotUploader</ShadowCopyFolder>
          <!-- Other SnapshotCollector configuration options -->
      </Add>
  </TelemetryProcessors>
  

• appsettings.json (for .NET Core applications):

  {
      "ApplicationInsights": {
          "ConnectionString": "<your connection string>"
      },
      "SnapshotCollectorConfiguration": {
          "ShadowCopyFolder": "D:\\\\SnapshotUploader"
      }
  }
  

By manually setting the ShadowCopyFolder, you can bypass the automatic folder selection process and ensure that the Snapshot Collector uses a directory with appropriate permissions, resolving common startup errors related to shadow copying. Ensure the specified folder exists and the application’s process has both read and execute permissions on it.

Use Application Insights Search to Find Exceptions with Snapshots¶

When a snapshot is successfully created, the exception that triggered the snapshot is tagged with a unique snapshot ID. This ID is then included as a custom property (ai.snapshot.id) when the exception telemetry is reported to Application Insights. Application Insights Search provides a powerful way to locate exceptions that have associated snapshots using this custom property.

Steps to Search for Snapshots in Application Insights:

1. Navigate to your Application Insights resource within the Azure portal.
2. Select Search from the left-hand navigation menu.
3. In the Search text box, type ai.snapshot.id and press Enter or click the search icon.

This search query will return all telemetry records within the selected time range that include the ai.snapshot.id custom property, effectively filtering for exceptions that have snapshots associated with them.

Troubleshooting Search Results:

• No Results: If the search returns no results, it indicates that no snapshots have been reported to Application Insights within the currently selected time range. This could mean that no exceptions with snapshots have occurred, or there might be an issue preventing snapshots from being captured or uploaded.

• Searching for a Specific Snapshot ID: If you have a specific snapshot ID from the Uploader logs, you can directly search for that ID in the Search box. This can help you pinpoint the corresponding exception record in Application Insights.

• Snapshot Uploaded but Exception Not Found: If you are certain a snapshot was uploaded (based on Uploader logs) but cannot find the corresponding exception record in Search, follow these steps:

  1. Verify Connection String: Double-check that you are looking at the correct Application Insights resource by carefully verifying the connection string used by your application. Incorrect connection string configuration is a common cause of telemetry data appearing in the wrong Application Insights resource.
  2. Adjust Time Range Filter: Using the timestamp information from the Uploader log file, adjust the Time Range filter in Application Insights Search to encompass the time period when the snapshot was uploaded. The default time range may not include the relevant timeframe.

If, after these steps, you still cannot locate an exception record with the specific snapshot ID, it is possible that the application crashed immediately after capturing the snapshot but before successfully reporting the exception telemetry to Application Insights. In such cases, examine the App Service logs under the “Diagnose and solve problems” section in the Azure portal. Look for any unexpected application restarts or unhandled exceptions that might have occurred around the time the snapshot was expected. These logs can provide valuable clues about application crashes that might prevent exception telemetry from being reliably reported.

Edit Network Proxy or Firewall Rules¶

If your application environment relies on a network proxy or firewall to control internet traffic, you may need to configure specific rules to allow communication with the Snapshot Debugger service. Snapshot Debugger relies on outbound network connections to Azure services for uploading minidumps and related data. If these connections are blocked by proxy or firewall rules, snapshot functionality will be impaired.

Service Tags for Azure Monitor:

The IP addresses used by Application Insights Snapshot Debugger are included within the broader Azure Monitor service tag. Service tags simplify network configuration in Azure by grouping IP addresses for Azure services under a single, named tag. Instead of managing individual IP address ranges, you can use service tags in your firewall or Network Security Group (NSG) rules.

Configuring Firewall/Proxy Rules:

To ensure proper communication, you need to allow outbound traffic to the Azure Monitor service tag. The specific configuration steps will depend on your firewall or proxy solution. Consult the documentation for your specific firewall or proxy for instructions on how to create rules based on Azure service tags.

For more detailed information on Azure service tags and how to use them in network configurations, refer to the Service Tags documentation for Azure. This documentation provides comprehensive information about service tags, including how to find the appropriate tags for Azure services and how to integrate them into your network security rules.

By correctly configuring your network proxy or firewall rules to allow traffic to the Azure Monitor service tag, you ensure that Snapshot Debugger can successfully communicate with the necessary Azure services and function as expected.

Are there any billing costs when using snapshots?¶

A significant advantage of using Application Insights Snapshot Debugger is that there are no additional billing charges specifically for the Snapshot Debugger feature itself. Your Azure subscription is not charged extra for enabling or using Snapshot Debugger.

The snapshot files that are collected during exception events are stored separately from the standard telemetry data collected by the Application Insights SDKs. Crucially, there are no charges associated with the ingestion or storage of these snapshot files. This means you can leverage the powerful diagnostic capabilities of Snapshot Debugger without incurring any incremental costs beyond your existing Application Insights usage.

This cost-effectiveness makes Snapshot Debugger a highly valuable tool for production debugging and issue resolution, allowing you to gain deep insights into exceptions without impacting your Azure spending.

---

If you’ve followed these troubleshooting steps and are still experiencing issues with Azure App Insights Snapshot Debugger, consider reaching out to Azure support for further assistance. Your feedback and experiences are valuable in improving this tool. Please feel free to leave comments below sharing your troubleshooting journey or any tips that have helped you resolve Snapshot Debugger problems!