Mastering Cloud Services: Deep Dive into Windows Azure Compute Diagnostic Data

Table of Contents

Troubleshooting issues in any computing environment necessitates a thorough understanding of the diagnostic data available. Without access to logs and other key information sources, diagnosing problems can devolve into guesswork or a time-consuming trial-and-error process. The ability to effectively locate and interpret diagnostic data significantly increases the likelihood of resolving issues efficiently, even those outside one’s primary area of expertise.

This article provides an in-depth look at the diagnostic data accessible within Windows Azure platform as a service (PaaS) compute environments. It specifically focuses on data sources available directly within a Windows PaaS virtual machine (VM), outlining how this critical information can be gathered to aid in troubleshooting common issues. The sections below detail the most frequently utilized data sources when diagnosing problems within a Windows PaaS VM, generally ordered by their common importance and frequency of use in troubleshooting scenarios.

Windows Azure Event Logs¶

The Windows Azure event logs are a fundamental source of diagnostic output generated by the Azure runtime itself. These logs capture crucial events related to the lifecycle and operation of your cloud service roles. Information recorded includes significant events such as the initiation and termination of role instances, the execution status of startup tasks defined for the role, the start and stop events for the OnStart method within your role code, the beginning of the OnRun method execution, and events related to application crashes or role recycles.

Accessing these logs is straightforward using the standard Windows Event Viewer tool. To view them, simply search for “Event Viewer” on the Start menu and open the application. Within the Event Viewer navigation pane, expand the “Applications and Services Logs” section, and then select “Windows Azure.”

These logs are invaluable for pinpointing the root cause of many common problems that prevent Azure roles from starting correctly or maintaining a healthy state. Issues like failures during the execution of startup tasks, or unhandled exceptions causing crashes within your OnStart or OnRun methods, are frequently logged here. The Event Viewer interface captures crash details from the Azure runtime host processes (like WaIISHost.exe for web roles or WaWorkerHost.exe for worker roles) that execute your primary role entry point code (e.g., WebRole.cs or WorkerRole.cs), often providing critical call stack information that helps developers identify the failing code segment. Understanding these logs is often the first step in resolving role startup or stability problems.

Application Event Logs¶

Standard Windows application event logs remain a vital resource for troubleshooting within an Azure PaaS environment, just as they are on traditional on-premises servers. These logs capture events and errors reported by applications running within the VM. For web roles hosting ASP.NET applications, it’s common to find errors originating from the w3wp.exe process (the IIS worker process) logged here when application-level issues occur, such as unhandled exceptions, application pool problems, or .NET runtime errors.

To access and review the application event logs, open the Event Viewer by searching for it on the Start menu. In the navigation pane, expand the “Windows Logs” section. Within this section, select “Application” to view the log entries.

Analyzing the application event logs provides insights into failures occurring within your deployed application code or its dependencies, distinct from issues related to the Azure runtime or host environment. While the Windows Azure event logs report on the health and state of the role instance itself, the Application logs focus on the health and behavior of the specific applications configured to run within that role. Correlating events between the Windows Azure and Application logs can often provide a clearer picture of whether a problem stems from the hosting environment or the application logic.

App Agent Runtime Logs¶

The application agent runtime log, typically located at C:\Logs\AppAgentRuntime.log, is generated by the WindowsAzureGuestAgent.exe executable. This file serves as a detailed chronicle of events managed directly by the guest agent running inside the VM. It records information about various operations and state changes impacting the VM and the role instance it hosts.

Key event categories captured in this log include firewall configuration updates, transitions in the role’s state (e.g., starting, ready, busy), role recycles and VM reboots, changes in the reported health status of the instance, role stops and starts, and certificate configuration activities.

This log is particularly valuable for gaining a chronological overview of major events affecting a role instance over time. Unlike heartbeat logs, it focuses on significant state changes and operational events rather than frequent status pings, making it easier to track the sequence of events leading up to a problem. If the guest agent encounters difficulties in correctly starting or configuring the role, such as issues preventing directory cleanup due to locked files or problems applying network settings, details of these failures are typically recorded in the App Agent Runtime log, offering crucial clues for diagnosis.

App Agent Heartbeat Logs¶

Located at C:\Logs\WaAppAgent.log, the App Agent Heartbeat log is also produced by the WindowsAzureGuestAgent.exe. Its primary function is to record the status of health probes sent by the guest agent to the host bootstrapper. These probes are essential for the guest agent to report the health and status of the role instance (such as Ready, Busy, or Stopped) back to the Azure fabric controller.

The health status reported in this log directly corresponds to the status displayed for your role instance in the Azure management portal. This log is highly useful for determining the real-time or historical state of a role within the VM. If you receive a report that your application experienced an outage during a specific time window, reviewing the heartbeat log for that period can confirm the reported state of the role instance (e.g., if it was marked as Busy or transitioned through different states) and help correlate the application issue with the VM’s perceived health by the fabric. It provides a clear timeline of the guest agent’s perspective on the role’s readiness.

Host Bootstrapper Logs¶

The host bootstrapper log can be found at C:\Resources\WaHostBootstrapper.log. This log file captures entries related to the host bootstrapper process, which is responsible for initializing and starting other components of the Azure runtime and your role. It logs information about the execution of startup tasks, including tasks configured by built-in Azure plugins such as Caching or Remote Desktop Protocol (RDP). Furthermore, this log records the results of health probes directed at the host process that executes your role entry point code (e.g., WaIISHost.exe for web roles).

A new log file is typically generated each time the host bootstrapper process restarts. This occurs whenever your role instance is recycled due to events like application crashes, explicit recycling requests, VM reboots, or service upgrades. The creation of new log files upon restart makes the host bootstrapper log particularly useful for quickly identifying how often or when a role instance has been recycled. Analyzing the sequence of log files and their timestamps can provide valuable insights into the stability and frequency of restarts for your role, helping to determine if excessive recycling is contributing to availability issues.

Internet Information Services Logs¶

Internet Information Services (IIS) logs are standard diagnostic files widely used for troubleshooting web server issues, both on-premises and within Azure Web Roles. In Azure PaaS, these logs are located within the diagnostic store directory structure, typically at C:\Resources\Directory\<DeploymentID>.<RoleName>.DiagnosticStore\LogFiles\Web. These logs provide detailed information about HTTP requests processed by IIS, including the requesting IP address, request timestamp, HTTP method, requested URL, HTTP status code returned, and the time taken to process the request.

Despite being a standard troubleshooting tool, IIS logs are sometimes overlooked in cloud environments, especially when diagnosing perceived outages. While it’s common to suspect the cloud infrastructure when a website becomes unavailable, the IIS logs often contain the definitive evidence of what was happening at the application level. For instance, they might show a sudden increase in response times leading up to the outage, or a flood of requests resulting in non-success HTTP status codes (like 500 Internal Server Error or 404 Not Found) being returned by IIS, indicating a problem within the website’s application code running in w3wp.exe rather than a fundamental issue with the Azure platform hosting the VM. Analyzing these logs is crucial for understanding the actual traffic handled by your web application and identifying client-side or application-level errors.

Performance Counters¶

Monitoring performance counters is a standard practice for diagnosing performance issues on any Windows server, including Azure PaaS VMs. This can be done in real-time using the Performance Monitor application (search for “perfmon” on the Start menu). For historical analysis, it is highly recommended to configure the Windows Azure Diagnostics (WAD) extension to collect performance counter data over time and transfer it to Azure Storage. Having historical performance data readily available is invaluable for troubleshooting past incidents, such as performance degradation or outages that occurred hours or days ago.

If WAD has been configured to collect performance counters, this data becomes one of the most valuable resources for retrospective troubleshooting. Beyond looking for specific counters related to a known issue (e.g., high CPU, low memory, disk queues), common diagnostic patterns observed in WAD-collected performance data include:

State	Description
Regular counter entries	Normal operation or typical load patterns.
A period of no entries	May indicate the VM was not running, diagnostics agent stopped, or a severe system stall preventing data collection.
The resuming of entries	Indicates the VM or diagnostics agent recovered. Useful for correlating with downtime periods.
Consistently high CPU usage	Often points to an infinite loop, resource contention, or inefficient code within the application itself (w3wp.exe or worker process).
High Memory usage	Could indicate memory leaks or inefficient memory management by the application.
High Disk I/O	Suggests excessive reading/writing to disk, potentially related to logging, caching, or data processing.

Analyzing performance counter trends and anomalies is essential for diagnosing resource bottlenecks, application inefficiencies, and correlating system state with reported problems. Key counters to monitor often include Processor Time, Available Memory, Disk Read/Write Operations per Second, and Network Interface Bytes Total per Second.

HTTP.SYS Logs¶

The HTTP.SYS logs are located at D:\Windows\System32\LogFiles\HTTPERR and are another critical source for diagnosing web connectivity issues, complementing the IIS logs. HTTP.SYS is a kernel-mode component in Windows that handles incoming HTTP requests before they even reach user-mode processes like IIS worker processes (w3wp.exe).

These logs are particularly important when troubleshooting scenarios where a hosted website is unresponsive, but IIS logs show no record of the requests. This can happen if requests are dropped or rejected at the kernel level by HTTP.SYS, for example, due to issues like the request queue being full (503 Service Unavailable), malformed requests, or excessive request rates overwhelming the system before IIS can process them.

While IIS logs detail what happened after a request reached the IIS worker process, HTTP.SYS logs capture events related to requests that failed to be handled by HTTP.SYS itself. Comparing and correlating entries in both HTTP.SYS and IIS logs is crucial for determining whether a web connectivity problem is occurring at the kernel level or within the user-mode web server and application. They are especially valuable in diagnosing scenarios resembling denial-of-service attacks or high-traffic spikes that overwhelm the server’s ability to queue or process requests.

IIS Failed Request Log Files¶

Internet Information Services (IIS) Failed Request Tracing, often referred to as FREB (Failed Request Event Buffering), generates detailed log files for requests that meet specific criteria (e.g., taking longer than a certain time, returning a specific status code). These logs are located within the diagnostic store directory, typically at C:\Resources\Directory\<DeploymentID>.<RoleName>.DiagnosticStore\FailedReqLogFiles.

By default, FREB tracing is not enabled in Windows Azure PaaS roles because it can introduce performance overhead. Consequently, these log files are rarely populated unless explicitly configured. However, when you are troubleshooting complex or intermittent issues specific to how IIS processes requests or how your ASP.NET application behaves within the IIS pipeline, enabling FREB tracing can provide an extremely granular view of the request’s journey.

A FREB log file for a single request can detail every step the request goes through in the IIS pipeline, including which modules processed it, the timing of each step, and any errors or warnings encountered along the way. This level of detail is invaluable for diagnosing problems like unexpected redirects, authentication/authorization failures within IIS, module conflicts, or performance bottlenecks within specific pipeline stages. While not for everyday use, FREB tracing is a powerful tool for deep-dive IIS/ASP.NET troubleshooting.

Windows Azure Diagnostics Tables and Configuration¶

The local cache files and configuration for the Windows Azure Diagnostics (WAD) extension reside within the diagnostic store on the VM, specifically at C:\Resources\Directory\<DeploymentID>.<RoleName>.DiagnosticStore\Monitor. When configured, WAD collects various diagnostic data sources (like performance counters, IIS logs, Windows Event Logs) as specified in its configuration. It temporarily stores this data in custom .tsf files within this local directory before transferring it to the configured Azure Storage account according to the defined schedule.

While the .tsf files containing the actual diagnostic data are in a proprietary format and not easily readable directly on the VM, the configuration files stored alongside them are human-readable and crucial for diagnosing issues with WAD itself. Within the Configuration subfolder, you can find a file named config.xml. This XML file contains the complete configuration applied to the WAD extension on this specific VM instance.

If you suspect that WAD is not collecting or transferring data correctly (e.g., performance counters are missing in storage, logs are not appearing), inspecting this config.xml file is the first step. You can verify that the configuration reflects your intentions, checking for correct data source definitions, transfer schedules, and storage account details. Problems with WAD configuration, permissions, or the agent itself can often be identified by examining this local configuration file and the agent logs.

Windows Azure Caching Log Files¶

For Azure PaaS roles configured to use the older Windows Azure Role-based Caching feature, specific diagnostic logs are generated. These logs are located within the diagnostic store at C:\Resources\Directory\<DeploymentID>.<RoleName>.DiagnosticStore\AzureCaching. These files provide detailed information about the operation of the distributed caching service running within your role instances.

These logs contain events related to the cache cluster formation, cache node status, cache operations (gets, puts, removes), configuration issues, and errors encountered by the caching client or server components running on the VM. If you are experiencing problems with your application’s caching layer, such as data not being cached, cache misses, connectivity issues between cache nodes, or performance degradation related to caching operations, these dedicated caching logs are the primary source for diagnosing the root cause.

WaIISHost Logs¶

The WaIISHost log file is found at C:\Resources\Directory\<DeploymentID>.<RoleName>.DiagnosticStore\WaIISHost.log. This log captures information specifically from the WaIISHost.exe process. For Web Roles, WaIISHost.exe is the host process that bootstraps and runs your main role entry point code, typically defined in WebRole.cs. It’s responsible for integrating your application with IIS and the Azure environment.

While much of the critical information captured by WaIISHost.exe is also mirrored in other logs discussed (such as the Windows Azure event logs), the WaIISHost.log can sometimes provide additional context or slightly different timing information that might be useful during deep troubleshooting. It records events related to the startup, shutdown, and overall health of the host process managing your web application’s lifecycle within the Azure environment.

IISConfigurator Logs¶

The IISConfigurator log is located at C:\Resources\Directory\<DeploymentID>.<RoleName>.DiagnosticStore\IISConfigurator.log. This log file contains details about the operation of the IISConfigurator.exe process. The role of this process is to configure the Internet Information Services (IIS) instance on the VM based on the settings defined in your cloud service model (primarily the <Sites> element in the ServiceDefinition.csdef file) and the deployed package.

The IISConfigurator.exe process translates the abstract service model definition into concrete IIS configuration settings, including website bindings, application pools, virtual directories, and other IIS features. Failures or errors within this process are relatively rare, but if you encounter situations where IIS or the w3wp.exe worker process does not appear to be set up correctly for your specific web role configuration as defined in your service model, this log is the definitive place to investigate. It will record any issues encountered while attempting to apply the desired IIS configuration.

Role Configuration Files¶

The role configuration file, typically located at C:\Config\<DeploymentID>.<RoleName>.<Version>.xml, holds the runtime configuration details specific to your role instance. This file is generated based on the settings provided in your ServiceConfiguration.cscfg file, combined with information about the specific deployment environment allocated by Azure.

The contents of this XML file include configuration settings defined in your .cscfg (like connection strings, application settings), details about local resource directories allocated to the role, IP addresses and ports assigned for both Dynamic IP (DIP) and Virtual IP (VIP) endpoints, certificate thumbprints configured for the role, load balancer probe settings, and information about other instances within the same cloud service deployment.

Similar to the role model definition file, the role configuration file does not contain dynamic runtime-generated data like logs or performance counters. However, it is incredibly useful for verifying that your role instance has received and is operating with the configuration settings you expect. Checking this file can quickly rule out or confirm issues related to misapplied or missing configuration settings, especially after updates or redeployments.

Role Model Definition File¶

The role model definition file is usually found at E:\RoleModel.xml or F:\RoleModel.xml (depending on the drive letter assigned to the temporary storage). This file represents the definition of your service model as interpreted and instantiated by the Azure runtime. It’s derived from your ServiceDefinition.csdef file, which is part of your cloud service package.

This XML file contains details about the structure and components of your role instance as the platform understands them. Entries include definitions for each startup task specified in your service definition, detailing how each task is configured to run (e.g., background, executionContext, taskType), including environment variables and the task’s location. For web roles, it also clearly shows how your <Sites> element and bindings are defined.

The role model definition file is a static representation and does not contain runtime operational data. Its primary utility lies in verifying that the Azure platform is using the correct service definition for your deployment. This can be particularly helpful when diagnosing issues that might arise from discrepancies between the service definition file on a developer’s machine and the version used by the build and packaging process or deployed to Azure. Comparing the RoleModel.xml on the running VM with your intended service definition file helps ensure consistency and proper configuration interpretation by the platform.

About ETL Files¶

Within the C:\Logs folder, you will find files with the .etl extension, typically named RuntimeEvents_<Iteration>.etl and WaAppAgent_<Iteration>.etl. These files are Event Trace Log (ETL) files, which contain data captured using Event Tracing for Windows (ETW). They essentially provide a compiled or aggregated view of information sourced from multiple underlying logs, including the Windows Azure event logs, the App Agent Runtime logs, and other system traces.

The ETL files offer a convenient consolidation of some of the most critical diagnostic data available within an Azure VM, presenting a more holistic view of activity. Because they are in the binary ETL format, viewing their contents requires specific tools designed to process ETW traces (e.g., Windows Performance Analyzer, tracerpt utility). If you are familiar with and prefer using ETW analysis tools, you might find that reviewing the data within these two ETL files can provide much of the information available across several of the individual text-based log files mentioned earlier, potentially streamlining the initial data collection and analysis process.

Supporting Media: Diagnostic Data Flow¶

The following diagram illustrates the flow of diagnostic data from various components within an Azure PaaS VM to the local log files discussed.

mermaid graph TD A[Azure Fabric] --> B(Guest Agent) B --> C(WaAppAgent.log <br> Heartbeat) B --> D(AppAgentRuntime.log <br> Runtime) B --> E(WaHostBootstrapper) E --> F(WaHostBootstrapper.log) E --> G{Host Processes <br> WaIISHost/WaWorkerHost} G --> H(Windows Azure Event Logs) G --> I(Application Event Logs) G --> J(Performance Counters) G --> K(WAD Local Data <br> *.tsf, config.xml) G --> L(WaIISHost.log <br> if WebRole) G --> M(Caching Logs <br> if configured) G --> N{IIS Components <br> if WebRole} N --> O(IIS Logs) N --> P(HTTP.SYS Logs) N --> Q(FREB Logs) E --> R(IISConfigurator.log <br> if WebRole) S(Configuration Files <br> RoleModel.xml, ServiceConfig.xml) Subgraph Aggregated ETL T(RuntimeEvents_*.etl) U(WaAppAgent_*.etl) End H,D --> T C --> U S -- Read by --> E, G, R

This diagram shows the key processes (Guest Agent, Host Bootstrapper, WaIISHost/WaWorkerHost, IISConfigurator, IIS components) and the primary log files they generate or contribute to. Configuration files are read by relevant processes, and some logs are aggregated into ETL files.

Understanding the purpose and location of these various diagnostic data sources is a fundamental skill for anyone managing and troubleshooting applications hosted in Windows Azure PaaS. By systematically reviewing the relevant logs based on the nature of the problem, you can significantly reduce the time and effort required to identify the root cause and implement a solution.

Have you leveraged these or other diagnostic data sources when troubleshooting your Azure PaaS deployments? Share your experiences, tips, or common scenarios you’ve solved using these logs in the comments below!