Azure VM High Memory? Troubleshoot and Optimize Performance Now!

Table of Contents

Performance issues are a common challenge across various operating systems and applications, each demanding a unique troubleshooting approach. Key factors often contributing to these issues include CPU, memory, networking, and input/output (I/O). Each of these areas can manifest distinct symptoms, sometimes simultaneously, necessitating different diagnostic strategies and solutions.

This article specifically delves into high memory usage, often referred to as Memory Pressure, encountered on Azure Virtual Machines (VMs) running the Windows operating system. For guidance on troubleshooting Azure VMs utilizing Linux distributions, please refer to documentation on performance bottlenecks in Linux environments. Our focus here remains squarely on optimizing Windows-based Azure VMs.

Understanding Memory Pressure Issues on Azure Windows VMs¶

Addressing CPU and memory performance issues in Azure Windows VMs largely mirrors the techniques used for on-premises servers. However, I/O and network latency concerns have specific considerations in a cloud environment. Azure VM support highly recommends PerfInsights as a primary diagnostic tool, available for both Windows and Linux VMs. This tool offers a user-friendly report with Azure VM best practice diagnoses.

PerfInsights is a comprehensive wrapper tool capable of collecting Perfmon, Xperf, and Netmon data, depending on the selected flags. You can run PerfInsights on your Azure VM independently, without requiring Microsoft Support intervention. The tool generates reports either locally on the VM or within a specified storage account, providing flexibility for data analysis.

Many standard performance troubleshooting tools used for on-premises servers, such as Perfmon or Procmon, are fully functional on Azure Windows VMs. Nevertheless, PerfInsights is custom-tailored for Azure environments. It provides enhanced insights, including Azure and SQL best practices, high-resolution I/O latency graphs, and dedicated memory tabs for more granular analysis. This article will focus on the aspects of PerfInsights most relevant to memory troubleshooting.

What is Memory Pressure?¶

Memory Pressure occurs when a system faces a crunch in available physical memory (RAM) or virtual memory (backed by a paging file), driven by the demands of running processes. In such scenarios, the operating system continuously adjusts active pages in RAM by swapping them with the paging file. This constant activity signifies that the system is under significant memory pressure, with the OS working diligently to allocate sufficient working areas for demanding applications.

A memory leak represents another variant of this issue, typically stemming from coding errors. In this situation, the operating system consistently seeks more memory allocations, eventually leading to the exhaustion of virtual memory. The type of workload running on a machine is a primary driver of resource consumption, including how much memory is utilized.

Common Factors Leading to Low Memory Situations¶

Several common factors can contribute to or exacerbate a low memory situation. Identifying any of these conditions can quickly lead you closer to the root cause of the issue. A proactive review of these areas is crucial for maintaining optimal VM performance.

Here are the most frequent culprits:

• Recent Code Changes or Deployments: Applications like Internet Information Services (IIS), Microsoft SharePoint, SQL Server, or third-party applications are particularly sensitive to new code deployments. A recent change might introduce inefficiencies or memory leaks.
• Operating System or Application Updates: Updates, whether OS-level cumulative updates or application-specific fixes, can sometimes introduce regressions or modify resource consumption patterns. Always monitor memory after applying updates.
• Query Changes or Outdated Indexes: In data-tier applications like SQL Server and Oracle, query plan optimization is a critical factor. Changes in data or a lack of appropriate indexing can cause queries to retrieve more data pages than necessary, putting stress on available memory and swap space.
• Azure VM-Specific Processes: Certain processes inherent to the Azure environment, such as RDAgent, or extension-specific processes like the Monitoring Agent, MMA agent, or security clients, can consume significant memory. These should be reviewed for configuration, known issues, or potential regression builds.

Troubleshooting Workflow for Memory Pressure¶

Effectively troubleshooting memory pressure aims to pinpoint the exact culprit process. Once identified, further analysis becomes specific to that process driving the high memory consumption. This targeted approach ensures that remediation efforts are precise and efficient.

For instance, if SQL Server (sqlservr.exe) is determined to be the problematic process, the next steps involve analyzing configurations such as MaxServerMemory. It is also essential to identify which specific queries consume the most memory cycles during a particular timeframe. For more comprehensive insights into SQL Server best practices within Azure, consult resources on VM size and performance guidelines for SQL Server on Azure VMs.

Scoping Memory Pressure Issues¶

When troubleshooting memory pressure, asking a series of targeted questions can help scope the issue and narrow down potential causes. This systematic inquiry provides a clearer picture of the problem’s context and behavior. Understanding the ‘when’, ‘what’, and ‘how’ of the issue is paramount for effective resolution.

Consider the following questions during your investigation:

• Pattern Recognition: Is there a discernible pattern to the low memory issue? For example, does it occur at a specific time daily, weekly, or monthly? If so, can you correlate it with a particular job, process, report generation, or user login activity?
• Application Tier Changes: Did the low memory problem emerge after a recent code deployment or application update? Was a new application build introduced, or was there a sudden surge in the user base? Could load balancing misconfigurations lead to an altered load pattern?
• Data Tier Changes: Did the issue begin following a change in workload, such as an increase in user count, higher data influx, or more reports being generated? Was this workload recently migrated to Azure, and did it function correctly before the migration?
• Azure-Specific Conditions: For Azure VMs, did the low memory issue start under any of these conditions?
  • Following a recent redeployment or VM restart?
  • After a change in VM type or tier?
  • Subsequent to adding a new extension to the VM?
  • After modifications were made to the load balancer configuration?
  • Was there a scale-in/out event for your VM Scale Set (VMSS) setup, leading to an unexpected load distribution?

Azure Caveats for Memory Management¶

Managing memory in Azure VMs requires a careful understanding of cloud-specific nuances. Several factors unique to Azure can significantly influence memory utilization and overall VM performance. Being aware of these caveats is essential for effective planning and troubleshooting.

Most importantly, you must thoroughly understand your workload’s memory demands. When selecting a VM SKU (offering/type), there’s a tendency to prioritize overall monthly hosting costs, potentially underestimating memory specifications. If your workload is memory-intensive, choosing a smaller VM SKU can invariably lead to persistent low memory issues. It is crucial to test different configurations for your specific workload to determine the optimal computing capability required.

Are you planning a Multi-User setup, such as Azure Virtual Desktop (AVD)? Resources, including memory, must be carefully factored based on resource sharing among active sessions. Similarly, if your application is designed to maintain its working context in memory, like MongoDB configured for In-Memory operations, it will heavily utilize both RAM and swap spaces. Therefore, the chosen VM type (offering) must adequately meet these stringent memory requirements.

For widely used applications and configurations such as SQL Server, Oracle, Remote Desktop Services (RDS), Azure Virtual Desktop, IIS, or SharePoint, Azure provides best practices articles. These resources offer recommendations for the minimal configuration required to optimally support these specific workloads, helping you avoid common memory-related pitfalls.

Addressing Ongoing Low Memory Issues¶

If a low memory issue is currently active, this presents the prime opportunity to capture process traces and precisely identify the root cause. Leveraging existing tools familiar from on-premises Windows servers, such as Procmon, Perfmon, RAMMAP, xPerf Windows Toolkit, or ResourceExplorer, can be highly effective. These tools allow for deep-dive analysis into real-time memory consumption.

For Azure VMs, Azure Support specifically recommends the following tools:

• PerfInsights
• Perfmon
• Azure monitoring tools

PerfInsights: Azure’s Recommended Diagnostic Tool¶

PerfInsights stands as the recommended tool from Azure support for diagnosing VM performance issues. It provides a comprehensive analysis, encompassing best practices and featuring dedicated tabs for memory, CPU, and high-resolution I/O graphs. You can execute PerfInsights on-demand directly through the Azure portal or from within the VM itself. Its deep integration with the Azure portal also extends to offering best practice recommendations for common Windows guest-tier applications.

If you have an active support ticket with the Azure team, PerfInsights facilitates easy sharing of diagnostic data. Alternatively, you can utilize the tool independently for proactive performance analysis, leveraging its robust reporting capabilities.

Obtaining PerfInsights¶

PerfInsights is readily available for both the Windows and Linux operating systems. This cross-platform compatibility ensures that administrators can leverage its powerful diagnostics across diverse Azure VM deployments. For Windows environments, several options exist for deployment and execution.

Running and Analyzing Reports via Azure Portal¶

When PerfInsights is installed through the Azure portal, it deploys an extension to the target VM. Alternatively, you can directly install PerfInsights as an extension by navigating to the ‘Extensions’ blade within your VM and selecting the performance diagnostics option. This streamlined installation process makes it accessible for immediate use.

Azure portal - Option 1

Select the Performance diagnostics option in your VM blade. You will be prompted to install this feature, which utilizes an extension, on your chosen VM. After installation, a Run diagnostics panel will appear, allowing you to select various analysis types to execute.

For memory analysis, the preferred choices are Performance Analysis and Advanced Performance Analysis. These options collect more detailed data crucial for identifying memory pressure.

Based on your selection, further configuration options will be presented. The following screenshot illustrates these choices.

The numbered options in the screenshot correspond to these actions:

1. Select Run diagnostics to initiate the process.
2. For the Analysis option, choose either Performance analysis or Advanced performance analysis. Basic collection primarily gathers VM configuration details and best practice suggestions, which are insufficient for detailed memory troubleshooting. The more advanced collection flags are necessary for comprehensive data capture.
3. Specify the trace duration for data collection. For memory troubleshooting, a duration of 15 minutes or more is recommended. In Azure portal mode, data collection is limited to 15 minutes. For longer periods, the program must be run as an executable from within the VM.
4. If requested by Azure Support, you can input the ticket number in the Support request number field. This field is optional but helpful for correlating data with support cases.
5. Accept the End User License Agreement (EULA) by checking the box I acknowledge that I am getting this software from Microsoft Corp. and that I have read and agree to the legal terms and privacy policy. Additionally, select I agree to share diagnostics information with Microsoft if you wish to make this report accessible to the Azure Support team assisting you.

The generated Performance diagnostics report is securely stored in one of the storage accounts under your subscription. It remains available for viewing and downloading at a later time. After submitting a diagnostic run, the same Performance diagnostics view will display a summary of past report collections, categorized by findings with High, Medium, or Low impact.

You can select any of the listed reports to delve into a more granular level of detail. For in-depth analysis, you also have the option to download the full report.

Azure portal - Option 2

Alternatively, navigate to Diagnose and Solve Problems within the VM blade and locate VM Performance Issues in the right-hand panel. This section offers another pathway to access performance diagnostics.

Upon selecting Troubleshoot, the PerfInsights installation page will load. This page also presents automated diagnostics, which run internal checks and highlight any instances of Memory Pressure on the VM over the preceding 24 hours. A “run performance diagnostics” link will also be displayed under Troubleshooting steps, allowing you to switch back to the PerfInsights view in the Azure portal.

Running PerfInsights from the VM¶

This method is particularly useful when you need to run PerfInsights for extended durations beyond the Azure portal’s 15-minute limit. The PerfInsights article provides a comprehensive walkthrough of the various commands and flags necessary to execute PerfInsights as an executable directly on the VM. To analyze high memory usage, you can utilize either of the following modes and commands:

• Advanced scenario:
  

  PerfInsights /run advanced xp /d 900 /AcceptDisclaimerAndShareDiagnostics
  

• VM Slow (Performance) scenario:
  

  PerfInsights /run vmslow /d 900 /AcceptDisclaimerAndShareDiagnostics
  

Each command will run for a duration of 15 minutes (900 seconds). The generated output will be saved in the same folder where you stored the PerfInsights executable. For more options and commands, refer to the detailed documentation on how to run PerfInsights.

Interpreting the PerfInsights Report¶

After running PerfInsights, the location of the collected content depends on whether it was executed through the Azure portal or as a standalone executable. In either case, you will need to access the generated log folder or download the report (if from Azure portal) locally for thorough analysis. The report provides a structured overview of the VM’s performance.

Accessing Collected Data¶

Your folder structure after a PerfInsights run should resemble the following layout:

1. Any supplementary collections, such as Perfmon, Xperf, Netmon, SMB logs, or Event logs, can be found within the designated output folder. These provide raw data for deeper investigation.
2. The primary report, including detailed analysis and actionable recommendations, is contained within the Perfinsights Report.html file. This HTML document offers an organized summary of findings.
3. For both Performance (VMSlow) and Advanced scenarios, PerfInsights automatically gathers Perfmon information throughout its execution, providing valuable performance counters.
4. The event logs offer a quick overview of critical system-level events or process crash details, which can be instrumental in identifying immediate issues.

Where to Begin Analysis¶

Open the PerfInsights Report.html file. Start by navigating to the Findings tab, which logs any significant outliers in terms of resource consumption. If instances of high memory usage or other resource constraints are detected, this tab will categorize them by their impact level: High, Medium, or Low. This prioritization helps you focus on the most critical issues first.

Expanding a Findings event reveals several crucial details. The tab meticulously lists processes in descending order based on their average memory consumption. Crucially, it also indicates whether a process is system-related or associated with a specific application, such as SQL Server, IIS, or a third-party application. This distinction is vital for directing subsequent troubleshooting efforts.

Deeper Dive into Memory Details¶

For a more granular pattern analysis, there is a dedicated subtab called Top Memory Consumers located under the Memory section. This tab provides detailed insights into processes during the data capture duration, allowing you to observe their memory usage patterns over time.

The Top Memory Consumers tab is further divided into two distinct sections: High Memory Usage Periods and Top Memory Consumers.

• High Memory Usage Periods: This section visually represents a time bar indicating when memory consumption was elevated. It also displays the overall Committed Memory percentage, a key indicator of memory exhaustion. The AvailablePhysicalMemory% metric highlights how thinly the system is running on memory resources, or what remains available for operations.
• Top Memory Consumers: This section is equally crucial as it provides process-level details of high memory consumers during the data capture interval. It shows both the process details and their respective memory usage patterns. The list is sorted to place processes with the highest average memory consumption at the top. Pay close attention to the EndTime - StartTime, which represents the actual duration a process consumed high memory within the broader data collection timeframe.

These two tabs provide all the necessary information to guide your next troubleshooting steps. Depending on the process identified as driving the high memory condition, you will need to address the questions posed earlier. For processes like SQL Server (sqlservr) or IIS (w3wp), a specific drill-down into query optimizations or recent code changes is often required.

For system processes such as WmiPrvSE or SvcHost, a different diagnostic path is necessary. A prerequisite for addressing issues with these processes is ensuring that your operating system is updated to the latest cumulative patch. For Azure VM-related processes like RDAgent, Operations Management Suite (OMS), and security extension executables, you must ensure these processes are running their latest versions. If issues persist even after upgrading to the newest build, it is advisable to seek assistance from the Azure Support team.

System event logs can also log Event ID 2004 as a warning, which often indicates memory pressure. This warning typically lists the top three processes based on their memory consumption, offering another quick insight into potential culprits.

Perfmon: The Classic Performance Monitor¶

Perfmon is one of the oldest and most fundamental tools for troubleshooting resource issues on Windows OS. While highly effective, it does not generate clear reports with immediate recommendations or findings. Instead, users are required to actively explore the collected data and apply specific filters across various counter categories for analysis.

PerfInsights automatically collects Perfmon data as an additional log for both VMSlow and Advanced scenarios. However, Perfmon can also be collected independently, offering several distinct advantages. These include its ability to be collected remotely, scheduled via Tasks, and run for extended durations or in continuous mode using its roll-over feature.

Consider the same example previously analyzed with PerfInsights to understand how Perfmon presents this data. A quick and reliable indicator of memory health is the Available MBytes counter. This counter shows the physical memory currently available for allocation to the system or any demanding process. While the OS will begin flagging processes to reduce their WorkingSets when memory contention is detected, a general threshold for a red flag is when Available Mbytes consistently drops below 200 MB.

Perfmon is a built-in utility in all Windows (Server or client) versions and can be invoked either through its graphical user interface (UI) or via the command line. To launch Perfmon in Windows, simply go to Start > Run, type Perfmon, and click OK. Any data collection performed with Perfmon will result in a file with a *.blg extension.

You can add various counters under different resource categories. For memory troubleshooting, selecting Memory > Available MBytes is a crucial first step. This counter provides a direct measure of the physical memory that remains unallocated and ready for use.

Once this counter is added, it will display the pattern for AvailableMBytes during the data capture interval or in real-time. Opening a Perfmon (*.blg) file from a PerfInsights output folder and adding this counter will reveal dips in the memory available to the machine. This visual representation quickly highlights periods of memory scarcity.

However, the fundamental question persists: which application is excessively consuming memory? Tools like PerfInsights, Resource Monitor, and Performance Analysis of Logs (PAL) simplify this identification process through user-friendly UIs or comprehensive reports. Without such aids, manually sifting through Perfmon data to pinpoint the exact culprit process can be quite tedious.

Perfmon includes a “Process” counter category, which contains several key counters such as Working Set, Private Bytes, and Virtual Bytes. These counters provide insights into the memory usage pattern of each individual process running throughout the data collection duration. Observing patterns of gradual or steep increases in these counters can indicate processes with continuous and escalating memory demands.

To narrow down culprit processes, you need to observe patterns that closely match the _Total/PrivateBytes counter, which represents the aggregate private bytes across all processes. This helps in identifying the specific processes contributing most significantly to overall memory consumption.

Simplifying Perfmon Analysis¶

Perfmon itself does not offer ready-made, user-friendly reports by default. While different views can alter the graph type, the process of filtering or identifying culprit processes remains a manual and labor-intensive task. Fortunately, several useful tools are available that can consume *.blg files and provide a more user-friendly output for analysis. One such popular tool is the PAL (Performance Analysis of Logs) Tool. The PAL Tool can parse *.blg files and generate detailed, easily digestible reports.

The list of available tools for performance monitoring extends far beyond just PerfInsights and Perfmon. You are free to utilize any existing tools you prefer, such as Procmon, RAMMAP, xPerf Windows Toolkit, or Resource Explorer. Additionally, numerous third-party tools are available that can be employed when necessary to enhance your diagnostic capabilities.

Azure Monitoring Tools¶

Any production system necessitates consistent monitoring to proactively identify issues, receive alerts, and establish historical patterns. This helps in better understanding the resource requirements of the application environment. Fortunately, Azure VMs, encompassing both Windows and Linux, are deeply integrated with Azure Monitoring capabilities. This integration provides robust automated metrics collection and offers various avenues to build custom reports or integrate with third-party tools tailored to your team’s needs.

If a critical production VM reaches a specific resource threshold, such as high memory usage, you can configure custom alerts. These alerts can notify your team via email, text message, or other communication channels, enabling a rapid response to potential issues. Azure VMs inherently offer reliable metrics that include essential information like memory utilization, network I/O, and disk I/O bytes. For more advanced metrics, leveraging Azure Monitor only requires a few selections to configure and utilize a specified storage account for data collection.

Basic (Default) Counters¶

Upon creation, Azure VMs come with a set of basic default metrics pre-enabled. These foundational metrics do not require any additional configuration to activate Azure Monitoring, as they function independently of the Azure Monitoring extension installation. While their capability isn’t as exhaustive as what Azure Monitoring provides, they still offer crucial insights into the VM’s basic resource health.

The following screenshot illustrates how the memory usage pattern is displayed within the Azure portal under a VM’s Memory metrics. This image corresponds to the same timeframe used earlier for the PerfInsights and Perfmon analyses, demonstrating consistency across diagnostic tools.

Enabling Azure Monitor¶

Once you enable Azure Monitor metrics, the necessary software installs an extension on the VM, initiating the collection of granular performance metrics. This allows for a much deeper and more detailed understanding of your VM’s operational health and resource consumption patterns.

You can explore the extensive customization options that Azure Monitoring provides for both Windows and Linux Azure virtual machines:

• Monitor Azure virtual machines (Example YouTube Video - Replace with actual relevant video if found)
• Monitor virtual machines with Azure Monitor (Example YouTube Video - Replace with actual relevant video if found)
• How to chart performance with VM Insights (Example YouTube Video - Replace with actual relevant video if found)

Example: An illustrative video on Azure Monitor, providing a broad overview of its capabilities.

Reactive Troubleshooting: Addressing Past Issues¶

If a high-memory issue has already occurred, the challenge shifts to retrospectively discovering its cause. This reactive stance can be tricky, as real-time data collection modes are less useful once the event has passed. As discussed previously, configured Azure Monitor metrics can play a significant role here, as they maintain historical data based on your specified retention time.

If the issue was a one-time, isolated occurrence, pinpointing the exact application or process responsible might prove difficult. Without ongoing monitoring or logs, the evidence can be sparse. However, if you are dealing with a recurring pattern, you have a valuable opportunity to collect data during the period when the issue is next expected to manifest. PerfInsights and Perfmon are invaluable tools for this purpose, but you are free to use any preferred diagnostic tools.

Currently, PerfInsights does not offer a scheduled run capability directly. Nevertheless, Perfmon can be scheduled and executed through the command line, as detailed in the following section. This allows for automated data collection during anticipated problem periods.

Logman Command for Scheduled Perfmon Collection¶

The Logman create counter command is specifically used to initiate Perfmon data collection via the command line. This command provides the flexibility to schedule the collection through Task Manager or run it remotely, enabling unattended data gathering for extended periods.

Here’s an example command for remote collection mode, which can be adapted for your specific environment:

Logman create counter LOGNAME -u DOMAIN\USERNAME * -f bincirc -v mmddhhmm -max 300 -c "\\SERVERNAME\LogicalDisk(*)\*" "\\SERVERNAME\Memory\*" "\\SERVERNAME\Network Interface(*)\*" "\\SERVERNAME\Paging File(*)\*" "\\SERVERNAME\PhysicalDisk(*)\*" "\\SERVERNAME\Process(*)\*" "\\SERVERNAME\Redirector\*" "\\SERVERNAME\Server\*" "\\SERVERNAME\System\*" "\\SERVERNAME\Terminal Services\*" "\\SERVERNAME\Processor(*)\*" "\\SERVERNAME\Cache\*" -si 00:01:00

This Logman.exe command can be executed from a peer Azure VM within the same Virtual Network (VNET), allowing for centralized performance monitoring. To learn more about these parameters and how to customize them for your specific needs, refer to the official documentation for logman create counter.

Once the Perfmon data has been collected during the occurrence of the issue, the subsequent steps for analyzing the data remain identical to those discussed earlier. This involves loading the .blg file into Perfmon or a compatible analysis tool to identify memory-intensive processes and patterns.

Conclusion¶

For any performance issue, a profound understanding of your workload is the cornerstone of effective problem-solving. When selecting Azure VM SKUs and various disk storage options, it is imperative to evaluate them by prioritizing the specific demands of your production workload. Utilizing tools like the Azure Virtual Machine selector can assist in making informed decisions. The process of testing different solutions across various VM configurations will ultimately help you determine the optimal setup.

Given the inherent variability in user operations and data volumes, always maintain a buffer in the memory, computing, networking, and I/O capabilities of your VM. This strategic buffer ensures that any sudden and unexpected surge in workload can be effectively absorbed by the additional resources, preventing performance degradation.

If you anticipate an imminent increase in workload, it is advisable to proactively transition to a higher VM offering or type that provides greater compute, memory, and I/O capabilities. Particularly for memory-intensive workloads, carefully choose VM SKUs with a higher memory-to-CPU ratio to ensure adequate resources are consistently available for your applications.

What are your experiences with troubleshooting high memory usage in Azure VMs? Share your tips and challenges in the comments below!