Troubleshooting ASP.NET Core: Capture and Analyze .NET Core Dumps with ProcDump
Applies to: .NET Core 2.1, .NET Core 3.1, .NET 5.
This article serves as a comprehensive guide to utilizing the ProcDump tool within a Linux environment specifically for capturing memory dump files from .NET Core applications. Memory dumps are invaluable artifacts in the process of diagnosing complex issues such as performance bottlenecks, memory leaks, or application crashes. By mastering the techniques presented here, you can gain deep insights into the state of your application at the time a problem occurred, significantly aiding your troubleshooting efforts.
The focus will be on installing ProcDump and exploring its capabilities for capturing dump files, particularly highlighting its conditional triggering mechanisms which are powerful for capturing transient issues. Understanding where these dump files are saved and how they can be automatically triggered based on resource thresholds are key skills for effective server-side application diagnostics on Linux.
Prerequisites for Troubleshooting¶
Before diving into capturing memory dumps with ProcDump, ensure you have the necessary environment set up. A foundational requirement is an operational ASP.NET Core application deployed on a Linux server. This application should ideally be one you can use to simulate or observe performance problems, high resource usage, or crashes, allowing you to practice capturing dumps under various conditions.
Furthermore, having a debugger capable of analyzing .NET Core dumps on Linux is essential. The lldb debugger, coupled with the SOS extension, is the standard tool for this purpose. Confirm that lldb is installed and configured correctly to load the SOS extension automatically when you open a .NET Core dump file. This setup is crucial for inspecting the managed state of your application within the dump.
For those following along from previous troubleshooting labs, your environment should already be well-prepared. This typically includes a Linux server configured with Nginx acting as a reverse proxy. Nginx is often set up to host multiple ASP.NET Core applications, routing requests based on host headers.
A common configuration involves Nginx listening on standard HTTP ports and forwarding traffic to your ASP.NET Core applications running on different internal ports. For instance, one host header like http://myfirstwebsite might route to an application on port 5000, while another like http://buggyamb routes to a separate sample application on port 5001. It’s important that these applications are running as system services, configured to start automatically on boot and restart if they encounter issues.
Finally, ensure your Linux server’s local firewall is properly configured. The firewall should permit incoming SSH traffic for remote access and management, as well as HTTP/HTTPS traffic on the ports Nginx is listening on, typically port 80 and 443. This standard setup provides a stable platform for deploying and troubleshooting your ASP.NET Core applications.
Objective of This Lab¶
In previous troubleshooting exercises, you may have become familiar with analyzing application crashes or performance degradation using tools like createdump. These tools are effective for manual dump capture or triggering dumps on specific signals like crashes. However, diagnosing issues that occur intermittently or only under specific resource loads requires a more sophisticated approach.
This lab focuses specifically on introducing and installing the ProcDump tool for Linux. The primary goal is to learn how to use ProcDump to capture core dump files from your .NET Core processes. You will explore its basic usage and understand how it can be configured to capture dumps based on criteria such as resource utilization thresholds.
We will not delve into the analysis of the captured dump files in this lab, as the analysis techniques using tools like lldb and SOS remain consistent regardless of how the dump was captured. The emphasis here is solely on the capture phase, leveraging the powerful features ProcDump offers for automated and conditional dump generation. By the end of this lab, you will be equipped to use ProcDump as part of your .NET Core troubleshooting toolkit on Linux.
Introducing ProcDump¶
ProcDump, originally a popular Sysinternals tool for Windows, has been reimagined and ported to Linux. According to its official GitHub repository, the Linux version aims to bring similar powerful process monitoring and dump capturing capabilities to the Linux platform. It is designed to be a versatile utility for capturing core dumps of processes based on various triggers, making it exceptionally useful for debugging hard-to-reproduce issues.
While the Linux version shares the core philosophy of its Windows counterpart, it currently has certain limitations. Not all features available in the Windows version have been ported. For example, as of now, the Linux version cannot be configured to automatically capture core dumps when a process crashes or when it throws a first-chance exception. This means for crash analysis, other tools like createdump or systemd-coredump might still be necessary depending on your specific needs.
Despite these limitations, ProcDump for Linux is a remarkably potent tool. Its strength lies in its ability to monitor process resources and trigger dump collection when specific thresholds are met or exceeded. This is invaluable for diagnosing performance issues tied to high CPU, excessive memory consumption, high thread counts, or large numbers of open file descriptors.
The command-line options available provide a glimpse into its power:
-C: Trigger core dump generation when CPU exceeds or equals specified value (0 to 100 * nCPU)
-c: Trigger core dump generation when CPU is less than specified value (0 to 100 * nCPU)
-M: Trigger core dump generation when memory commit limit exceeds or equals specified value (MB)
-m: Trigger core dump generation when memory commit limit is less than specified value (MB)
-T: Trigger core dump generation when thread count exceeds or equals specified value.
-F: Trigger core dump generation when filedescriptor count exceeds or equals specified value.
These options allow you to set up automated monitoring for specific resource conditions. For example, you could monitor your ASP.NET Core process and automatically capture a dump the moment its memory usage surpasses a certain limit. This is far more efficient than manually monitoring resource usage and attempting to capture a dump reactively.
Installation of ProcDump on Linux¶
Installing ProcDump on most modern Linux distributions, particularly those based on Debian or Ubuntu, is straightforward, assuming you have already configured the Microsoft package repository. Recall from previous labs that setting up this repository was a prerequisite for installing the .NET Core SDK and runtime itself. ProcDump is distributed through the same repository, simplifying its installation process.
To install ProcDump, open your terminal and run the following command:
sudo apt update
sudo apt install procdump
The sudo apt update command refreshes your package list, ensuring you can access the latest version information, including that for ProcDump from the Microsoft repository. The sudo apt install procdump command then downloads and installs the ProcDump package onto your system. You may be prompted to enter your password to authorize the installation with superuser privileges.
Once the installation is complete, you can verify that ProcDump is installed and accessible by running procdump -h or procdump --help. This should display the command-line help information for the tool, confirming it’s ready for use. The ease of installation through standard package managers makes ProcDump readily available on your troubleshooting server.
Basic Usage: Manual Dump Capture¶
Beyond its conditional triggering capabilities, ProcDump can also be used to simply capture a memory dump of a running process manually. This is analogous to using tools like createdump without any arguments, capturing the current state of the process on demand. This manual method is useful when you observe a problem happening in real-time and want to immediately capture a dump for later analysis.
The basic command structure for manually capturing a dump is procdump -p <PID>, where <PID> is the Process ID of your target ASP.NET Core application. To find the PID of your ASP.NET Core application, you can use commands like htop, top, or ps aux | grep <application_name>. Locate the process corresponding to your application, typically running the dotnet command, and note its PID.
Because capturing a memory dump of a process requires privileged access to the process’s memory space, you must run the procdump command with sudo. For example, if your application’s PID is 12345, the command to capture a manual dump would be:
sudo procdump -p 12345
Executing this command will instruct ProcDump to create a core dump file for the process with PID 12345. ProcDump will provide output indicating that it is writing the dump file and report the filename once completed. This simple command provides a quick way to get a snapshot of your application’s state whenever you detect an issue.
Understanding Where ProcDump Saves Core Dump Files¶
One crucial piece of information when using ProcDump, especially for the first time, is understanding where it saves the generated core dump files. Unlike some other Linux diagnostic tools that might default to standard locations like /tmp/ or /var/lib/systemd/coredump/, ProcDump has a different default behavior that can initially be confusing.
When ProcDump successfully captures a dump, its output typically shows the name of the generated file, such as core_dotnet_12345_timestamp. However, it doesn’t explicitly state the full path to this file in the console output. If you aren’t aware of its default behavior, you might spend significant time searching for the file in common dump locations, only to find they aren’t there.
ProcDump, by default, creates the core dump files in the working directory of the target process. The working directory is the directory from which the application was executed or the directory specified in its service unit file if it’s running as a system service. This is a key difference compared to tools that save dumps to system-wide temporary or coredump directories.
To find the working directory of your ASP.NET Core application running as a service, you can inspect its service unit file. You can typically do this using the systemctl status <your_service_name> command, replacing <your_service_name> with the actual name of your service (e.g., systemctl status buggyamb). The output of this command usually includes a line indicating the WorkingDirectory.
For example, if your service unit file specifies /var/BuggyAmb_v1.1 as the WorkingDirectory, then any dump file captured by ProcDump targeting this service’s process will be created within the /var/BuggyAmb_v1.1 directory. Navigating to this directory using standard Linux commands like cd /var/BuggyAmb_v1.1 and then listing the files using ls -l will reveal the core dump files generated by ProcDump. Remember this location when you need to retrieve the dump files for analysis.
Sample Scenario: Capturing Dumps Based on Memory Usage¶
To illustrate the power of ProcDump’s conditional triggering, let’s consider a common troubleshooting scenario: diagnosing a high-memory consumption issue that occurs intermittently. Imagine your ASP.NET Core application experiences excessive memory usage, but this problem doesn’t happen constantly; it only manifests under specific, unpredictable load patterns. You’ve identified that the issue seems to start when the application’s committed memory usage reaches a certain threshold, say 750 MB.
Manually monitoring memory usage continuously is impractical and prone to missing the critical moment. You need an automated way to capture diagnostic information precisely when the problem occurs. Your goal is to automatically capture core dump files of the problematic process as soon as its memory usage exceeds the 750 MB threshold. To get multiple snapshots during the high-memory event, you decide you need two consecutive dump files, captured with a minimum interval of five seconds between them to observe the state changes.
This scenario is perfectly suited for ProcDump’s conditional triggering capabilities. You can configure ProcDump to monitor the target process and automatically initiate dump capture when the memory condition is met.
Based on the ProcDump command-line options discussed earlier, you can construct the command to meet the requirements of this scenario:
-p <PID>: Specify the Process ID of the target ASP.NET Core application. You need to replace<PID>with the actual PID.--M 750: This is the trigger condition. It tells ProcDump to monitor the process’s committed memory and activate the trigger when it reaches or exceeds 750 MB.-n 2: This specifies the number of dump files to capture after the trigger condition is met. We want two dumps.-s 5: This sets the sequential cool-down period in seconds between the generation of consecutive dump files. We require a minimum of five seconds between the two dumps.-d: (Optional but recommended) This option tells ProcDump to write diagnostic logs to Syslog, which can be helpful for understanding ProcDump’s activity and troubleshooting its own operation.
Combining these options, the command to start monitoring your application (assuming PID 11724 for example) for this scenario would be:
sudo procdump -p 11724 -n 2 -s 5 -M 750 -d
Executing this command with sudo starts ProcDump in monitoring mode. It will attach to the process with PID 11724 and continuously monitor its memory usage. ProcDump will patiently wait in the background until the committed memory usage of process 11724 reaches or exceeds 750 MB. It will continue monitoring until the trigger condition is met or until you manually interrupt it (e.g., by pressing Ctrl+C).
Executing the Sample Scenario¶
To demonstrate the sample scenario, ensure your target ASP.NET Core application is running and identify its PID. Let’s assume, as in the command example, the PID is 11724. Execute the sudo procdump -p 11724 -n 2 -s 5 -M 750 -d command in your terminal. You will see output from ProcDump indicating that it has attached to the process and is awaiting the trigger condition (-M 750).
While ProcDump is monitoring, you need to generate load on your application that will cause its memory usage to increase and eventually cross the 750 MB threshold. If you are using the BuggyAmb sample application mentioned in the prerequisites, you can typically achieve this by hitting the “Slow” scenario endpoint multiple times or by using a load generator tool configured to target this endpoint. Sending six requests to the “Slow” scenario, as suggested in the original context, is often sufficient to significantly increase memory consumption due to simulated work and potential memory allocation patterns.
Monitor the application’s memory usage using another terminal window with tools like top or htop (filtering by PID 11724). Observe the memory column (VSZ or RSS, though ProcDump monitors ‘commit’ memory which correlates closely with VSZ and RSS combined with swap). As the memory usage climbs and eventually surpasses the 750 MB mark, you will see ProcDump’s output change.
ProcDump will detect that the -M 750 condition has been met. It will then proceed to capture the first core dump file. After a pause of at least 5 seconds (due to the -s 5 flag), if the process is still running and the condition is still relevant (ProcDump checks this), it will capture the second dump file. ProcDump will print messages to the console indicating when each dump file is being written and its name.
Once the second dump file is successfully written, ProcDump will exit, as the -n 2 condition (capture 2 dumps and exit) has been satisfied. If the memory usage fluctuates above and below 750 MB, ProcDump’s behavior might be more complex, potentially capturing dumps on rising edges if the condition is met and held for the required -s duration between dumps.
Locating and Verifying the Captured Dumps¶
After ProcDump completes the capture based on the scenario trigger, the next step is to locate the generated dump files. As emphasized earlier, ProcDump saves the dumps to the working directory of the target process, not a system-wide location.
Navigate to the working directory of your ASP.NET Core application. If you followed the previous labs, this might be a path like /var/BuggyAmb_v1.1. Use the cd command to change your current directory:
cd /var/BuggyAmb_v1.1
Then, list the contents of the directory using the ls -l command:
ls -l core_*
This command will list all files starting with core_, which is the naming convention ProcDump uses for its core dump files. You should see two new files in this directory, corresponding to the two dumps captured during the memory scenario. The filenames will typically follow a pattern like core_dotnet_<PID>_<timestamp>.
These files are your captured memory dumps. They contain a snapshot of the process’s memory and state at the exact moments the memory usage threshold was met and the cool-down period elapsed. You can now transfer these files to a machine where you have lldb and SOS configured for analysis, or perform the analysis directly on the server if resources allow.
Comparing ProcDump and Createdump¶
It’s worth noting that the core dump files generated by ProcDump are fundamentally similar to those generated by the createdump tool, which is part of the .NET Core runtime. Both tools capture the necessary information from the target process’s memory space to allow for subsequent analysis using debuggers like lldb with the SOS extension.
From an analysis perspective, there is generally no difference between a dump file captured by ProcDump and one captured by createdump. The information contained within them regarding the managed heap, threads, stack traces, and other runtime details is comparable.
The primary distinction between the tools lies in their triggering mechanisms. createdump is primarily designed for manual dump capture or capturing dumps on specific signals (like crashes). ProcDump, on the other hand, excels at monitoring resource usage (CPU, memory, threads, file descriptors) and triggering dumps based on predefined thresholds. This makes ProcDump particularly powerful for diagnosing performance issues or resource leaks that manifest under specific load conditions or over time, where manual intervention is impractical. Your choice of tool will depend on the specific problem you are trying to diagnose and the conditions under which it occurs. For automated, condition-based capture, ProcDump is often the preferred tool.
Next Steps¶
Having successfully installed ProcDump and learned how to capture memory dump files manually and based on resource thresholds, you now possess a powerful capability for diagnosing production issues. The next logical step in the troubleshooting workflow is to analyze the dump files you have captured.
Analyzing core dumps typically involves using a debugger like lldb along with the SOS extension. This process allows you to inspect the state of the .NET Core runtime, including examining the managed heap to identify potential memory leaks, analyzing thread stacks to understand what the application was doing, and inspecting garbage collection details. Future labs or documentation would guide you through the process of loading the dump file into the debugger and using SOS commands to investigate the root cause of the issues indicated by the conditions you captured.
Mastering both the capture phase (using tools like ProcDump or createdump) and the analysis phase (using lldb/SOS) is crucial for effective troubleshooting of .NET Core applications on Linux. Practice capturing dumps under various simulated conditions to become proficient with ProcDump’s options.
We hope this guide has provided a clear understanding of how to install and use ProcDump for capturing .NET Core dumps on Linux. Your feedback is valuable! Do you have any questions about using ProcDump, or would you like to share your experiences with using it for troubleshooting? Please leave your comments below.
Post a Comment