Troubleshooting ASP.NET Core Crashes: A Practical Reproduction and Debugging Guide

Table of Contents

This article delves into the essential process of replicating .NET Core crash scenarios within a Linux environment. It is specifically designed for applications running on .NET Core 2.1, .NET Core 3.1, and .NET 5. Furthermore, we will explore effective methods for examining Nginx tools and system logs to identify symptoms and indicators that point towards the root cause of these crashes. Understanding these diagnostic techniques is crucial for maintaining the stability and reliability of your ASP.NET Core applications in production.

Prerequisites¶

To effectively follow the troubleshooting labs outlined in this guide, you will need a functioning ASP.NET Core application. This application should be capable of demonstrating both low-CPU and high-CPU performance issues. These performance variations are often indicative of underlying problems that can eventually lead to application instability or crashes.

Several sample applications are readily available online that can serve this purpose. A highly recommended option is Microsoft’s simple web API sample, which is specifically designed for diagnostic scenarios. This sample is easily downloadable and can be quickly set up to exhibit undesirable behaviors that are useful for learning troubleshooting techniques. Alternatively, the BuggyAmb ASP.NET Core application provides another excellent sample project that is intentionally designed to be problematic, making it ideal for practicing crash reproduction and debugging.

If you have been following previous articles in this series, you should already have a suitable environment configured. This setup should include the following key components:

• Nginx Configuration: Nginx should be set up to host two distinct websites. The first website should be configured to listen for requests directed to the myfirstwebsite host header (http://myfirstwebsite). These requests should be routed to a demo ASP.NET Core application that is listening on port 5000. The second website should listen for requests with the buggyamb host header (http://buggyamb) and route them to a second ASP.NET Core sample application, BuggyAmb, which listens on port 5001. This dual-site configuration allows for testing and isolation of issues in different application contexts.
• ASP.NET Core Applications as Services: Both ASP.NET Core applications must be running as system services. This is crucial for ensuring automatic restarts in case of server reboots or application failures. Service management ensures high availability and resilience for your web applications.
• Linux Firewall Configuration: The local Linux firewall must be enabled and properly configured to allow essential traffic. Specifically, it should permit SSH traffic for remote access and HTTP traffic for web application access. This secure configuration is vital for both accessibility and security.

To proceed with this lab, you must have at least one ASP.NET Core web application that is experiencing issues and is deployed behind an Nginx reverse proxy. This problematic application will serve as the subject for our crash reproduction and troubleshooting exercises.

Goal of this lab¶

This article serves as the first part of a two-part lab series specifically designed to guide you through the process of diagnosing and resolving ASP.NET Core application crashes. The lab work is strategically divided into two parts to ensure a comprehensive and manageable learning experience.

Part 1: Crash Reproduction and Initial Troubleshooting: In this part, you will actively reproduce a crash issue within the ASP.NET Core application. You will then learn how to examine both Nginx and system logs. This examination aims to identify crash symptoms and indicators that can provide initial clues about the nature of the problem. Following log analysis, you will be guided through the process of generating a crash dump file. This file is a snapshot of the application’s memory at the time of the crash and is crucial for in-depth analysis. Finally, you will learn how to retrieve the system-generated core dump file. This file is automatically created by the Ubuntu system manager, apport, upon application crashes.

Part 2: Core Dump Analysis with lldb Debugger: The second part of this lab focuses on the detailed analysis of the crash dump file obtained in Part 1. You will install and configure lldb, which is a powerful debugger commonly used in Linux environments. Furthermore, you will configure lldb to work seamlessly with SOS, a .NET Core debugger extension. SOS provides invaluable tools for inspecting .NET runtime internals and managed code state within the dump file. Using lldb and SOS together, you will perform a thorough analysis of the dump file to pinpoint the root cause of the application crash.

By completing both parts of this lab, you will gain a robust understanding of the entire crash troubleshooting process, from initial symptom identification to deep-dive root cause analysis.

Reproduce a crash problem¶

To effectively troubleshoot application crashes, the first crucial step is to reliably reproduce the crash scenario. In the context of the BuggyAmb application, navigating to the site URL, http://buggyamb/, and selecting the “Problem Pages” link will present you with a set of links that simulate various problem scenarios. Among these, you will find three distinct crash scenarios. For the purpose of this lab, we will focus specifically on troubleshooting the third crash scenario, which provides a representative example for learning debugging techniques.

Before initiating any of the crash scenarios, it is highly recommended to verify the normal operation of your application. Select the “Expected Results” link. This action should load a page that demonstrates the application working as intended. You should observe an output that is similar to the expected baseline performance.

The page should load quickly, ideally in less than one second, and display a clear list of products. This confirms that the application is initially in a healthy state before we introduce any problem scenarios.

To test the first problem scenario, which simulates a “slow page” issue, select the “Slow” link. Upon selecting this link, the page will eventually load and display the same product list as the “Expected Results” page. However, the key difference is that it will render significantly slower than expected. This slow rendering is indicative of a performance bottleneck, although not a crash in this case.

Before you proceed to reproduce the crash problem, it is important to note the process ID (PID) of the BuggyAmb application. This PID will be used later to verify that your application indeed restarts after a crash, confirming the service management configuration is working. Execute the command systemctl status buggyamb.service in your terminal. This command will provide the current status of the BuggyAmb service, including its PID. In the example output shown below, you can see that the PID of the process currently running the service is 2255. This number will likely be different in your environment.

● buggyamb.service - BuggyAmb ASP.NET Core Application
     Loaded: loaded (/etc/systemd/system/buggyamb.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2024-07-02 10:00:00 UTC; 5min ago
   Main PID: 2255 (dotnet)
      Tasks: 21 (limit: 4915)
     Memory: 30.0M
        CPU: 1.234s
     CGroup: /system.slice/buggyamb.service
             └─2255 /usr/bin/dotnet /app/BuggyAmb.dll

Now, to trigger the crash scenario, select the “Crash 3” link on the “Problem Pages” of the BuggyAmb website. After selecting this link, the page will load and display a specific message. This message is intentionally designed to be somewhat misleading, prompting the user to consider: “Will this page cause the process to crash?”. At this point, run the systemctl status buggyamb.service command again. You should observe that the PID remains the same as before (e.g., still 2255). This indicates that despite the suggestive message, a crash has not yet occurred.

To proceed and actually trigger the crash, first select “Expected Results” again and then immediately select “Slow”. While you might briefly see the correct page after selecting “Expected Results”, the subsequent selection of “Slow” should quickly result in the following error message being displayed in your browser: “HTTP 502 - Bad Gateway”. This error indicates that the web server (Nginx in this case) is unable to communicate with the backend application (BuggyAmb).

Furthermore, if you attempt to select any other link on the webpage at this point, you will likely encounter the same “HTTP 502 - Bad Gateway” error. This state of unresponsiveness will persist for a short period, typically around 10–15 seconds. After this brief period, the application should automatically recover and start responding as expected again. This recovery is due to the service configuration that automatically restarts the application upon failure.

To confirm whether the PID has changed and the application has indeed restarted, run the systemctl status buggyamb.service command once more. This time, you should notice a significant change. The process appears to have stopped and restarted, as evidenced by the PID being different from the initial PID. In our example, the initial PID was 2255. After the crash and restart, it has changed to 2943. This PID change definitively confirms that the website’s “promise” to crash the process was fulfilled.

● buggyamb.service - BuggyAmb ASP.NET Core Application
     Loaded: loaded (/etc/systemd/system/buggyamb.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2024-07-02 10:05:00 UTC; 10s ago
   Main PID: 2943 (dotnet)
      Tasks: 21 (limit: 4915)
     Memory: 30.1M
        CPU: 0.234s
     CGroup: /system.slice/buggyamb.service
             └─2943 /usr/bin/dotnet /app/BuggyAmb.dll

Troubleshooting the steps of repro¶

Let’s summarize the steps we took to reproduce the crash and the observed behavior. Understanding these steps is crucial for systematically diagnosing the underlying issue.

1. Select “Crash 3”: Initially, navigating to the “Crash 3” link loads the page correctly. However, it presents a confusing message that suggests a potential crash, even though no crash occurs immediately.
2. Select “Slow”: Upon selecting the “Slow” link, instead of the expected product table, you receive an “HTTP 502 - Bad Gateway” error message in your browser. This indicates a problem with communication between the web server and the application.
3. Unresponsiveness: After the “HTTP 502” error first appears, the entire application becomes unresponsive for approximately 10–15 seconds. During this period, none of the pages render correctly, and all requests result in the same “Bad Gateway” error.
4. Automatic Recovery: After the brief period of unresponsiveness (10–15 seconds), the application automatically recovers and begins responding correctly again. This recovery is due to the service configuration that automatically restarts the ASP.NET Core application upon detecting a failure.

The “HTTP 502 - Bad Gateway” error message itself, while seemingly generic, provides a critical first clue. This error is specifically a proxy error. It typically arises when a proxy server, such as Nginx in our setup, is unable to establish or maintain communication with the application server running behind it. In our architecture, Nginx is deliberately configured as a reverse proxy for the ASP.NET Core application. Therefore, this “HTTP 502” error from Nginx strongly suggests that it was unable to reach the backend ASP.NET Core application when it attempted to forward incoming user requests. This inability to communicate is a strong indicator that the ASP.NET Core application may have crashed or become unresponsive, leading Nginx to be unable to fulfill its proxy duties.

Verify that Nginx works correctly¶

Before diving deeper into application-specific troubleshooting, it is a good practice to quickly verify that Nginx, our reverse proxy, is functioning correctly. Although we strongly suspect the application is crashing, confirming Nginx’s operational status can help isolate the problem and rule out potential proxy-related issues. This step, while not strictly mandatory in this specific crash scenario since we have other indicators, is a valuable habit in general troubleshooting workflows.

One of the most effective ways to check Nginx’s health and identify any potential issues it might be encountering is by examining its logs. Nginx maintains two primary types of logs: access logs and error logs. These logs are typically stored in the /var/log/nginx/ directory on Linux systems.

Access logs record all incoming requests processed by Nginx, similar to IIS log files in Windows environments. You can open and examine these logs using standard text editors or command-line tools like cat or less. In our scenario, inspecting the access logs might not reveal much new information beyond what we already observed in the browser, such as the “HTTP 502” response status codes encountered during navigation attempts on the BuggyAmb website. However, reviewing them can still be a good practice to confirm the requests are indeed reaching Nginx and the responses it’s sending back.

Error logs, on the other hand, are often more informative for troubleshooting problems. To inspect the error logs, use the command cat /var/log/nginx/error.log. This log file typically contains details about any errors or warnings encountered by Nginx while processing requests or during its own operation. In our crash scenario, examining the error logs is likely to be more helpful and provide clearer insights into the problem.

Upon inspecting the Nginx error logs, you might find entries that indicate Nginx was able to successfully process the incoming request from the browser. However, the crucial detail often revealed in the error log is that the connection between Nginx and the backend BuggyAmb application was unexpectedly closed or terminated before the complete response could be sent back to the client. This log information strongly suggests that the issue is not within Nginx itself but rather in its communication with the backend application. The premature closure of the connection points towards a problem on the application side, such as a crash or unhandled exception that caused the application to terminate abruptly, thus disrupting the communication with Nginx.

This log analysis provides a significant clue, indicating that the “HTTP 502 - Bad Gateway” error is indeed a consequence of the ASP.NET Core application’s failure, rather than a problem with Nginx’s configuration or operation.

Check system logs by using the journalctl command¶

If an ASP.NET Core application is crashing, it is almost certain that symptoms and diagnostic information will be recorded somewhere within the system. Operating systems, especially Linux distributions, are designed to log important events, errors, and application behavior. For applications running as system services, these logs become an invaluable resource for troubleshooting.

Since the BuggyAmb application is configured to run as a system service under systemd, its operational activities, including any crashes or errors, are systematically logged in the system log files. These system log files are analogous to system event logs in Windows environments, providing a centralized repository of system-wide events. In Linux systems that use systemd, the primary tool for viewing and managing these system logs is the journalctl command.

Executing the basic journalctl command without any switches will display a comprehensive view of all system logs. However, the output from this command can be quite extensive, especially on a system that has been running for some time. Navigating through such a large volume of log data to find relevant information can be time-consuming and inefficient. Therefore, it is highly beneficial to learn how to effectively filter the log content using journalctl’s various options and switches. Filtering allows you to narrow down the log output to focus only on the events that are pertinent to your troubleshooting efforts.

For example, you can use the following command to filter logs specifically for the BuggyAmb application within a recent time frame:

journalctl -r --identifier=buggyamb-identifier --since "10 minute ago"

Let’s break down the switches used in this command:

• -r: This switch instructs journalctl to print the logs in reverse chronological order. This is particularly useful for troubleshooting recent issues, as it displays the newest log entries at the top, allowing you to quickly see the most recent events.
• --identifier=buggyamb-identifier: This switch is crucial for filtering logs by a specific application or service. Recall that in the service file configuration for the BuggyAmb application, we included the line SyslogIdentifier=buggyamb-identifier. This line sets a unique identifier for log entries generated by this service. By using --identifier=buggyamb-identifier, we are telling journalctl to only show log entries that are associated with the BuggyAmb application, effectively isolating its logs from other system events.
• --since "10 minute ago": This switch allows you to filter logs based on a time period. --since "10 minute ago" specifically instructs journalctl to display log entries that have been generated within the last 10 minutes. You can adjust the time period as needed, for example, --since "2 hour ago" to see logs from the last two hours, or --since today to see logs from the beginning of the current day.

journalctl offers a wide range of useful switches and options that provide powerful filtering capabilities for system logs. To gain a deeper understanding of all the available options and how to use them effectively, it is highly recommended to consult the journalctl help page. You can access the manual page by running the command man journalctl in your terminal. The manual page provides comprehensive documentation on all aspects of journalctl, including various filtering options, output formats, and configuration settings.

To further refine our log analysis for the BuggyAmb crash, run the following command in your terminal:

journalctl -r --identifier=buggyamb-identifier --since today -o cat

This command combines several useful options:

• -r: Displays logs in reverse order (newest first).
• --identifier=buggyamb-identifier: Filters logs to show only entries from the BuggyAmb application.
• --since today: Limits the logs to entries from the current day onwards.
• -o cat: This output format option is particularly helpful. It simplifies the log output by removing metadata and formatting, presenting just the raw log messages in a concatenated (cat) format. This makes it easier to read and scan through the actual log content.

After executing this command, carefully examine the output. You should notice that some warning messages are present within the logs. These warning messages are often indicative of potential issues or anomalies in the application’s behavior.

To examine the details of these warning messages and any other relevant log entries, use the arrow keys (specifically the down arrow) to scroll down through the log output. As you scroll, pay close attention to the content of the log messages. In this particular crash scenario, you are likely to find a System.Net.WebException exception reported in the logs. This exception type is a strong indicator of a network-related issue within the application, such as a problem with making HTTP requests or handling network connections.

If you meticulously examine the log messages, you might be able to discern the code file name and even the specific line number where the problem originated. This level of detail can be extremely helpful in pinpointing the exact location of the error in the application’s source code. However, for the purposes of this lab and to simulate real-world troubleshooting scenarios, we will assume that this detailed information is not readily available. In many real-world situations, especially in production environments, you might not have immediate access to source code line numbers from logs. Therefore, the objective of the subsequent steps is to demonstrate how to proceed with troubleshooting by analyzing a crash dump file, even when detailed log information is limited. Crash dump analysis provides a powerful alternative approach to diagnosing crashes, especially when source code context is not immediately available or when the crash occurs in complex or third-party code.

Get a core dump file after a crash¶

To effectively diagnose and resolve application crashes, obtaining a core dump file is often a crucial step. A core dump is essentially a snapshot of the application’s memory at the exact moment of a crash. This snapshot contains valuable information about the application’s state, including variable values, call stacks, and loaded modules, which are indispensable for debugging.

It’s important to recall some key system behaviors related to process termination, particularly in Linux environments:

• Default Core Dump Generation: By default, when a process terminates unexpectedly due to a crash (e.g., segmentation fault, unhandled exception), the operating system is configured to generate a core dump file. This default behavior is a fundamental aspect of system diagnostics and crash recovery.
• Core Dump File Location and Naming: The generated core dump file is typically named core by default. The location where this file is created can vary depending on system configuration. It may be placed in the current working directory of the crashing process or in a system-wide directory such as /var/lib/systemd/coredump. The specific location is often determined by system-wide settings or process-specific configurations.

Although the default system behavior is to generate a core dump file upon a crash, this setting can be overridden or customized. In Linux systems, the behavior of core dump generation is often controlled by the /proc/sys/kernel/core_pattern file. This file specifies a pattern that dictates how core dumps are handled. It can be configured to directly pipe the resulting core dump data to another application for processing, such as a crash reporting tool or a core dump management utility.

In Ubuntu, which has been used in previous parts of this series, a utility called apport is responsible for managing core dump file generation. Apport is a crash reporting system that intercepts core dumps and processes them to generate detailed crash reports. In Ubuntu, the /proc/sys/kernel/core_pattern file is typically configured to pipe core dumps to apport. This means that instead of directly creating a raw core file, the system redirects the core dump data to apport for further handling.

Apport, upon receiving a core dump, processes it and stores its report files, along with the core dump data, in the /var/crash folder. If you inspect the contents of the /var/crash folder after an application crash, you are likely to find a file that was generated as a result of the crash. The filename often includes details about the crashed process, such as the executable name and process ID.

However, it’s important to note that the file you find in /var/crash is not the raw core dump file itself. Instead, it is an apport report file. This report file is essentially an archive or container that bundles together several pieces of information related to the crash, including the core dump data along with other diagnostic information that apport collects. To access the actual core dump file, you need to unpack or extract the contents of this apport report file.

To extract the core dump file from the apport report, you can use the apport-unpack command. This command is specifically designed to unpack apport report files and extract their constituent parts, including the core dump.

First, create a directory named dumps in your home folder. This directory will serve as the destination for extracting the contents of the apport report. You can create this directory using the command mkdir ~/dumps.

Next, you will use the apport-unpack command to extract the apport report file into the newly created dumps directory. The command syntax is as follows:

sudo apport-unpack /var/crash/_usr_share_dotnet_dotnet.33.crash ~/dumps/dotnet

Let’s break down this command:

• sudo: apport-unpack might require root privileges to access and unpack the report file located in /var/crash, hence the use of sudo.
• apport-unpack: This is the command itself, which initiates the unpacking process.
• /var/crash/_usr_share_dotnet_dotnet.33.crash: This is the path to the apport report file that you want to unpack. The filename _usr_share_dotnet_dotnet.33.crash is an example and might vary depending on the crashed application and process ID. You should replace this with the actual filename of the report file found in your /var/crash directory.
• ~/dumps/dotnet: This is the destination directory where apport-unpack will extract the contents of the report file. In this case, it will create a subdirectory named dotnet inside the ~/dumps directory and extract the files there.

Executing this apport-unpack command will perform the following actions:

1. If the dumps directory does not already exist in your home folder, the command will create it.
2. The apport-unpack command will then create a subdirectory named dotnet inside the /dumps directory.
3. Finally, it will extract the contents of the specified apport report file into the ~/dumps/dotnet directory.

After the command completes successfully, navigate to the ~/dumps/dotnet directory. Inside this directory, you should find the extracted core dump file. The file is typically named CoreDump (without any file extension). The size of this CoreDump file is expected to be substantial, around 191 MB in this example, as it represents a full memory snapshot of the crashed application.

While extracting the auto-generated core dump file using apport-unpack is a viable method, it can be somewhat cumbersome and involve extra steps. In the next part of this lab, you will explore a more direct and often easier approach to capturing core dump files using the createdump utility. createdump provides more control over the core dump generation process and can simplify the workflow for obtaining core dumps specifically for .NET Core applications.

Next steps¶

Having successfully reproduced a crash, examined relevant logs, and obtained a core dump file, the next crucial phase is to analyze this core dump to pinpoint the root cause of the crash. The subsequent lab, Part 2: Analyzing Core Dumps with lldb Debugger, will guide you through this analysis process. You will learn how to use the lldb debugger in conjunction with the SOS extension to inspect the .NET runtime state within the core dump and identify the underlying issue that led to the application crash. This analysis is key to developing effective fixes and preventing future occurrences of similar crashes.

---

If you found this guide helpful in understanding how to reproduce and gather diagnostic information for ASP.NET Core crashes on Linux, please feel free to leave a comment below! Your feedback and experiences are valuable to the community.