Mastering Core Dumps: Debugging ASP.NET Core Applications Like a Pro

Table of Contents

This comprehensive guide delves into the intricate process of debugging ASP.NET Core applications on Linux using core dump files. It specifically focuses on installing and configuring the lldb debugger, a powerful tool for low-level analysis, and then leveraging it to open and meticulously analyze system-generated .NET Core dump files. This process is crucial for diagnosing and resolving complex issues such as application crashes and performance bottlenecks in production environments.

The techniques discussed herein are applicable to various versions of .NET Core, including .NET Core 2.1, .NET Core 3.1, and .NET 5. Understanding these steps is paramount for any developer or operations professional responsible for maintaining robust ASP.NET Core applications. By the end of this guide, you will have a clear understanding of how to set up your debugging environment and commence the analysis of crash dumps.

Essential Prerequisites for Debugging¶

Before embarking on the debugging journey, certain prerequisites must be in place to ensure a smooth and effective troubleshooting process. These foundational elements provide the environment necessary to reproduce and analyze application behavior, especially in scenarios involving high or low CPU usage and application crashes. Having a controlled environment is key to accurate diagnostics.

The minimum requirement for following these troubleshooting labs is to have a functional ASP.NET Core application that can exhibit specific performance problems, such as high-CPU consumption or application crashes. Various sample applications are available online to simulate these undesirable behaviors. For instance, Microsoft’s simple webapi sample or the BuggyAmb ASP.NET Core application can serve as excellent demonstration projects. These applications are designed to be intentionally problematic, making them ideal for debugging exercises.

If you have been following the previous parts of this series, your environment should already be configured with the following setup:

• Nginx Configuration: Nginx, a high-performance web server, should be configured to host two distinct websites. The first website should respond to requests using the myfirstwebsite host header (e.g., http://myfirstwebsite) and efficiently route these requests to a demo ASP.NET Core application listening on port 5000. The second website should similarly listen for requests via the buggyamb host header (e.g., http://buggyamb), forwarding them to a second ASP.NET Core sample application, specifically the buggy application, which listens on port 5001. This setup simulates a common production environment with multiple applications.
• Application Services: Both ASP.NET Core applications must be configured to run as services. This ensures that they automatically restart if the server reboots or if the application unexpectedly stops responding, maintaining application availability. Running applications as services is a standard practice for production deployments.
• Firewall Configuration: The Linux local firewall should be enabled and properly configured to permit both SSH and HTTP traffic. SSH access is vital for remote management and debugging, while HTTP traffic is essential for the web applications to be accessible. Proper firewall rules ensure security without hindering necessary access.

To proceed with this lab effectively, it is imperative to have at least one problematic ASP.NET Core web application running behind the Nginx reverse proxy. This allows you to generate and then analyze a core dump related to a real issue. Without an actively misbehaving application, the core dump analysis will lack a specific context for troubleshooting.

Objective of This Debugging Lab¶

This article serves as the second installment in a two-part lab series dedicated to debugging crashes within ASP.NET Core applications deployed on Linux. Understanding the context from the previous part is crucial for a complete learning experience, as it builds upon the initial steps of crash reproduction and preliminary troubleshooting.

In the preceding Lab 1.1: Reproduce and troubleshoot a crash problem, you meticulously followed a series of steps designed to reproduce a crash problem within your ASP.NET Core application. During that lab, you initiated the troubleshooting process by examining Nginx logs and system logs for clues regarding the crash. A significant step involved gathering and analyzing a memory dump file, specifically extracting the crash core dump file that was automatically generated by apport, Ubuntu’s default core dump file manager. While apport provides a convenient mechanism for capturing these dumps, their utility for deep debugging can be limited without further tools and configuration.

This current part of the lab focuses on advancing your debugging capabilities. Here, you will systematically install and configure the lldb debugger, a powerful command-line debugger. Furthermore, you will integrate lldb with the .NET Core debugger extension known as SOS (Son of Strike). Once configured, the primary objective is to open the previously extracted core dump file within lldb and commence its in-depth analysis. This integration will enable you to inspect the managed state of your .NET application, providing critical insights into the crash’s root cause.

Installing the LLDB Debugger¶

The lldb debugger is an essential tool for low-level debugging on Linux, especially when dealing with core dump files from .NET Core applications. For this lab, it is imperative that you install lldb version 3.9 or a later release to ensure compatibility with the .NET Core debugging extensions. The installation process is straightforward, and instructions are widely available for various Linux distributions.

A convenient and recommended method for installation on Ubuntu-based systems is through the apt package manager. You can execute the command: sudo apt install lldb. This command will automatically download and install lldb along with any necessary dependencies. The package manager handles resolving library requirements, making the process seamless.

Upon successful execution, you will typically see output confirming the installation of lldb (e.g., lldb-6.0 or a newer version) and its associated packages. This indicates that the core lldb debugger is now present on your system. With the debugger installed, the next crucial step is to configure it properly to load the necessary .NET Core debugging extensions automatically when a core dump file is opened, preparing it for in-depth analysis.

Configuring LLDB for .NET Core Debugging¶

After installing the lldb debugger, a critical phase involves configuring it to work seamlessly with .NET Core applications. This configuration ensures that lldb can effectively interpret the managed code structures within your core dump files. Before you proceed to open any core dump file in lldb, it is essential to follow these required steps: setting the symbol path, downloading the relevant symbols, and ensuring that the SOS extension automatically loads upon lldb initialization.

The configuration process involves several sequential steps, each crucial for a successful debugging experience. These steps automate the setup that would otherwise be manual and error-prone during each debugging session.

1. Install the dotnet-symbol tool: This command-line tool is fundamental for downloading debugging symbols required to resolve function names, source line information, and local variable names within the debugger. To install it globally, execute:
   

   dotnet tool install -g dotnet-symbol
   

   
   This command fetches and installs the dotnet-symbol tool, making it accessible from any directory in your terminal. It’s a one-time setup that significantly enhances the debugger’s ability to provide meaningful information.

2. Download the symbols for the target dump file: Once dotnet-symbol is installed, you can use it to download the specific symbols pertinent to your core dump. These symbols provide the necessary mapping between the compiled binaries and your original source code. The command format is:
   

   dotnet-symbol <path_of_dump_file>
   

   
   Replace <path_of_dump_file> with the actual path to your core dump file. This step is critical because without the correct symbols, the debugger will display raw memory addresses and assembly code, which is much harder to interpret.

3. Install SOS (Son of Strike): SOS is the managed debugging extension that provides a rich set of commands for inspecting the managed state of .NET applications. It integrates directly with lldb to allow inspection of the .NET runtime.

   1. Install the dotnet-sos global tool: This tool facilitates the installation and management of the SOS extension.
      

      dotnet tool install -g dotnet-sos
      

      
      This command installs the utility that manages the SOS extension, preparing your environment for its deployment.
   2. Install SOS: With the dotnet-sos tool in place, you can now install the SOS extension itself.
      

      dotnet-sos install
      

      
      This command performs the actual installation of the SOS extension, configuring lldb to automatically load it whenever a core dump file is opened. This automatic loading streamlines the debugging workflow, ensuring that .NET-specific commands are immediately available.

Completing these steps prepares your lldb environment to fully leverage the power of .NET Core debugging extensions, enabling you to dive deep into the intricacies of your application’s state at the time of a crash.

Installing the dotnet-symbol Tool¶

The dotnet-symbol tool is a crucial component in the .NET Core debugging toolkit, primarily responsible for downloading the necessary symbol files. These symbol files are paramount for debuggers to translate low-level memory addresses into human-readable information, such as function names, variable names, and source code line numbers. Without them, debugging becomes significantly more challenging, as you would be navigating raw machine code.

It is possible that you might have already installed the dotnet-symbol tool, along with other essential diagnostic tools like dotnet-dump and dotnet-gcdump, during the earlier stages of this debugging series. These tools collectively form a powerful suite for diagnosing various issues in .NET Core applications. However, if you haven’t installed them yet, now is the opportune moment to do so before proceeding with the core dump analysis.

To ensure dotnet-symbol is installed, execute the command: dotnet tool install -g dotnet-symbol. This command installs the tool globally, making it accessible from any directory. Furthermore, if you haven’t already, also install dotnet-dump and dotnet-gcdump using similar commands (dotnet tool install -g dotnet-dump and dotnet tool install -g dotnet-gcdump). These two tools are valuable for capturing memory dumps and garbage collection dumps, respectively, offering different perspectives on application state.

Upon successful installation, you should have all three diagnostic tools—dotnet-symbol, dotnet-dump, and dotnet-gcdump—available on your system. This comprehensive set of tools empowers you to effectively capture, analyze, and interpret various types of diagnostic data from your .NET Core applications, setting the stage for thorough problem resolution.

Downloading Symbols for the Dump File¶

With the dotnet-symbol tool now installed, the next pivotal step in preparing your core dump for effective analysis is to download the corresponding symbol files. As previously highlighted in debugging contexts, symbols are abstract representations that operate at a higher level than compiled binaries. They serve as vital mappings between the original source code and the compiled binaries, enabling debuggers to translate cryptic memory addresses into meaningful insights. When a debugger reads a call stack, these mappings are utilized to resolve function or method names, retrieve source line information, and identify local variable names, making the debugging process significantly more intuitive.

In Part 1 of this lab, you learned how to unpack the core dump file from an apport report, which typically extracts the raw dump file to a specified location. Now, it’s time to enrich that raw dump with the necessary symbolic information. You will use the dotnet-symbol tool for this purpose, directing it to download symbols specifically for your memory dump file.

The command to download the symbols is: dotnet-symbol ~/dumps/dotnet/CoreDump -o ~/dumps/symbols --host-only. This command specifies the path to your core dump file (~/dumps/dotnet/CoreDump) and instructs dotnet-symbol to output the downloaded symbols to the ~/dumps/symbols directory. The --host-only switch is particularly important and often overlooked.

During the symbol download process, you might encounter several “HTTP 404 not found” error messages. These messages often appear when dotnet-symbol attempts to download symbols for every module loaded into the process, many of which might not have publicly available symbols or are not relevant for managed debugging. You can safely ignore these 404 errors, especially when using the --host-only switch. The --host-only parameter ensures that only the symbols for the host program—typically the dotnet executable or your self-contained application’s executable—are downloaded. This is precisely what lldb requires to commence debugging the ASP.NET Core application and is sufficient for most managed debugging scenarios.

The successful completion of this step provides the debugger with the critical contextual information it needs to display meaningful call stacks and variable values. The next logical step is to install the SOS-managed debugging extension, which will expose the full suite of .NET-specific debugging commands essential for in-depth analysis of the application’s managed state.

Installing SOS: The Managed Debugging Extension¶

SOS, or Son of Strike, is an indispensable debugger extension for anyone performing in-depth analysis of .NET applications. According to its official documentation, SOS empowers developers to inspect the managed state of a .NET application. This capability extends to a wide range of .NET-based applications, including ASP.NET Core, .NET WPF, and .NET Windows Forms applications. Essentially, SOS acts as a bridge, allowing native debuggers like WinDbg (on Windows) or lldb (on Linux and macOS) to understand and interact with the .NET runtime and its managed data structures. It provides specialized commands for examining managed heaps, objects, threads, and exceptions.

To fully leverage SOS within your lldb environment, you must first install the dotnet-sos global tool. This tool acts as a utility to facilitate the installation and management of the SOS extension itself. You can install it by executing the following command in your terminal:

dotnet tool install -g dotnet-sos

This command downloads and installs the dotnet-sos manager, making it globally available on your system. Once the dotnet-sos tool is successfully installed, the next step is to use it to install the SOS extension. This is achieved with a simple command:

dotnet-sos install

This command performs the actual installation of the SOS extension, integrating it with your lldb debugger. A key benefit of using dotnet-sos install is its ability to automatically configure the lldb debugger. This configuration ensures that the SOS extension is loaded automatically every time lldb starts, especially when you open a core dump file. This automatic loading eliminates the need for manual configuration during each debugging session, significantly streamlining your workflow.

The screenshot after running dotnet-sos install typically confirms a successful installation and explicitly states that the lldb debugger has been configured for automatic SOS loading. This confirmation is vital, as it signifies that all the necessary preparations are complete. With SOS successfully installed and configured, you are now fully equipped and ready to open the core dump file using lldb and begin the actual process of debugging and analysis.

Opening the Core Dump in LLDB¶

Having meticulously prepared your environment by installing lldb, downloading symbols, and configuring SOS, you are now ready to open the core dump file for analysis. This is the moment where all the setup culminates in practical debugging. To open a core dump with lldb, you must use a specific command syntax that provides both the path to the dump file and the host program that initiated the .NET Core application.

The general syntax for opening a core dump in lldb is as follows:

lldb --core <dump path> <host-program>

Here, <dump path> refers to the full path to your core dump file, which you extracted in the previous lab part. The <host-program> parameter is crucial; it represents the native executable that started the .NET Core application. In most scenarios for ASP.NET Core applications, this will be the dotnet executable. However, if your application is self-contained (meaning it bundles the .NET runtime with the application), then <host-program> would be the name of your application’s executable file, typically without the .dll extension. Identifying the correct host program is essential for lldb to correctly load the debugging context.

Assuming you have followed the recommended folder structure and naming conventions from the previous sections, the path to your memory dump file should be ~/dumps/dotnet/CoreDump. Given that the application is likely run by the dotnet host, the complete command to open the file would be:

lldb --core ~/dumps/dotnet/CoreDump dotnet

Upon executing this command, lldb will initiate and attempt to load the core dump file. If successful, you will be presented with the lldb debugger prompt, indicating that the dump file has been opened and the debugger is ready to accept commands. This is a significant milestone, as it means you have successfully loaded the crash snapshot into your debugging tool.

The screenshot depicts the lldb debugger successfully opening the memory dump file, displaying introductory messages and the prompt. At this stage, while the dump is loaded, the debugger might not yet have full symbolic information or the SOS extension fully activated. The next steps will involve confirming and setting the symbol path and ensuring SOS is ready to provide managed debugging capabilities.

Setting Symbol Paths within LLDB¶

Once the core dump file has been successfully loaded into lldb, the immediate next step is to inform the debugger where to find the symbol files you previously downloaded using the dotnet-symbol command. These symbols are vital for lldb and SOS to provide meaningful output by translating cryptic memory addresses into readable function names, variable values, and source code line numbers. Without correctly setting the symbol path, much of the debugger’s output will remain obscure.

Recall that you downloaded these symbol files into the ~/dumps/symbols directory. The first command you should execute inside the lldb debugger is setsymbolserver -directory ~/dumps/symbols. This command instructs lldb to configure its symbol server to look for symbol files in the specified directory. It’s crucial to ensure that the path provided here exactly matches the output directory used during the dotnet-symbol download process.

After setting the symbol server directory, the next command you need to run is loadsymbols. This command triggers lldb to actively load the symbols from the configured path into its memory. It allows the debugger to parse the symbol files and create the necessary internal mappings. Without loadsymbols, merely setting the directory is insufficient, as the symbols won’t be actively used by the debugger.

(lldb) setsymbolserver -directory ~/dumps/symbols
(lldb) loadsymbols

The output from these commands typically confirms that the symbol server has been configured and that symbols are being loaded. Although lldb often attempts to resolve some basic symbols automatically, explicitly setting the path and loading them ensures that all available managed and native symbols are accessible. This greatly enhances the depth and clarity of the debugging information you will receive when inspecting call stacks and managed objects, preparing you for the powerful diagnostic capabilities offered by SOS.

Executing LLDB and SOS Commands for Analysis¶

With lldb loaded, symbols configured, and SOS ready, you can now begin to execute various commands to analyze the core dump. lldb offers a wide array of commands for low-level debugging, and the SOS extension layers on top of this with commands specifically designed for .NET applications. To get an initial overview of available commands, you can simply type help at the lldb prompt.

Upon running the help command, lldb will display a comprehensive list of its built-in commands. What’s particularly useful for .NET debugging is that the SOS commands are also integrated into this list, typically appearing under a section like “user defined commands” or “plugin commands.” This seamless integration means you don’t need to switch contexts to use SOS; its commands are available directly from the lldb prompt. You can also get more specific help on SOS commands by typing help sos or help <sos_command_name>.

To begin your analysis, it’s often insightful to first examine the native call stack of the active thread. The bt (short for “back trace”) command in lldb serves this purpose. It displays the sequence of function calls that led to the current point of execution for the currently selected thread. This can reveal the native context of the crash, including calls into operating system libraries or native components of the .NET runtime.

Next, for .NET-specific insight, you would typically examine the managed call stack using the SOS clrstack command. This command is designed to display the call stack of managed functions for the current thread, providing a view of the .NET methods that were active at the time of the dump.

However, upon executing clrstack on a system-generated core dump (like those produced by apport), you might notice that you are unable to retrieve any meaningful information. The stack walk will likely fail, or it will report an incomplete stack. This is a common limitation of automatically generated core dump files; they often do not collect all the necessary managed states or register contexts required by SOS to construct a complete and accurate managed call stack. This limitation underscores the challenge of relying solely on default system dumps for deep .NET debugging.

Recall that an exception occurred, leading to the process crash. To investigate the nature of this exception, you can use the SOS pe command (short for “PrintException”). This command is designed to display information about the last exception that was thrown on the current thread, if any.

As you can observe from the output of the pe command, it successfully indicates that an exception occurred. The exception message in this particular case is “resource temporarily unavailable”. However, a significant drawback here is that the type of exception and the specific function names associated with it are not resolved; instead, their values are indicated as “unknown”. This again points to the limitations of the system-generated dump’s capture fidelity.

The pe command also displays the memory address of the exception object. You might attempt to pass this address as a parameter to the pe command to see if more granular details are available, for instance: pe 00007F8244048538. (Remember to replace the example address with the actual address displayed in your dump file’s output, as addresses are specific to each dump.)

Unfortunately, even when attempting to display the objects referenced in the stack or providing the explicit exception address, you will likely encounter the same recurring issue: values are consistently shown as “unknown”. This indicates that the necessary metadata or object structure information is either incomplete or entirely missing within this particular dump file.

You might then try to retrieve more information by selecting the address of one of the seemingly relevant objects on the stack (even if its type is “unknown”) and reviewing the object’s contents using the SOS dumpobj <address> command. However, you will likely reach the same conclusion: this command also has a limited effect, returning only more “unknown” messages or minimal, unhelpful data.

At this juncture, it becomes evident that while system-generated dump files provide some basic information and confirm a crash, they severely lack the crucial managed state and symbolic detail required for in-depth root cause analysis in .NET Core applications. The data collected by these default dumps is insufficient for comprehensively understanding the managed call stack, exception types, and object states. It is now time to conclude the session with this limited auto-generated dump file. You can gracefully exit the lldb session by typing the exit command.

In summary, although the dump file generated by the system gives you initial information about a crash, it leaves many important details unresolved. The inability to retrieve complete managed call stacks, precise exception types, and detailed object information significantly hinders effective debugging. This limitation highlights the need for a more robust and controlled method for capturing crash dumps.

Next Steps¶

This lab has demonstrated the process of setting up lldb and SOS to analyze a system-generated core dump, while also exposing the inherent limitations of such dumps for comprehensive .NET Core debugging. While these dumps can confirm a crash and provide some low-level native context, their lack of detailed managed state information often makes root cause analysis challenging and incomplete.

To overcome these limitations and truly master the art of debugging ASP.NET Core application crashes, the next logical step is to explore more reliable methods for capturing crash dumps. The recommended approach involves using specialized .NET Core diagnostic tools that are designed to capture a complete snapshot of the managed process state, ensuring that all necessary information—including managed call stacks, heap contents, and detailed exception information—is present and interpretable by tools like lldb with SOS.

Understanding and implementing these advanced dump capture techniques will be the focus of the subsequent part of this series. This will enable you to move beyond the “unknown” values and gain the deep insights necessary to pinpoint the exact cause of application crashes, transforming you into a truly proficient debugger of ASP.NET Core applications.

---

We hope this detailed guide has provided valuable insights into debugging ASP.NET Core applications using core dumps and lldb. What are your experiences with debugging crashes in production? Have you encountered similar limitations with system-generated dumps? Share your thoughts, challenges, or tips in the comments below!