Home azure debugging system administration virtual machines

Unlock Deeper VM Control: Azure Serial Console for SysRq & NMI Diagnostics

Updated: • 0 • min read
Table of Contents

Azure Virtual Machines (VMs) offer robust computational power, but like any server, they can encounter situations where the operating system becomes unresponsive. In such critical scenarios, traditional remote administration tools like SSH might fail, leaving administrators without a clear path to diagnose or recover their systems. This is where the Azure Serial Console, combined with powerful Linux kernel features like System Request (SysRq) and Non-Maskable Interrupt (NMI), becomes an indispensable tool for regaining control and performing vital diagnostics.

The Azure Serial Console provides an out-of-band management interface, acting as if you have a direct physical connection to the VM’s serial port. This means it bypasses the network stack and operating system’s normal functioning, offering a low-level access point critical for troubleshooting kernel-level issues. It’s a lifeline for situations where a VM appears hung, frozen, or is otherwise inaccessible through conventional means. This article delves into how SysRq and NMI can be leveraged through the Azure Serial Console to perform advanced diagnostics and recovery operations on your Linux VMs.

System Request (SysRq)

A System Request (SysRq) is a sequence of keys that the Linux kernel understands, capable of triggering a variety of pre-defined actions. These commands are invaluable when a virtual machine becomes unresponsive and traditional administration methods are no longer viable. For instance, if a VM appears to be frozen or experiencing a kernel panic (though SysRq cannot be used if the kernel is completely non-responsive in a panic state, as it relies on some kernel responsiveness). Using the SysRq feature within the Azure Serial Console simulates the pressing of the SysRq key and subsequent character on a physical keyboard, delivering direct commands to the kernel.

The kernel’s configuration dictates how the system responds to a SysRq sequence. Administrators have the flexibility to enable all, none, or only specific SysRq commands, providing a balance between powerful diagnostic capabilities and system security. Understanding how to configure and utilize SysRq is crucial for effective emergency troubleshooting and ensuring system stability in a cloud environment.

Leveraging SysRq via Azure Serial Console

The Azure Serial Console simplifies the process of sending SysRq commands to an Azure virtual machine. Within the console interface, a dedicated keyboard icon typically presents options to send SysRq commands. This functionality is crucial because cloud VMs lack physical keyboards, making direct key sequences impossible.

Azure Serial Console keyboard icon

Selecting the “Send SysRq Command” option typically opens a dialog box. This dialog offers a selection of common SysRq options, making it easy to execute frequently used commands. Alternatively, it allows users to enter a custom sequence of SysRq commands, providing maximum flexibility. This capability is particularly useful for executing high-level operations, such as a graceful system reboot using the well-known REISUB sequence.

It’s important to remember that the SysRq command cannot be utilized if the virtual machine is in a stopped state or if the kernel is in an utterly non-responsive state, such as a complete kernel panic where no processing can occur. In these extreme cases, other recovery methods, such as VM restart from the Azure portal, might be necessary. However, for most unresponsive but still somewhat active kernels, SysRq provides a powerful diagnostic path.

Enabling SysRq

To fully utilize the power of SysRq, it must be appropriately configured on your Linux VM. The SysRq Admin Guide details various configuration options, allowing you to control which commands are accessible. For diagnostic purposes, it’s often beneficial to enable all SysRq commands, though this can be reverted or restricted after troubleshooting.

For a temporary enablement that does not persist across reboots, you can execute the following command:

echo "1" >/proc/sys/kernel/sysrq

This command writes the value ‘1’ to the /proc/sys/kernel/sysrq file, effectively enabling all SysRq functions until the next system restart. For persistent configuration, especially in environments where robust recovery capabilities are paramount, a more permanent solution is recommended. This involves modifying a system configuration file.

To persistently enable all SysRq commands, follow these steps:

  1. Add the line kernel.sysrq = 1 to the /etc/sysctl.conf file. This file manages kernel parameters that are applied at boot time.
  2. After modifying sysctl.conf, either reboot the virtual machine or update the sysctl settings immediately by running the command sysctl -p. The sysctl -p command reloads parameters from sysctl.conf, applying the changes without requiring a full system reboot.

Common SysRq Command Keys

The SysRq facility offers a wide array of commands, each designed to perform a specific action within the kernel. These commands are typically single characters, making them quick to type and execute during critical situations. Understanding these commands is key to effective system recovery and diagnostics.

Below is a comprehensive list of SysRq command keys and their functions:

| Command | Function A SysRq is a sequence of keys understood by the Linux operation system kernel, which can trigger a set of pre-defined actions. These commands are often used when virtual machine troubleshooting or recovery can’t be performed through traditional administration. For example, if the VM isn’t responding. Using the SysRq feature of Azure Serial Console mimics pressing of the SysRq key and characters entered on a physical keyboard.

Once the SysRq sequence is delivered, the kernel configuration controls how the system responds. For information on enabling and disabling SysRq, see the SysRq Admin Guide.

The Azure Serial Console can be used to send a SysRq to an Azure virtual machine using the keyboard icon in the following command bar.

Choosing “Send SysRq Command” opens a dialog, which provides common SysRq options or accept a sequence of SysRq commands entered into the dialog. This “Send SysRq Command allows for series of SysRq’s to perform a high-level operation such as a safe reboot using: REISUB.

The SysRq command can’t be used on virtual machines that are stopped or whose kernel is in a non-responsive state. For example, a kernel panic.

Enable SysRq

As described in the SysRq Admin Guide, SysRq can be configured such that all, none, or only certain commands are available. You can enable all SysRq commands using the following step, but it doesn’t survive a reboot:

echo "1" >/proc/sys/kernel/sysrq

To make the SysReq configuration persistent, you can do the following to enable all SysRq commands

  1. Adding this line to /etc/sysctl.conf
    kernel.sysrq = 1
  2. Rebooting or updating sysctl by running
    sysctl -p

Command Keys

From the SysRq Admin Guide:

| Command | Function
| b | Immediately reboots the system without syncing or unmounting your disks. Used for an immediate, less graceful reboot. |

The user requires a rewrite of the provided article content in a professional and informative English style, formatted in markdown.
I need to expand the content to reach a minimum of 1000 words if possible, add relevant supporting media (tables, diagrams, YouTube videos), and end with a call to action.
No introductory text or comments. Directly start with the # heading.

Let’s begin.

System Request (SysRq)

A System Request (SysRq) is a potent sequence of keys recognized directly by the Linux operating system kernel. This special command mechanism can trigger a predefined set of actions, providing a critical lifeline for system administrators when a virtual machine becomes unresponsive and traditional access methods, such as SSH, are no longer functional. Scenarios where SysRq proves invaluable include a frozen VM, an unresponsive graphical interface, or situations where kernel processes are deadlocked. It allows for advanced troubleshooting and recovery that bypasses most user-space applications and directly interacts with the kernel itself.

In the cloud environment, where direct physical keyboard access to a virtual machine is impossible, the Azure Serial Console bridges this gap. The SysRq feature within the Azure Serial Console precisely mimics the act of pressing the physical SysRq key along with a subsequent character. This simulation delivers the command sequence directly to the underlying Linux kernel, enabling administrators to execute diagnostic and recovery operations remotely. It’s an essential tool for maintaining high availability and resolving critical issues in Azure-hosted Linux VMs.

Once a SysRq sequence is successfully delivered to the VM, the system’s behavior is governed by the kernel’s configuration. This level of control allows administrators to predefine how the system should react to such commands, whether for immediate reboot, memory dump collection, or process termination. Comprehensive details on how to enable, disable, and fine-tune SysRq functionalities are extensively documented in the official SysRq Admin Guide. Administrators are encouraged to consult this guide to understand the nuances of SysRq configuration and ensure their systems are prepared for various troubleshooting scenarios.

The Azure Serial Console integrates this powerful feature seamlessly into its interface. Within the command bar, a distinctive keyboard icon is provided, specifically designed to initiate SysRq operations. This user-friendly access point eliminates the need for complex command-line gymnastics for sending these kernel-level signals, making it accessible even during stressful outage situations.

Choosing the “Send SysRq Command” option from the console’s interface will typically prompt a dedicated dialog box. This dialog not only presents a list of commonly used SysRq options for quick selection but also offers the flexibility for users to input a custom sequence of SysRq commands. This advanced capability is particularly beneficial for executing multi-step operations, such as performing a controlled and safe reboot using the widely recognized REISUB sequence. This sequence systematically handles processes, syncs disks, and then reboots, minimizing data loss and corruption risk.

It’s paramount to understand the limitations of SysRq. This powerful diagnostic tool cannot be employed on virtual machines that are in a stopped or deallocated state, as there is no active kernel to respond to commands. Furthermore, if the Linux kernel enters a completely non-responsive state—such as a hard kernel panic where it ceases all processing—SysRq commands will also be ineffective. In such extreme cases, administrators might need to resort to harder resets or reboots initiated directly from the Azure portal or management plane.

The REISUB Sequence: A Graceful Reboot in Crisis

One of the most valuable SysRq sequences for system recovery is REISUB (sometimes remembered by the mnemonic “Raising Elephants Is So Utterly Boring”). This sequence is designed to perform a controlled and safe system reboot, even when the system appears frozen, by systematically taking steps to prevent data loss.

Let’s break down the REISUB sequence:

  • R (Raw): Unraws the keyboard from XLATE mode. This ensures the kernel correctly interprets subsequent SysRq commands.
  • E (Terminate): Sends a SIGTERM to all processes, except init. This allows processes to gracefully shut down, saving their state.
  • I (Kill): Sends a SIGKILL to all processes, except init. This forcefully terminates any processes that did not respond to SIGTERM.
  • S (Sync): Attempts to sync all mounted filesystems to disk. This is crucial for preventing data corruption by writing all pending data from cache to storage.
  • U (Unmount): Attempts to remount all mounted filesystems read-only. This prevents further writes to the disk, safeguarding data integrity before a reboot.
  • B (Reboot): Immediately reboots the system. With the previous steps, this reboot is significantly safer than a hard power cycle.

Executing this sequence via the Azure Serial Console offers a much more controlled recovery path than merely force-restarting a VM, which can lead to filesystem corruption or data loss.

Enabling SysRq Configuration

For SysRq commands to be active and usable, the feature must be enabled within the Linux kernel. The level of SysRq functionality can be configured, allowing administrators to choose between enabling all commands, restricting to specific ones, or disabling it entirely. While disabling SysRq might offer a marginal security benefit by preventing unauthorized kernel interaction, its utility in emergency diagnostics often outweighs this.

To enable all SysRq commands temporarily, which is useful for one-time troubleshooting sessions and does not persist across system reboots, execute the following command in the shell:

echo "1" > /proc/sys/kernel/sysrq

This command directly modifies a kernel parameter exposed through the /proc filesystem. A value of 1 signifies that all SysRq functions are enabled. Other values can be used to enable specific subsets of commands, offering granular control.

For a permanent SysRq configuration that endures system reboots, modifying the /etc/sysctl.conf file is the recommended approach. This file is designed to hold persistent kernel parameters.

Here are the steps to enable all SysRq commands permanently:

  1. Edit /etc/sysctl.conf: Open the /etc/sysctl.conf file using a text editor (e.g., nano or vi) and add the following line:
    kernel.sysrq = 1

    If a kernel.sysrq entry already exists, simply modify its value to 1.
  2. Apply Changes: After saving the changes to /etc/sysctl.conf, you have two options to apply them:
    • Reboot the system: A full system reboot will read all parameters from /etc/sysctl.conf and apply them.
    • Update sysctl at runtime: To apply the changes immediately without a reboot, execute the command:
      sysctl -p

      This command instructs sysctl to reload all parameters from /etc/sysctl.conf, making the SysRq functionality available instantly.

The choice between temporary and persistent enablement often depends on the environment’s security policies and the frequency of troubleshooting needs. For production systems, persistent enablement ensures that diagnostic capabilities are always ready when required.

SysRq Command Keys Reference

Understanding each SysRq command is crucial for effective emergency troubleshooting. Here is a detailed reference of the command keys and their respective functions, derived from the SysRq Admin Guide:

| Command | Function |

SysRq Flow Chart

This flow chart illustrates the basic decision tree for utilizing SysRq when a Linux VM becomes unresponsive.
mermaid graph TD A[Linux VM Unresponsive] --> B{Can you access Azure Serial Console?}; B -- Yes --> C{SysRq enabled on VM?}; B -- No --> D[Check network, Azure Health, VM status]; C -- No --> E[Enable SysRq temporarily (echo "1" > /proc/sys/kernel/sysrq)]; C -- Yes --> F{Identify the problem/desired action}; E --> F; F -- Crash dump --> G[Send SysRq 'c']; F -- Kill processes --> H[Send SysRq 'e' then 'i']; F -- Sync filesystems --> I[Send SysRq 's']; F -- Reboot gracefully --> J[Send SysRq 'REISUB']; F -- Other diagnostics --> K[Send relevant SysRq command]; G --> L[Collect crash dump for analysis]; H --> M[Check VM responsiveness]; I --> M; J --> N[VM reboots, check accessibility]; K --> O[Analyze output in Serial Console];
This diagram offers a simplified representation; actual troubleshooting might involve iteration and combining multiple commands.

Distribution-Specific Documentation for Crash Dumps

For robust post-mortem analysis of system failures, it is often necessary to configure Linux to generate a crash dump when it receives a SysRq “Crash” command (c). These crash dumps contain the system’s memory state at the time of the crash, which is invaluable for identifying the root cause of kernel panics or other critical system malfunctions. The configuration steps for enabling crash dump generation can vary slightly between different Linux distributions.

Here are specific references for configuring crash dump capture on popular Linux distributions:

Ubuntu

  • Kernel Crash Dump: Ubuntu’s documentation provides detailed instructions on setting up kdump, a reliable kernel crash dumping mechanism. This ensures that memory dumps are captured effectively for analysis.

Red Hat Enterprise Linux (RHEL) / CentOS

  • What is the SysRq Facility and how do I use it?: This article explains the fundamental concepts of SysRq and its practical applications on RHEL systems.
  • How to use the SysRq facility to collect information from a RHEL server: Provides guidance on using SysRq for information gathering and diagnostics.
  • How to configure RHEL to create a crash dump on SysRq ‘c’: Specific instructions on enabling kdump for SysRq-triggered crash dumps.

SUSE Linux Enterprise Server (SLES)

  • Configure kernel core dump capture: SUSE’s knowledge base offers comprehensive steps for setting up kdump on SLES, ensuring that kernel core dumps are generated and stored correctly.

CoreOS

  • Collecting crash logs: CoreOS documentation provides insights into how crash logs, including kernel crash information, can be collected and utilized for diagnostics in their container-optimized operating system.

Consulting these distribution-specific resources is crucial to ensure that your crash dump configuration is correct and that valuable diagnostic data can be retrieved when needed.

Non-Maskable Interrupt (NMI)

A Non-Maskable Interrupt (NMI) represents a hardware-generated signal that, unlike standard interrupts, cannot be ignored or “masked” by the operating system software. Historically, NMIs served as a critical mechanism for monitoring and reacting to severe hardware issues on systems that demanded immediate and specific responses, such as parity errors or critical temperature thresholds. Today, while still used for hardware anomaly detection, programmers and system administrators frequently employ NMI as a robust mechanism to debug or troubleshoot systems that have become completely unresponsive, even to software-level interrupts. Because an NMI cannot be ignored by the CPU, it forces an immediate reaction, making it an ideal trigger for collecting diagnostic information when the system is otherwise locked up.

The Azure Serial Console extends its diagnostic capabilities to include sending an NMI to an Azure virtual machine. Similar to SysRq, this is facilitated through the keyboard icon in the console’s command bar. Once the NMI signal is delivered, the VM’s configured response dictates its subsequent action. Linux operating systems can be specifically configured to react to an NMI by crashing and generating a memory dump, which is crucial for deep-level debugging and post-mortem analysis of severe system hangs or kernel failures. This allows administrators to obtain vital forensic data even from a seemingly completely frozen system.

Azure Serial Console NMI button

Enabling NMI Triggered Panics

For Linux systems, leveraging sysctl to configure kernel parameters provides a straightforward method to enable a system panic upon receiving an NMI. This configuration ensures that when an NMI is sent via the Azure Serial Console, the VM will intentionally crash and, if properly configured with a crash dump utility like kdump, will generate a memory dump. This dump is an invaluable artifact for root cause analysis.

To configure a Linux system to panic when an NMI is received, follow these steps:

  1. Add Configuration to /etc/sysctl.conf: Open the /etc/sysctl.conf file using your preferred text editor and add the following line:
    kernel.panic_on_unrecovered_nmi=1

    This parameter instructs the kernel to panic if an NMI is received and not handled by any other NMI handler. Setting it to 1 ensures a crash, leading to a potential crash dump capture.
  2. Apply Changes: After saving the changes to /etc/sysctl.conf, you must apply them for the setting to take effect. This can be done by:
    • Rebooting the virtual machine: A full system reboot will load all sysctl parameters, including the new NMI setting.
    • Updating sysctl at runtime: To apply the configuration immediately without a reboot, execute the command:
      sysctl -p

      This command reloads the kernel parameters from /etc/sysctl.conf, making the panic_on_unrecovered_nmi setting active.

For a deeper understanding of Linux kernel NMI configurations, including unknown_nmi_panic, panic_on_io_nmi, and panic_on_unrecovered_nmi, consult the official kernel documentation for /proc/sys/kernel/*. These parameters offer nuanced control over how the kernel handles various NMI conditions.

Visualizing NMI Panic Flow

Here’s a simple diagram illustrating how an NMI can trigger a kernel panic and subsequent crash dump.
mermaid graph TD A[Azure Serial Console] --> B(Send NMI Command); B --> C[Linux VM Kernel]; C --> D{NMI Handler Configured?}; D -- No (kernel.panic_on_unrecovered_nmi=1) --> E[Kernel Panics]; D -- Yes (e.g., specific hardware NMI handler) --> F[Execute NMI Handler]; E --> G[kdump utility captures memory]; G --> H[Crash Dump File Generated]; H --> I[Analysis for root cause]; F --> J[System continues or takes pre-defined action];
This flow highlights the diagnostic path, particularly when the kernel is set to panic on an unhandled NMI, which is critical for collecting post-mortem data.

Distribution-Specific Documentation for NMI and Crash Dumps

Configuring a Linux system to generate a crash dump upon receiving an NMI is a vital step for advanced troubleshooting. The specific procedures for enabling this functionality, and for setting up the kdump service which collects these memory dumps, can vary across different Linux distributions. It is essential to refer to the documentation provided by your specific distribution to ensure correct implementation.

Here are tailored references for popular Linux distributions:

Ubuntu

  • Kernel Crash Dump: Ubuntu’s server guide provides comprehensive documentation on setting up kdump and configuring kernel crash dump capture. This is crucial for enabling NMI-triggered dumps.

Red Hat Enterprise Linux (RHEL) / CentOS

  • What is an NMI and what can I use it for?: This Red Hat article offers an overview of NMIs and their diagnostic utility on RHEL systems.
  • How can I configure my system to crash when NMI switch is pushed?: Provides specific guidance on configuring the kernel to panic in response to an NMI.
  • Crash Dump Admin Guide: A detailed guide on configuring and using kdump for collecting kernel crash dumps, which is necessary to capture NMI-induced panics.

SUSE Linux Enterprise Server (SLES)

  • Configure kernel core dump capture: SUSE’s knowledge base article walks through the process of setting up and verifying kdump on SLES, ensuring that NMI-triggered kernel core dumps are successfully recorded.

CoreOS

  • Collecting crash logs: The CoreOS documentation includes information on how to access and collect various types of crash logs, which would include memory dumps if kdump or a similar mechanism is configured to respond to an NMI.

These resources provide the necessary details to prepare your Azure Linux VMs for comprehensive NMI-driven diagnostics, ensuring that you can collect critical data even from the most stubborn system freezes.

Enhancing Diagnostics with Azure Boot Diagnostics

Beyond SysRq and NMI, the Azure Serial Console integrates with Azure Boot Diagnostics to provide a more comprehensive view of your VM’s boot process. Boot diagnostics captures serial console output and screenshots of your VM, which can be invaluable when troubleshooting boot failures or hangs that occur even before the kernel is fully operational or able to respond to SysRq/NMI commands. This feature provides a historical log of the VM’s console, allowing administrators to review messages from the bootloader (like GRUB), kernel initialization, and early system services. This “always-on” logging capability means that even if a VM fails to boot entirely, you’ll still have critical clues to diagnose the problem. The combination of Boot Diagnostics’ persistent logging and the interactive, in-band capabilities of SysRq and NMI through the Serial Console creates a powerful diagnostic suite for any Linux VM administrator.

Next Steps

With a deeper understanding of SysRq and NMI, along with the powerful capabilities of the Azure Serial Console, you are now equipped to handle complex troubleshooting scenarios for your Linux virtual machines in Azure.

We encourage you to experiment with these features in a non-production environment to become proficient in their use. Your feedback and experiences are invaluable. Have you used SysRq or NMI for a particularly challenging VM recovery? Share your insights and questions in the comments below!

Post a Comment