Fix Linux VM Boot Failures: Troubleshooting Filesystem Errors

Table of Contents

Linux virtual machines (VMs) are crucial for many modern IT infrastructures, but they can occasionally encounter boot issues that hinder connectivity and operation. One of the most common and challenging causes of these failures stems from filesystem errors. This comprehensive guide provides detailed instructions to diagnose and resolve such boot problems, ensuring your Linux VMs return to a healthy operational state. It applies specifically to Linux VMs running in cloud environments, focusing on the troubleshooting steps necessary to repair filesystem corruption.

Understanding Linux VM Boot Failures¶

Boot failures in Linux VMs due to filesystem errors typically manifest when the operating system cannot properly mount essential partitions or logical volumes during startup. This inability to read or write data on the disk prevents the VM from fully initializing, leading to a stalled boot process. The underlying causes of disk corruption are diverse, ranging from sudden power loss or unclean shutdowns to more complex issues like Linux kernel bugs, faulty drivers, or even problems within the underlying physical or virtual hardware infrastructure. Identifying the root cause is critical for effective resolution.

Common Symptoms of Filesystem Corruption¶

When a Linux VM experiences filesystem corruption, you often cannot connect via Secure Shell Protocol (SSH), and the VM Agent status in the Azure portal might display as Not Ready. Accessing Boot diagnostics or the Serial Console reveals critical log entries that pinpoint the issue. These logs provide invaluable clues about the specific filesystem and device experiencing problems.

Here are some typical log examples you might encounter:

Example 1: EXT4-fs Read-Only and Journal Errors¶

EXT4-fs (sda1): INFO: recovery required on readonly filesystem
EXT4-fs (sda1): write access will be enabled during recovery
EXT4-fs warning (device sda1): ext4_clear_journal_err:4531: Filesystem error recorded from previous mount: IO failure
EXT4-fs warning (device sda1): ext4_clear_journal_err:4532: Marking fs in need of filesystem check.

This output indicates that the sda1 partition, formatted with EXT4, has encountered an input/output (IO) failure. The filesystem is marked as read-only, requiring a recovery process to enable write access and a filesystem check for consistency.

Example 2: EXT4-fs Logical Volume Manager (LVM) Device Issues¶

[   14.382472] EXT4-fs error (device dm-0): ext4_iget:4398: inode #8: comm mount: bad extra_isize 4060 (inode size 256)
[   14.389648] EXT4-fs (dm-0): no journal found
<snipped>
[FAILED] Failed to mount /opt/data.

Here, an EXT4 filesystem residing on a Logical Volume Manager (LVM) device (dm-0) reports an inode error and a missing journal. This often leads to a failure in mounting the associated /opt/data directory, preventing the system from accessing critical application data.

Example 3: XFS Metadata CRC Errors¶

[    8.543984] XFS (sdc1): Metadata CRC error detected at xfs_agi_read_verify+0xd0/0xf0 [xfs], xfs_agi block 0x10
[    8.553867] XFS (sdc1): Unmount and run xfs_repair
[    8.558993] XFS (sdc1): First 128 bytes of corrupted metadata buffer:
[    8.564893] 00000000: 58 41 47 49 00 00 00 01 00 00 00 00 00 1f ff c0  XAGI............
... (snipped metadata buffer lines) ...
[    8.629731] XFS (sdc1): metadata I/O error in "xfs_trans_read_buf_map" at daddr 0x10 len 8 error 74
[    8.637799] XFS (sdc1): xfs_imap_lookup: xfs_ialloc_read_agi() returned error -117, agno 0
[FAILED] Failed to mount /data.
See 'systemctl status data.mount' for details.
[DEPEND] Dependency failed for Local filesystems.

This detailed XFS error on sdc1 indicates metadata corruption, specifically a CRC error. The log explicitly advises to unmount the filesystem and run xfs_repair, confirming a severe integrity issue that prevents mounting /data.

Example 4: Booting into Emergency Mode¶

You are in emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or "exit"
to boot into default mode.
Give root password for maintenance
(or press Control-D to continue):

When a system cannot resolve critical boot-time issues, it may drop into emergency mode. This prompt indicates that manual intervention is required to diagnose and fix the underlying problem, often a filesystem error preventing normal startup.

Comprehensive Resolution Steps for Disk Corruption¶

To resolve Linux VM boot issues caused by filesystem errors, the primary goal is to recover the VM by repairing the corrupted disk. This involves a systematic approach to identify, prepare, and execute the repair process. Before initiating any repair, remember two crucial points:
* Backup critical data: Data loss may occur on the recovered disk, so backing up essential information is paramount.
* Take a snapshot: Always create a snapshot of the disk before making any changes, even if it’s already in an error state. This preserves the current state and provides a rollback option.

The overall resolution process is divided into the following key steps:

1. Identify which disk is corrupted.
2. Identify the filesystem type used on the corrupted disk.
3. Select the appropriate recovery mode (online or offline).
4. Prepare the recovery environment based on the chosen mode.
5. Use command-line tools to repair the problematic filesystem.

Step 1: Identifying the Corrupted Disk¶

The first crucial step is to pinpoint exactly which disk or partition is affected. This is done by reviewing the serial log of your VM, accessible via the Serial Console or Boot diagnostics in the Azure portal. Look for error messages during the boot sequence that explicitly name the failing disk or mount point.

Consider these three examples from serial logs:

• Example A: XFS (sdc1): Metadata CRC error detected... [FAILED] Failed to mount /opt/parent.
  Here, the corrupted device is clearly indicated as sdc1.
• Example B: EXT4-fs (sda1): INFO: recovery required on readonly filesystem... [FAILED] Failed to mount /boot.
  In this case, the partition experiencing a filesystem error is sda1.
• Example C: EXT4-fs (dm-2): VFS: Can't find ext4 filesystem... [FAILED] Failed to mount /home.
  This example points to dm-2, which is a Linux Device Mapper device, typically indicating an LVM volume.

If the device name follows the sdXN format (e.g., sda1, sdc1), it signifies a raw disk partition, directly addressable via /dev/sdXN. However, if you see names like /dev/mapper/vgname/lvname, /dev/vgname/lvname, or dm-N, an LVM device is in use. For LVM, it’s essential to identify all associated physical volumes (PVs).

It’s important to understand the typical disk layout in Azure Linux VMs:
* OS Disk: For marketplace images, the root filesystem (/), /boot, and /boot/efi are located on the OS disk.
* LVM-based Images: These may include additional system mounts like /home, /tmp, /usr, /var, /var/log, and /opt on the OS disk.
* Data Disks: Extra filesystems for applications, such as /data or /sap, are usually on data disks. These should be configured to allow the system to boot even if they encounter errors.

Image: Visual representation of identifying corrupted disks from boot logs.

Important Note on LVM: While LVM allows for flexible storage management, it is generally not supported for a single LVM volume group (VG) to span both the OS disk and multiple data disks, as this carries a high risk of data loss. Multiple data disks within an LVM VG, however, are permissible.

Step 2: Determining the Filesystem Type¶

Once you’ve identified the corrupted disk, the next step is to determine its filesystem type (e.g., EXT4, XFS). During the initial identification phase, the serial log is your primary source. The Linux kernel module reporting errors will often explicitly mention EXT4-fs or XFS. Pay close attention to these indicators.

After gaining access to an interactive shell (either in emergency mode, single-user mode, or a rescue VM), you can use the lsblk command with the -f flag to get a definitive list of devices, their mount points, and their filesystem types as read directly from the disk.

[root@localhost ~]# lsblk -f
NAME              FSTYPE      LABEL UUID                                   MOUNTPOINT
sda
|-sda1            vfat              93DA-8C20                              /boot/efi
|-sda2            xfs               d5da486e-fdfe-4ad8-bc01-aa72b91fd47d   /boot
|-sda3
`-sda4            LVM2_member       pdSI2Q-ZEzV-oT6P-R2JG-ZW3h-cmnf-iRN6pU
  |-rootvg-tmplv  xfs               9098eb05-0176-4997-8132-9152a7bef207   /tmp
  |-rootvg-usrlv  xfs               2f9ff36c-742d-4914-b463-d4152801b95d   /usr
  |-rootvg-optlv  xfs               aeacea8e-3663-4569-af25-c52357f8a0a3   /opt
  |-rootvg-homelv xfs               a79e43dc-7adc-41b4-b6e1-4e6b033b15c0
  |-rootvg-varlv  xfs               c7cb68e9-7865-4187-b3bd-e9a869779d86   /var
  `-rootvg-rootlv xfs               d8dc4d62-ada5-4952-a0d9-1bce6cb6f809   /
sdb
`-sdb1            ext4              1dac7c4c-bf8e-4964-8a59-7359eef53d0a   /mnt
sdc               LVM2_member       CRWEZQ-iLhH-ev0b-BAaA-dfLD-nbPT-GgtG0r
`-vgapp-lvapp     xfs               733e25ee-565f-4bfa-a2a1-2451efd25cd1
sdd
`-sdd1            ext4              704d9fb1-2207-4bb9-998c-029f776dc6d2   /opt/data

From this output, we can observe several important details:
* The ASCII art clearly shows LVM volumes, indicated by LVM2_member FSTYPE for sda4 and sdc, which contain logical volumes like rootvg-rootlv and vgapp-lvapp.
* A volume such as rootvg-homelv is listed but has an empty MOUNTPOINT, meaning it’s not currently mounted.
* The FSTYPE for rootvg-homelv is xfs. This is crucial, as the filesystem type read directly from the disk (lsblk) should be prioritized over entries in /etc/fstab if there’s an inconsistency.

Image: A flowchart illustrating the process of determining filesystem type.

Step 3: Choosing the Appropriate Recovery Mode¶

Depending on the specific circumstances of the boot failure and your access capabilities, you can choose between two main recovery modes: online recovery (via emergency mode or single-user mode) or offline recovery (using a rescue VM).

Online Recovery Requirements¶

Online recovery is feasible when:
* You have active Serial Console access to the VM.
* If emergency mode is presented, the root account must be unlocked, and its password known.
* If single-user mode is used, the root password is not required. This mode is particularly useful when corruption affects filesystems other than critical system partitions like root (/) or /usr.

Offline Recovery Requirements¶

Offline recovery is necessary when the requirements for online recovery cannot be met, or if the corruption is so severe that the VM cannot even enter emergency or single-user mode. To perform an offline recovery:
* You must have the ability to create VMs and manage disks within Azure.
* Alternatively, you can use an existing, functioning Linux VM with Azure-level access to attach and work with the corrupted disks.
* Crucially, when using a rescue VM, do not mount the corrupted volumes. Filesystem repair utilities require the filesystem to be unmounted to operate safely and effectively.

Step 4: Preparing the Recovery Environment¶

The steps to prepare your environment will vary based on whether you’ve chosen online or offline recovery.

Online Recovery Environment Preparation¶

If the VM lands in emergency mode and presents a sign-in prompt like:

Welcome to emergency mode! After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to
try again to Give root password for maintenance
(or press Control-D to continue):

Enter the root password to proceed. If the root password is unknown or the root account is locked (as indicated by “Cannot open access to console, the root account is locked.”), then you should opt for single-user mode. This mode typically grants root access without requiring a password, facilitating immediate troubleshooting. If neither option allows access, online recovery is not viable.

Offline Recovery Environment Preparation¶

For single-disk VMs, or when the root filesystem (/) or /usr partition is compromised, an offline recovery using a rescue VM is often the most reliable method. Azure provides tools for both automated and manual creation of a rescue VM:
* Automated Rescue VM: Utilize Azure Virtual Machine Repair commands. This streamlines the process of creating a new VM with the problematic disk attached.
* Manual Rescue VM: Follow the steps for creating a recovery VM. This involves creating a new VM and manually attaching the data disk(s) from the original, corrupted VM.

Critical reminder for offline recovery: Regardless of whether you use automated or manual methods, it is imperative not to mount the corrupted volumes on the rescue VM. Filesystem repair tools perform direct low-level operations that can be destructive if the filesystem is actively mounted.

Image: A diagram showing the workflow for preparing both online and offline recovery environments.

Step 5: Performing the Filesystem Repair¶

Before you begin the actual repair, ensure all preceding steps have been completed:
* You have accurately identified the problem disk, partition, or LVM volume structure.
* The exact filesystem type (e.g., EXT4, XFS) is confirmed.
* (Optional but recommended) A copy of the problem disk(s) is attached to a rescue VM.
* You have secured interactive shell access to the recovery environment.

Important Note: Filesystem repair commands are powerful. While they aim to fix corruption, data loss may still occur. It is crucial to have backups and a snapshot of the disk before proceeding. The goal is to get the filesystem into a “clean” state so the VM can boot, but this sometimes comes at the cost of some unrecoverable data.

The commands for filesystem repair remain consistent, irrespective of whether you are in an emergency shell, single-user mode, or a rescue VM. If the emergency shell proves too limited (e.g., commands are unavailable or error with “unknown filesystem types”), revert to preparing an offline recovery environment.

In the following examples, **/dev/sdc1** will represent a corrupted raw filesystem, and **/dev/rootvg/homelv** will denote a corrupted LVM logical volume. Always substitute these placeholders with the actual device paths for your specific corrupted filesystem.

Repairing EXT4 Filesystems¶

For EXT4 filesystems, the fsck utility is your primary tool. Use the command fsck [-y] FILESYSTEM. Specify the filesystem as a disk partition (e.g., /dev/sdc1) or an LVM logical volume path (e.g., /dev/rootvg/homelv).

Here’s an example of fsck in action:

[root@vm1dev ~]# fsck /dev/sdc1
fsck from util-linux 2.23.2
e2fsck 1.42.9 (28-Dec-2013)
ext2fs_check_desc: Corrupt group descriptor: bad block for block bitmap
fsck.ext4: Group descriptors look bad... trying backup blocks...
/dev/sdc1 was not cleanly unmounted, check forced.
Resize inode not valid.  Recreate<y>? yes
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong for group #0 (23508, counted=23509).
Fix<y>? yes
Free blocks count wrong (8211645, counted=8211646).
Fix<y>? yes

/dev/sdc1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sdc1: 11/2097152 files (0.0% non-contiguous), 176706/8388352 blocks
[root@vm1dev ~]#

The output shows multiple prompts for confirmation to modify the filesystem (<y>? yes). If there are numerous prompts, you can restart fsck with the -y flag (fsck -y /dev/sdc1) to automatically answer “yes” to all questions. If any files are moved to lost+found, you will need to manually identify and relocate them to their correct locations later.

After the initial repair, run the fsck command again. You should repeat this process until fsck exits with a clean status, indicating no further errors are detected.

[root@vm1dev ~]# fsck /dev/sdc1
fsck from util-linux 2.23.2
e2fsck 1.42.9 (28-Dec-2013)
/dev/sdc1: clean, 11/2097152 files, 176706/8388352 blocks
[root@vm1dev ~]#

Image: A terminal screenshot showing the fsck command in action, repairing an EXT4 filesystem.

Repairing XFS Filesystems¶

For XFS filesystems, use the xfs_repair utility. Follow these steps:

1. Check for errors first (dry run):
   Run xfs_repair -n FILESYSTEM to perform a check without making any modifications. This gives you an idea of the extent of the corruption.
   

   xfs_repair -n /dev/rootvg/homelv
   

2. Perform the actual repair:
   If the dry run indicates issues, proceed with the repair mode by omitting the -n flag. This will attempt to fix all encountered errors.
   

   xfs_repair /dev/rootvg/homelv
   

Handling Journaled Changes:
XFS filesystems use a journal to ensure data integrity. If xfs_repair outputs an error like “ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed”, it means uncommitted changes are present in the journal. These changes are typically replayed when the filesystem is mounted.

• If you are in a recovery VM, create a temporary mount point (e.g., /recovery) and attempt to mount the filesystem:
  

  mkdir /recovery
  mount /dev/rootvg/homelv /recovery
  

• If you are in an emergency or single-user mode environment, attempt to mount the filesystem to its intended location:
  

  mount /home
  

  
  This attempt to mount will trigger the journal replay. If it succeeds, unmount the filesystem and run xfs_repair again without the -L flag.

If the journaled changes cannot be written upon mounting, or if the mount itself fails, you can use the -L flag with xfs_repair to explicitly discard the journal. This tells XFS to proceed as if all prior changes were successfully committed, which might resolve the issue but carries a higher risk of data loss. Use this option only as a last resort.

xfs_repair -L /dev/rootvg/homelv

After any xfs_repair operation, always attempt to mount the filesystem to verify its integrity.

Image: A visual explanation of XFS filesystem repair steps, including journal handling.

Preventing Future Boot Failures¶

To minimize the risk of future boot failures caused by non-critical filesystem corruption, consider implementing the nofail option in your /etc/fstab entries for data disks or other non-essential mounts. When nofail is specified, the system will attempt to mount the filesystem, but if the mount fails (e.g., due to corruption or the disk being unavailable), the boot process will continue without dropping into emergency mode. This allows your VM to start fully, enabling you to address the corrupted disk without immediate service disruption.

For instance, an entry for a data disk might look like this:

UUID=your-disk-uuid /data ext4 defaults,nofail 0 2

Most mounts, excluding core system partitions like root (/), /usr, and /var, can safely use the nofail option. This preventative measure significantly improves the resilience of your Linux VMs against localized filesystem issues.

Conclusion¶

Troubleshooting Linux VM boot failures due to filesystem errors requires a methodical approach, from identifying the corrupted component to executing precise repair commands. By systematically following these steps—identifying the disk and filesystem type, selecting the appropriate recovery mode, preparing the environment, and utilizing fsck for EXT4 or xfs_repair for XFS—you can effectively resolve many common boot issues. Remember to prioritize data backup and snapshots to mitigate the risk of data loss during the repair process. Proactive measures, such as using the nofail option for non-critical mounts, can also significantly enhance your VM’s resilience.

Have you encountered similar filesystem corruption issues in your Linux VMs? What strategies have worked best for you? Share your experiences and insights in the comments below!