Troubleshooting GRUB Rescue: Fixing Boot Issues in Your Linux Virtual Machine

Table of Contents

Applies to: ✔️ Linux VMs

This article provides a comprehensive guide to understanding and resolving GRUB rescue issues that may arise in Linux Virtual Machines (VMs). GRUB rescue mode is an emergency console that the GRUB bootloader enters when it cannot locate the necessary files or instructions to proceed with the normal boot process. This situation can prevent your Linux VM from starting up correctly, leaving it inaccessible. Understanding the common causes and troubleshooting steps for GRUB rescue is crucial for maintaining the availability and stability of your Linux VMs.

Identify GRUB Rescue Issue¶

The first step in addressing a GRUB rescue situation is to confirm that your VM is indeed in this state. The GRUB rescue console is not directly visible in the Azure serial console logs. Instead, you will need to utilize Boot diagnostics within the Azure portal to capture a screenshot of the VM’s boot process. This screenshot will visually display the GRUB rescue prompt, allowing you to definitively identify the issue.

Examine the boot diagnostics screenshot for error messages. A typical GRUB rescue issue will present an error message similar to the following:

error: file '/boot/grub2/i386-pc/normal.mod' not found.
Entering rescue mode...
grub rescue>

This error message clearly indicates that the VM has entered GRUB rescue mode and requires intervention to restore normal boot functionality.

Troubleshoot GRUB Rescue Issue Offline¶

Resolving GRUB rescue issues typically requires an offline troubleshooting approach. This involves creating a separate rescue or repair VM to access and modify the affected VM’s operating system disk.

1. Create a Rescue/Repair VM: Utilize Azure VM repair commands to create a dedicated repair VM. These commands automate the process of creating a new VM and attaching a copy of the problematic VM’s OS disk to it. This setup allows you to work on a copy of the OS disk without risking further damage to the original. Once the repair VM is created, mount the file systems from the copied OS disk within the repair VM using chroot. This effectively places you inside the environment of the affected VM’s operating system.

Alternatively, a rescue VM can be created manually via the Azure portal. This manual approach offers more control over the rescue VM’s configuration, but requires more steps.

2. Identify and Address Specific GRUB Rescue Errors: Once you have access to the affected VM’s file system through the repair VM, you need to pinpoint the specific GRUB rescue error message displayed in the boot diagnostics. Common GRUB rescue errors include:

   • Error: unknown filesystem: This error often indicates corruption within the /boot file system or that GRUB is pointing to an incorrect disk or partition.
   • Error 15: File not found: This error suggests that GRUB is unable to locate a specific file it needs to boot, often within the /boot directory.
   • Error: file ‘/boot/grub2/i386-pc/normal.mod’ not found: This specific file not found error points to missing or corrupted GRUB modules necessary for normal boot operation.
   • Error: no such partition: This error typically arises when the /boot partition is missing, deleted, or incorrectly configured in the partition table.
   • Error: symbol ‘grub_efi_get_secure_boot’ not found: This error is specific to UEFI-based VMs with Secure Boot enabled and may occur with certain older kernel versions.
   • Other GRUB rescue errors: Various other error messages can lead to GRUB rescue, often related to configuration issues or missing boot files.

   Navigate to the corresponding section in this article for detailed troubleshooting steps specific to each of these error types.

3. Post-Resolution Steps: After implementing the necessary fixes for the GRUB rescue issue:

   1. Unmount File Systems: Within the rescue/repair VM, carefully unmount all file systems that were mounted from the copied OS disk. This ensures data integrity and prevents potential corruption during the disk swapping process.
   2. Swap OS Disks: Execute the az vm repair restore command. This crucial step swaps the repaired OS disk copy with the original OS disk of the affected VM. This action effectively applies the fixes you’ve implemented in the repair VM to the original VM’s disk.
   3. Verify VM Startup: Monitor the Azure serial console or attempt to connect to the VM to verify if it starts successfully after the disk swap. Successful boot indicates that the GRUB rescue issue has been resolved.

4. Data Restoration from Backup: In severe cases where the entire /boot partition or critical boot-related data is missing or irrecoverable, restoring the VM from a recent backup is the recommended course of action. Azure Backup provides robust VM backup and restore capabilities.

The following sections provide detailed guidance on diagnosing and resolving specific GRUB rescue errors, along with step-by-step instructions to restore your Linux VM to a bootable state.

Reinstall GRUB and Regenerate the GRUB Configuration File using Azure Linux Auto Repair¶

Azure Linux Auto Repair (ALAR) offers a streamlined, automated approach to resolving common GRUB rescue scenarios. ALAR scripts are integrated into the VM repair extension, simplifying the repair process.

ALAR automates the reinstallation of GRUB and the regeneration of the GRUB configuration file. To utilize ALAR for GRUB repair, use the repair-button command within the VM repair extension. Specify the --button-command parameter with either grubfix for Generation 1 VMs (BIOS-based) or efifix for Generation 2 VMs (UEFI-based).

Automated GRUB Repair Commands:

• Linux VMs without UEFI (BIOS based - Gen1):
  

  az extension add -n vm-repair
  az extension update -n vm-repair
  az vm repair repair-button --button-command 'grubfix' --verbose $RGNAME --name $VMNAME
  

• Linux VMs with UEFI (Gen2):
  

  az extension add -n vm-repair
  az extension update -n vm-repair
  az vm repair repair-button --button-command 'efifix' --verbose $RGNAME --name $VMNAME
  

The repair-button script, in conjunction with ALAR scripts, orchestrates the following actions:

• Creation of a temporary resource group.
• Deployment of a repair VM within the temporary resource group.
• Creation of a copy of the affected VM’s OS disk and attachment to the repair VM.
• Automated reinstallation of GRUB and regeneration of the GRUB configuration file within the repair VM environment.
• Swapping of the repaired OS disk copy with the original OS disk of the broken VM.
• Automatic deletion of the temporary resource group and the repair VM.

This automated process significantly reduces the manual effort and complexity involved in GRUB repair, especially for common scenarios.

Reinstall GRUB and Regenerate the GRUB Configuration File Manually¶

For scenarios where automated repair is not feasible or preferred, manual GRUB reinstallation and configuration regeneration can be performed.

1. Prepare Rescue/Repair VM and chroot Environment: If you haven’t already created a rescue/repair VM and established a chroot environment as described in the “Troubleshoot GRUB rescue issue offline” section, complete those steps first. Ensure that all necessary file systems, including / and /boot, are mounted within the rescue/repair VM and that you are operating within the chroot environment of the affected VM’s OS.

2. Reinstall GRUB and Regenerate Configuration: Execute the appropriate commands based on your Linux distribution and VM generation (Gen1 or Gen2) to reinstall GRUB and regenerate its configuration file.

   • RHEL/CentOS/Oracle 7.x/8.x/9.x Linux VMs without UEFI (BIOS based - Gen1):
     

     grub2-install /dev/sdX
     grub2-mkconfig -o /boot/grub2/grub.cfg
     sed -i 's/hd2/hd0/g' /boot/grub2/grub.cfg
     

     
     (Replace /dev/sdX with the correct device identifier for the OS disk copy attached to the repair VM, e.g., /dev/sdc)

   • RHEL/CentOS/Oracle 7.x/8.x/9.x Linux VMs with UEFI (Gen2):
     

     yum reinstall grub2-efi-x64 shim-x64
     grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg
     sed -i 's/hd2/hd0/g' /boot/efi/EFI/redhat/grub.cfg
     

     
     (For CentOS, replace redhat with centos in the grub.cfg path.)
     (Replace /dev/sdX with the correct device identifier for the OS disk copy attached to the repair VM, e.g., /dev/sdc)

   • SLES 12/15 Gen1 and Gen2:
     

     grub2-install /dev/sdX
     grub2-mkconfig -o /boot/grub2/grub.cfg
     sed -i 's/hd2/hd0/g' /boot/grub2/grub.cfg
     

     
     (Replace /dev/sdX with the correct device identifier for the OS disk copy attached to the repair VM, e.g., /dev/sdc)

   • Ubuntu Gen1 and Gen2:
     

     grub-install /dev/sdX
     update-grub
     

     
     (Replace /dev/sdX with the correct device identifier for the OS disk copy attached to the repair VM, e.g., /dev/sdc)

3. Swap OS Disks and Verify: Proceed to step 3 in the “Troubleshoot GRUB rescue issue offline” section to swap the OS disk and verify if the VM now boots correctly.

Error: unknown filesystem¶

The “Error: unknown filesystem” message in GRUB rescue indicates that GRUB cannot recognize the file system type of the partition it is trying to access. This can stem from several underlying issues:

• /boot File System Corruption: The file system on the /boot partition itself might be corrupted. This corruption can be caused by disk errors, improper shutdowns, or file system inconsistencies. To address this, you need to repair the /boot file system.
• Invalid GRUB Boot Loader Configuration: GRUB might be configured to point to an incorrect disk or partition that either doesn’t exist or doesn’t contain a recognizable file system. Reinstalling GRUB and regenerating its configuration can correct these pointers.
• OS Disk Partition Table Issues: Human errors during partition table modifications can lead to inconsistencies or damage. If the /boot partition is missing or incorrectly defined in the partition table, GRUB will fail to recognize it.

Fix /boot File System Corruption¶

If the “unknown filesystem” error is suspected to be caused by /boot file system corruption, follow these steps to attempt repair:

1. Prepare Rescue/Repair VM: Ensure you have created a rescue/repair VM as described in the “Troubleshoot GRUB rescue issue offline” section.

2. Repair File System Corruption: Refer to dedicated guides on troubleshooting file system corruption errors in Azure Linux. These guides provide detailed instructions on using file system repair utilities (like fsck) to identify and attempt to fix errors on the /boot partition.

3. Swap OS Disks and Verify: After attempting file system repair, proceed to step 3 in the “Troubleshoot GRUB rescue issue offline” section to swap the OS disk and check if the VM boots.

Error 15: File not found¶

“Error 15: File not found” in GRUB rescue signifies that GRUB is unable to locate a specific file required for the boot process. This often points to issues within the /boot file system.

To resolve this error:

1. Prepare Rescue/Repair VM and chroot Environment: Create a rescue/repair VM and enter the chroot environment as outlined in “Troubleshoot GRUB rescue issue offline”. Mount both / and /boot file systems.

2. Inspect /boot Contents: Examine the contents of the /boot file system within the chroot environment to determine which files are missing. Use commands like ls -l /boot to list the files and directories.

3. Reinstall GRUB and Regenerate Configuration (if GRUB configuration missing): If the GRUB configuration file (e.g., grub.cfg) is missing from /boot/grub2 or /boot/efi/EFI/redhat (depending on your system), follow the “Reinstall GRUB and regenerate the GRUB configuration file manually” section to recreate it.

4. Verify File Permissions: Check the file permissions within the /boot file system. Incorrect permissions can prevent GRUB from accessing necessary files. Compare the permissions to a working VM with the same Linux distribution to identify and correct any discrepancies using chmod and chown.

5. Data Restoration from Backup (if /boot missing or unrecoverable): If the entire /boot partition or essential boot files are missing and cannot be recovered, restoring from a backup is the recommended approach.

6. Swap OS Disks and Verify: After addressing the file missing issue, proceed to step 3 in “Troubleshoot GRUB rescue issue offline” to swap the OS disk and test VM startup.

Error: file ‘/boot/grub2/i386-pc/normal.mod’ not found¶

The specific error “file ‘/boot/grub2/i386-pc/normal.mod’ not found” indicates that GRUB is missing the normal.mod module, which is essential for loading the normal GRUB menu and boot process.

To fix this:

1. Prepare Rescue/Repair VM and chroot Environment: Create a rescue/repair VM and enter the chroot environment, mounting / and /boot as described previously.

2. /boot File System Corruption Check: If you encounter errors mounting /boot, suspect file system corruption and follow the steps in “Fix /boot file system corruption”.

3. Verify and Copy GRUB Modules: Within the chroot environment, verify the contents of /boot/grub2/i386-pc. If the directory is empty or missing essential modules, copy the contents from /usr/lib/grub/i386-pc to /boot/grub2. Use these commands:

ls -l /boot/grub2/i386-pc
cp -rp /usr/lib/grub/i386-pc /boot/grub2

4. Recreate /boot Partition Contents (if /boot is empty - Gen1 RHEL/CentOS/Oracle 7.x/8.x): If the entire /boot partition is empty (this step applies to Gen1 VMs with RHEL-based distributions):

5. Reinstall GRUB: Reinstall GRUB to the correct disk (replace /dev/sd[X] with the OS disk device):
   

   grub2-install /dev/sd[X]
   

6. Verify DNS in /etc/resolv.conf: Ensure /etc/resolv.conf contains a valid DNS entry to resolve repository names for package installation.
   

   cat /etc/resolv.conf
   

7. Reinstall Kernel: Reinstall the Linux kernel:
   

   yum reinstall $(rpm -qa | grep -i kernel)
   

8. Regenerate grub.cfg: Create the GRUB configuration file:
   

   grub2-mkconfig -o /boot/grub2/grub.cfg
   sed -i 's/hd2/hd0/g' /boot/grub2/grub.cfg
   

9. Swap OS Disks and Verify: Proceed with step 3 in “Troubleshoot GRUB rescue issue offline” to swap the OS disk and test the VM.

Error: no such partition¶

The “Error: no such partition” message on RHEL-based VMs (Red Hat, Oracle Linux, CentOS) typically indicates issues with the /boot partition itself:

• /boot Partition Deletion: The /boot partition may have been accidentally deleted.
• /boot Partition Re-creation Error: The /boot partition might have been re-created with incorrect start and end sectors, making it unusable.

Solution: Re-create /boot partition¶

If the /boot partition is missing, you need to re-create it. The process differs slightly depending on whether the disk uses a DOS or GPT partition table.

1. Prepare Rescue/Repair VM: Create a rescue/repair VM as described in “Troubleshoot GRUB rescue issue offline”.

2. Identify Partition Table Type: Determine if the disk uses a DOS or GPT partition table using fdisk:
   

   sudo fdisk -l /dev/sdX
   

   
   Examine the output for “Disk label type: dos” or “Disk label type: gpt”.

3. Re-create /boot Partition (DOS or GPT):

• DOS Partition Table: Follow the steps in “Re-create /boot partition in dos systems”.
• GPT Partition Table: Follow the steps in “Re-create /boot partition in GPT systems”.

4. Reinstall GRUB: After re-creating the partition, reinstall GRUB as described in “Reinstall GRUB and regenerate the GRUB configuration file manually”.

5. Swap OS Disks and Verify: Proceed with step 3 in “Troubleshoot GRUB rescue issue offline” to swap the OS disk and test VM boot.

Re-create /boot partition in dos systems¶

1. Use fdisk to Re-create Partition: In the rescue/repair VM, use fdisk to interactively create the /boot partition:
   

   sudo fdisk /dev/sdX
   

   
   Follow the prompts:
   • n (for new partition)
   • p (for primary partition)
   • Partition number: 1 (or the appropriate number if partitions already exist)
   • First sector: Accept the default
   • Last sector: Accept the default (or specify size like +1G for 1GB)
   • t (to change partition type)
   • Partition number: 1 (or the partition number you created)
   • Hex code: 83 (for Linux partition)
   • a (to make partition bootable)
   • Partition number: 1
   • w (to write changes to disk)

2. Verify /boot File System Detection: Check if the /boot file system is detected using blkid:
   

   sudo blkid /dev/sdX1
   

   
   You should see output similar to /dev/sdc1: UUID="..." TYPE="ext4".

3. Recreate File System and Restore from Backup (if necessary): If blkid doesn’t show the file system, you’ll need to format the partition (using the same UUID and file system type from /etc/fstab) and restore /boot contents from a backup.

Re-create /boot partition in GPT systems¶

1. Use gdisk to Re-create Partition: Use gdisk to create the /boot partition for GPT disks:
   

   sudo gdisk /dev/sdX
   

   
   Follow the prompts:
   • n (for new partition)
   • Partition number: 1 (or appropriate number)
   • First sector: Accept default (or specify start sector)
   • Last sector: Accept default (or specify size like +500M for 500MB)
   • Hex code: 8300 (for Linux filesystem)
   • w (to write changes)
   • Y (to confirm write)

2. Verify /boot File System Detection: Check if the /boot file system is detected using blkid:
   

   sudo blkid /dev/sdX1
   

   
   Look for output similar to /dev/sdc1: UUID="..." BLOCK_SIZE="4096" TYPE="xfs" PARTLABEL="Linux filesystem" PARTUUID="...".

3. Recreate File System and Restore from Backup (if necessary): If blkid doesn’t detect the file system, you’ll need to format the partition and restore /boot data from a backup.

Error: symbol ‘grub_efi_get_secure_boot’ not found¶

The “Error: symbol ‘grub_efi_get_secure_boot’ not found” error is specific to UEFI-based (Gen2) VMs with Secure Boot enabled, particularly when using older kernel versions like 4.12.14 (used in SLES 12 SP5). This kernel version lacks full Secure Boot support.

Solution¶

To resolve this boot error, disable Secure Boot in the VM’s bootloader configuration:

1. Prepare Rescue/Repair VM and chroot Environment: Create a rescue/repair VM and enter chroot, mounting / and /boot.

2. Disable Secure Boot in YaST: Use the YaST bootloader configuration tool within the chroot environment:
   

   yast2 bootloader
   

3. Clear Secure Boot Option: In the YaST bootloader interface, deselect or uncheck the “Enable Secure Boot Support” option. Save the changes by selecting F10.

4. Swap OS Disks and Verify: Proceed to step 3 in “Troubleshoot GRUB rescue issue offline” to swap the OS disk and test VM startup.

Other GRUB rescue errors¶

Various other error messages can lead to GRUB rescue. These errors often stem from:

• Missing GRUB Configuration File: The main GRUB configuration file (grub.cfg) might be absent.
• Incorrect GRUB Configuration: A corrupted or misconfigured grub.cfg can prevent proper booting.
• /boot Partition or Contents Missing: As discussed previously, issues with the /boot partition are a common cause of GRUB rescue.

To troubleshoot these general GRUB rescue errors:

1. Prepare Rescue/Repair VM and chroot Environment: Create a rescue/repair VM and enter chroot, mounting / and /boot.

2. Verify /etc/default/grub Configuration: Ensure the /etc/default/grub file is correctly configured. Azure-endorsed Linux images typically have appropriate default configurations. Consult distribution-specific documentation for recommended settings.

3. Reinstall GRUB and Regenerate Configuration: Follow the steps in “Reinstall GRUB and regenerate the GRUB configuration file manually” to ensure GRUB is correctly installed and its configuration is up-to-date.

• Note: If the error mentions /boot/grub/menu.lst, it indicates an older OS version (RHEL 6.x, CentOS 6.x, Ubuntu 14.04) using GRUB version 1. GRUB version 1 troubleshooting is not covered in this article.

4. /boot Partition Check: If the entire /boot partition is missing, refer to “Error: no such partition” for steps to re-create it.

5. Swap OS Disks and Verify: Proceed to step 3 in “Troubleshoot GRUB rescue issue offline” to swap the OS disk and test VM startup.

Next steps¶

If you encounter boot errors that are not specifically GRUB rescue issues, consult the general “Troubleshoot Azure Linux Virtual Machines boot errors” guide for broader troubleshooting options.

Third-party information disclaimer

The third-party products discussed in this article are manufactured by companies independent of Microsoft. Microsoft provides no warranty, implied or otherwise, regarding the performance or reliability of these products.

Third-party contact disclaimer

Microsoft provides third-party contact information to assist you in finding additional information on this topic. This contact information may change without notice. Microsoft does not guarantee the accuracy of third-party contact information.

---

Do you have any questions or further troubleshooting steps you’d like to discuss regarding GRUB rescue issues in Linux VMs? Share your experiences or additional tips in the comments below!