Home linux suse troubleshooting virtualization

SUSE Pacemaker Cluster Won't Start? Here's How to Troubleshoot VMs

Updated: • 0 • min read
Table of Contents

Hey folks! Ever wrestled with a stubborn SUSE Pacemaker cluster that just refuses to start? You’re not alone. Let’s dive into some common startup hiccups and how to kick them to the curb.

SUSE Pacemaker Cluster
image just illustration

SysRq-Triggered Reboot Woes

Sometimes, after a SysRq reboot, Pacemaker plays dead. This usually boils down to a timing clash between the STONITH Block Device (SBD) msgwait and the speedy restart of Azure VMs. The culprit? This little snippet in /etc/sysconfig/sbd:

SBD_DELAY_START=no

The Fix:

  1. Maintenance Mode: First things first, put your cluster in maintenance mode:
sudo crm configure property maintenance-mode=true

  1. Tweak SBD: Edit /etc/sysconfig/sbd and flip SBD_DELAY_START to yes.

  2. Out of Maintenance: Time to bring the cluster back online:

sudo crm configure property maintenance-mode=false
  1. Restart Everything: A good ol’ restart usually does the trick:
sudo systemctl restart sbd
sudo systemctl restart Pacemaker

Pacemaker’s Post-Fencing Freeze (Exit Code 100)

Ever fence a node, only to have Pacemaker throw a tantrum and exit with code 100? Frustrating, right? This happens when a node tries to rejoin the party too soon after being fenced – before the msgwait timeout expires.

systemctl status Pacemaker.service

● Pacemaker.service - Pacemaker High Availability Cluster Manager
   Loaded: loaded (/usr/lib/systemd/system/Pacemaker.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Wed 2020-05-13 23:38:21 UTC; 25s ago
     Docs: man:Pacemakerd
           https://clusterlabs.org/Pacemaker/doc/en-US/Pacemaker/1.1/html-single/Pacemaker_Explained/index.html
 Main PID: 1570 (code=exited, status=100)

Cause and Resolution:

This boils down to the same msgwait conflict. The node’s eagerness to rejoin clashes with the SBD timeout. Let’s look at two ways to tackle this.

Resolution 1: The Delay Game

  1. Maintenance Mode: Activate maintenance mode:
sudo crm configure property maintenance-mode=true

  1. SBD Delay: Set SBD_DELAY_START to yes in /etc/sysconfig/sbd.

  2. Service File Backup: Make a copy of the sbd.service file:

cp /usr/lib/systemd/system/sbd.service /etc/systemd/system/sbd.service
  1. Timeout Tweak: Add these lines to /etc/systemd/system/sbd.service:
[Unit]
Before=corosync.service
[Service]
TimeoutSec=144
  1. Disable Maintenance Mode: Deactivate maintenance mode:
sudo crm configure property maintenance-mode=false
  1. Restart Services: Restart both SBD and Pacemaker (or reboot both nodes for good measure).
sudo systemctl restart sbd
sudo systemctl restart Pacemaker

Resolution 2: Fine-Tuning msgwait

Alternatively, you can fine-tune the SBD msgwait timeout itself. Make it shorter than the time it takes for SBD fencing to complete and sbd.service to restore after a restart. As a rule of thumb, set the watchdog parameter to half the new msgwait value. This is system-specific, so some experimentation might be needed.

Key Takeaways and Troubleshooting Tips

Here are some additional things to keep in mind:

  • Understand SBD: The STONITH Block Device (SBD) is crucial for preventing split-brain scenarios in your cluster. A deep dive into its functionality can greatly improve your troubleshooting skills.

  • Logs Are Your Friends: Don’t underestimate the power of log files! They often hold crucial clues about what’s causing Pacemaker’s headaches. Check both Pacemaker and SBD logs for error messages.

  • Community Support: The SUSE and Pacemaker communities are incredibly helpful. Don’t hesitate to reach out for assistance if you’re still stuck.

  • Testing is Key: After making any changes, thoroughly test your cluster to ensure stability and proper functionality.

So there you have it – a guide to conquering those pesky SUSE Pacemaker startup gremlins. What are your experiences with troubleshooting Pacemaker? Share your stories, tips, and questions in the comments below! We’re all learning together, and your insights could save someone else a lot of frustration. Come back again if you want to get other information.

Post a Comment