Home high availability linux pacemaker troubleshooting

Resolving SBD Service Failures in RHEL Pacemaker Clusters: A Troubleshooting Guide

Updated: • 0 • min read
Table of Contents

This guide provides detailed instructions for identifying and resolving common issues that prevent the STONITH Block Device (SBD) service from starting in a Red Hat Enterprise Linux (RHEL) Pacemaker cluster. Understanding these scenarios and their respective solutions is crucial for maintaining the high availability of your cluster resources. Effective SBD operation is the cornerstone of a stable fencing mechanism, preventing data corruption and ensuring smooth failovers.

Understanding STONITH Block Device (SBD) Operation

The SBD mechanism is a critical component in Pacemaker clusters, designed to prevent split-brain scenarios by ensuring that a failed node is completely isolated (fenced) from shared storage before its resources are brought online elsewhere. This is achieved by using a shared block device where cluster nodes write heartbeats and messages. If a node fails to write its heartbeat, other nodes can infer its failure and initiate fencing.

SBD relies on at least one external virtual machine (VM) acting as an Internet Small Computer System Interface (iSCSI) target server, which provides the shared SBD device(s). These iSCSI target servers can be shared across multiple Pacemaker clusters, offering a flexible and scalable solution. The primary benefit of using an SBD device, particularly for hybrid cloud scenarios, is its compatibility with existing on-premises SBD implementations, minimizing operational changes.

For Azure-based Pacemaker clusters, SBD fencing can be configured using two main approaches: establishing SBD with an iSCSI target server or leveraging Azure Shared Disks directly. Both methods provide the necessary shared block storage for SBD messaging, ensuring robust cluster fencing.

Pacemaker SBD Architecture
Figure 1: Conceptual diagram of a Pacemaker cluster utilizing SBD and an iSCSI target server.

Common Symptoms of SBD Service Failure

When the SBD device is unreachable or improperly configured, the SBD daemon will fail to start. This critical failure directly prevents the Pacemaker cluster from initializing or operating correctly, leading to resource unavailability and potential outages. Identifying the root cause swiftly is paramount for cluster stability.

Key indicators of SBD service failure often include error messages during cluster startup or when checking service statuses. You can gather vital information about SBD server details by examining logs within the /var/log/messages file. Additionally, the output of the iscsiadm discovery command and the status of the iSCSI service provide insights into the accessibility and configuration of your SBD server IP addresses.

Analyzing SBD Failure Impact

A non-functional SBD service means the cluster lacks a reliable fencing mechanism. This exposes the cluster to severe risks, including data corruption if multiple nodes attempt to access shared resources simultaneously after an uncoordinated failure. Therefore, prompt diagnosis and resolution are essential for maintaining the integrity and availability of your high-availability setup.

Scenario 1: SBD Service Failure Due to iSCSI Service Issues

iSCSI services are fundamental for SBD communication, facilitating data exchange between initiator (cluster nodes) and target (SBD server) nodes using iSCSI Qualified Names (IQNs). If these iSCSI services encounter failures, the SBD disks become inaccessible to the cluster nodes. This inaccessibility directly results in the SBD service failing to start, consequently preventing the Pacemaker service from running on any cluster node.

Diagnosing SBD and Pacemaker service failures stemming from iSCSI issues involves a systematic check of various service statuses and dependencies. This methodical approach helps pinpoint the exact point of failure within the high-availability stack.

Diagnosis Steps

  1. Check Pacemaker Cluster Status:
    Begin by verifying the overall health of your Pacemaker cluster. If Pacemaker is not running, this command will indicate a connection refusal, signaling a deeper underlying issue preventing the cluster services from starting.

    sudo pcs status

    Expected Output (if Pacemaker is not running):

    Error: error running crm_mon, is pacemaker running?
  error: Could not connect to launcher: Connection refused
  crm_mon: Connection to cluster failed: Connection refused

  2. Check Corosync Service Status:
    Corosync is the messaging layer for Pacemaker, responsible for cluster membership and communication. Verify that this foundational service is active and running. A healthy Corosync status suggests that nodes can communicate, but Pacemaker might still be held back by other dependencies like SBD.

    sudo systemctl status corosync

    Expected Output (if Corosync is running):

    ● corosync.service - Corosync Cluster Engine
     Loaded: loaded (/usr/lib/systemd/system/corosync.service; enabled; preset: disabled)
     Active: active (running) since Thu 2024-08-01 11:22:28 UTC; 1min 22s ago
       Docs: man:corosync
             man:corosync.conf
             man:corosync_overview
   Main PID: 12793 (corosync)
      Tasks: 9 (limit: 48956)
     Memory: 113.9M
        CPU: 401ms
     CGroup: /system.slice/corosync.service
             └─12793 /usr/sbin/corosync -f

Aug 01 11:22:43 node1 corosync[12793]:   [KNET  ] link: Resetting MTU for link 0 because host 2 joined
Aug 01 11:22:43 node1 corosync[12793]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Aug 01 11:22:43 node1 corosync[12793]:   [KNET  ] pmtud: PMTUD link change for host: 2 link: 0 from 469 to 1397
Aug 01 11:22:43 node1 corosync[12793]:   [KNET  ] pmtud: Global data MTU changed to: 1397
Aug 01 11:22:45 node1 corosync[12793]:   [QUORUM] Sync members[2]: 1 2
Aug 01 11:22:45 node1 corosync[12793]:   [QUORUM] Sync joined[1]: 2
Aug 01 11:22:45 node1 corosync[12793]:   [TOTEM ] A new membership (1.3e) was formed. Members joined: 2
Aug 01 11:22:45 node1 corosync[12793]:   [QUORUM] This node is within the primary component and will provide service.
Aug 01 11:22:45 node1 corosync[12793]:   [QUORUM] Members[2]: 1 2
Aug 01 11:22:45 node1 corosync[12793]:   [MAIN  ] Completed service synchronization, ready to provide service.

  3. Check Pacemaker Service Status:
    The Pacemaker service heavily relies on the SBD service to start successfully, as SBD provides the fencing mechanism. If Pacemaker is failing due to unmet dependencies, its status will reflect this, indicating that it’s inactive or dead.

    sudo systemctl status pacemaker

    Expected Output (if Pacemaker fails due to dependencies):

    ○ pacemaker.service - Pacemaker High Availability Cluster Manager
     Loaded: loaded (/usr/lib/systemd/system/pacemaker.service; enabled; preset: disabled)
     Active: inactive (dead) since Thu 2024-08-01 11:22:22 UTC; 2min 9s ago
       Docs: man:pacemakerd
             https://clusterlabs.org/pacemaker/doc/
Aug 01 11:24:28 node1 systemd[1]: Dependency failed for Pacemaker High Availability Cluster Manager.
Aug 01 11:24:28 node1 systemd[1]: pacemaker.service: Job pacemaker.service/start failed with result 'dependency'.

  4. List Pacemaker Service Dependencies:
    To confirm that SBD is indeed a required dependency for Pacemaker, examine the dependency tree. This command reveals which services must be active before Pacemaker can successfully launch. A failed or missing SBD service in this list strongly points to the problem.

    sudo systemctl list-dependencies pacemaker

    Expected Output (showing SBD as a failed dependency):

    pacemaker.service
● ├─corosync.service
● ├─dbus-broker.service
× ├─sbd.service
● ├─system.slice
● ├─resource-agents-deps.target
● └─sysinit.target

  5. Check SBD Service Status:
    Finally, directly check the status of the SBD service. This will provide explicit error messages if SBD failed to initialize, often indicating issues with device accessibility within a specified timeout.

    sudo systemctl status sbd

    Expected Output (if SBD fails):

    × sbd.service - Shared-storage based fencing daemon
     Loaded: loaded (/usr/lib/systemd/system/sbd.service; enabled; preset: disabled)
    Drop-In: /etc/systemd/system/sbd.service.d
             └─sbd_delay_start.conf
     Active: failed (Result: exit-code) since Thu 2024-08-01 11:24:28 UTC; 1min 56s ago
   Duration: 55min 34.369s
       Docs: man:sbd(8)
    Process: 12794 ExecStart=/usr/sbin/sbd $SBD_OPTS -p /run/sbd.pid watch (code=exited, status=1/FAILURE)
        CPU: 46ms

Aug 01 11:22:27 node1 systemd[1]: Starting Shared-storage based fencing daemon...
Aug 01 11:22:27 node1 sbd[12794]:  warning: open_any_device: Failed to open /dev/disk/by-id/scsi-360014056eadbecfeca042d4a66b9d779. Trying any other config>
Aug 01 11:24:28 node1 sbd[12794]:    error: open_any_device: No devices were available at start-up within 120 seconds.
Aug 01 11:24:28 node1 systemd[1]: sbd.service: Control process exited, code=exited, status=1/FAILURE
Aug 01 11:24:28 node1 systemd[1]: sbd.service: Failed with result 'exit-code'.
Aug 01 11:24:28 node1 systemd[1]: Failed to start Shared-storage based fencing daemon.

Resolution: Ensure iscsid and iscsi Services Are Enabled and Running

The inability of SBD to find available devices often points to issues with the underlying iSCSI services. These services are responsible for discovering and logging into iSCSI targets, making the shared SBD block devices available to the operating system. Ensuring their proper function is critical.

  1. Verify Core iSCSI Configuration:
    Before proceeding, confirm that your iSCSI setup adheres to the best practices outlined in the RHEL documentation for Pacemaker clusters. This includes proper network configuration, firewall rules allowing iSCSI traffic, and correct iSCSI initiator/target settings.

  2. Enable iscsi and iscsid Services:
    The iscsid service is the iSCSI daemon responsible for managing iSCSI sessions, while the iscsi service handles the login and scanning of iSCSI devices during system startup. Both must be enabled to ensure automatic discovery and connection to SBD disks.

    sudo systemctl enable iscsi
sudo systemctl enable iscsid

  3. Start and Verify Services:
    After enabling, start both services and then confirm their operational status. An “active (exited)” status for iscsi.service is normal, as it performs login operations and then exits, leaving iscsid to maintain sessions.

    sudo systemctl status iscsi
sudo systemctl status iscsid

    Expected Output (if services are running):

    ● iscsi.service - Login and scanning of iSCSI devices
     Loaded: loaded (/usr/lib/systemd/system/iscsi.service; enabled; preset: enabled)
     Active: active (exited) since Thu 2024-08-01 10:28:36 UTC; 1h 0min ago
       Docs: man:iscsiadm(8)
             man:iscsid(8)
    Process: 11174 ExecStart=/usr/sbin/iscsiadm -m node --loginall=automatic (code=exited, status=24)
   Main PID: 11174 (code=exited, status=24)
        CPU: 33ms

Aug 01 10:28:36 nfs-0 systemd[1]: Starting Login and scanning of iSCSI devices...
Aug 01 10:28:36 nfs-0 iscsiadm[1823]: Logging in to [iface: default, target: iqn.2006-04.nfs.local:nfs, portal: 10.0.0.17,3260]
Aug 01 10:28:36 nfs-0 iscsiadm[1823]: Logging in to [iface: default, target: iqn.2006-04.nfs.local:nfs, portal: 10.0.0.18,3260]
Aug 01 10:28:36 nfs-0 iscsiadm[1823]: Logging in to [iface: default, target: iqn.2006-04.nfs.local:nfs, portal: 10.0.0.19,3260]
Aug 01 10:28:36 nfs-0 systemd[1]: Started Login and scanning of iSCSI devices.

    Once these services are confirmed to be running, attempt to restart the SBD service and then the Pacemaker cluster. This should allow SBD to discover and utilize the shared devices, enabling Pacemaker to start.

Scenario 2: SBD Service Failure Due to Configuration Issues

SBD service failures can also originate from misconfigurations within the SBD setup itself. These issues typically involve either the complete absence of SBD configuration files or inaccuracies within them, such as incorrect device paths or syntax errors. A proper SBD configuration is vital for the SBD daemon to correctly identify and use the shared fencing devices.

Diagnosis Steps

  1. Validate SBD Configuration Existence:
    Confirm that Pacemaker has a SBD stonith resource configured. This command checks the cluster’s high-level SBD resource definition.

    sudo pcs stonith config sbd

    Expected Output (if SBD configuration exists):

    Resource: sbd (class=stonith type=fence_sbd)
  Attributes: sbd-instance_attributes
    devices=/dev/disk/by-id/scsi-360014056eadbecfeca042d4a66b9d779,/dev/disk/by-id/scsi-36001405cbac988092e448059d25d1a4a,/dev/disk/by-
        id/scsi-36001405a29a443e4a6e4ceeae822e5eb
  Operations:
    monitor: sbd-monitor-interval-600
      interval=600
      timeout=15

  2. Validate /etc/sysconfig/sbd File:
    Inspect the local SBD configuration file on each cluster node. This file contains crucial parameters for the SBD daemon, including the SBD_DEVICE variable that specifies the path to the SBD disks. Any discrepancies here, like incorrect device IDs or missing critical parameters, can prevent SBD from starting.

    sudo cat /etc/sysconfig/sbd

    Recommended Parameters:

    SBD_PACEMAKER=yes
SBD_STARTMODE=always
SBD_DELAY_START=no
SBD_WATCHDOG_DEV=/dev/watchdog
SBD_WATCHDOG_TIMEOUT=5
SBD_TIMEOUT_ACTION=flush,reboot
SBD_MOVE_TO_ROOT_CGROUP=auto
SBD_OPTS=
SBD_DEVICE="/dev/disk/by-id/scsi-360014056eadbecfeca042d4a66b9d77;/dev/disk/by-id/scsi-36001405cbac988092e448059d25d1a4a;/dev/disk/by-id/scsi-36001405a29a443e4a6e4ceeae822e5eb"

    Ensure that the SBD_DEVICE parameter accurately lists all SBD device paths, separated by semicolons, and that all other parameters are correctly set according to your cluster’s requirements.

Resolution: Ensure the Correct SBD Configuration

Carefully review and correct any errors found in the SBD configuration files. This often involves ensuring that device paths are correct and that essential SBD parameters are present and properly configured.

  1. Correct SBD_DEVICE Paths:
    Update the SBD_DEVICE variable in /etc/sysconfig/sbd to include the correct /dev/disk/by-id/ paths for all your SBD devices. Using by-id paths is crucial for persistent device naming across reboots.

  2. Set Essential Parameters:
    Ensure SBD_PACEMAKER=yes is set to integrate SBD with Pacemaker. Confirm SBD_STARTMODE=always for consistent SBD startup. Adjust SBD_WATCHDOG_TIMEOUT and SBD_TIMEOUT_ACTION according to your fencing strategy, typically allowing enough time for healthy nodes to respond while enforcing aggressive fencing for unresponsive ones.

  3. Reload and Restart:
    After making changes to /etc/sysconfig/sbd, restart the SBD service and then the Pacemaker cluster to apply the new configuration.

    sudo systemctl daemon-reload
sudo systemctl restart sbd
sudo systemctl restart pacemaker

Scenario 3: iSCSI Devices Not Available on Cluster Nodes

A common underlying issue for SBD failures is the unavailability of the iSCSI block devices themselves on the cluster nodes. Even if the iSCSI services are running, if the actual disks aren’t presented or logged in correctly, SBD will not function. This problem manifests as SBD disks not appearing in standard block device listings.

Diagnosis Steps

  1. Check for iSCSI Devices:
    Use lsscsi or lsblk to list block devices. If your SBD devices are not visible, it indicates a problem with iSCSI discovery or login.

    sudo lsscsi

    Expected Output (if SBD devices are missing):

    [0:0:0:0]    disk    Msft     Virtual Disk     1.0   /dev/sda
[1:0:1:0]    disk    Msft     Virtual Disk     1.0   /dev/sdb
[5:0:0:0]    cd/dvd  Msft     Virtual CD/ROM   1.0   /dev/sr0


    sudo lsblk

    Expected Output (if SBD devices are missing):

    NAME              MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda                 8:0    0   64G  0 disk
├─sda1              8:1    0  200M  0 part /boot/efi
├─sda2              8:2    0  500M  0 part /boot
├─sda3              8:3    0    1M  0 part
└─sda4              8:4    0 63.3G  0 part
  ├─rootvg-tmplv  253:0    0    2G  0 lvm  /tmp
  ├─rootvg-usrlv  253:1    0   10G  0 lvm  /usr
  ├─rootvg-homelv 253:2    0    1G  0 lvm  /home
  ├─rootvg-varlv  253:3    0    8G  0 lvm  /var
  └─rootvg-rootlv 253:4    0    2G  0 lvm  /
sdb                 8:16   0   16G  0 disk
└─sdb1              8:17   0   16G  0 part /mnt
sr0                11:0    1  628K  0 rom

  2. Validate Initiator Names:
    Ensure that each cluster node has a unique and correctly configured iSCSI initiator name. This is crucial for distinguishing nodes when connecting to iSCSI targets.

    sudo cat /etc/iscsi/initiatorname.iscsi

    Expected Output (example for different nodes):

    # On node1
InitiatorName=iqn.2006-04.nfs-0.local:nfs-0


    # On node2
InitiatorName=iqn.2006-04.nfs-1.local:nfs-1

  3. List SBD Devices in Configuration:
    Review the SBD_DEVICE entry in /etc/sysconfig/sbd to confirm the expected device paths. These paths will be used in subsequent checks.

    sudo grep SBD_DEVICE /etc/sysconfig/sbd

    Expected Output:

    # SBD_DEVICE specifies the devices to use for exchanging sbd messages
SBD_DEVICE="/dev/disk/by-id/scsi-360014056eadbecfeca042d4a66b9d779;/dev/disk/by-id/scsi-36001405cbac988092e448059d25d1a4a;/dev/disk/by-id/scsi-36001405a29a443e4a6e4ceeae822e5eb"

  4. Check SBD Device Accessibility:
    Attempt to list SBD messages on each configured SBD device using the sbd command. If the devices are not accessible, you will see “unreadable!” errors, confirming that the SBD daemon cannot interact with the storage.

    sudo /usr/sbin/sbd -d /dev/disk/by-id/scsi-360014056eadbecfeca042d4a66b9d779 list

    Expected Output (if device is unreadable):

    == disk /dev/disk/by-id/scsi-360014056eadbecfeca042d4a66b9d779 unreadable!
sbd failed; please check the logs.

    Repeat this for all devices listed in SBD_DEVICE.

Resolution: Enable Automatic Login to iSCSI Target Servers

The core of this issue is the failure to properly log in and maintain sessions with the iSCSI target servers that host the SBD devices. Manually logging in and configuring automatic login ensures that these devices are persistently available to the cluster nodes.

  1. Perform iSCSI Discovery:
    Execute an iscsiadm discovery operation on each cluster node. This command identifies available iSCSI targets on your network, displaying their IP addresses and default port (3260).

    sudo iscsiadm -m discovery

    Expected Output:

    10.0.0.17:3260 via sendtargets
10.0.0.18:3260 via sendtargets
10.0.0.19:3260 via sendtargets

  2. Discover Specific iSCSI Target:
    Perform a more targeted discovery to identify the specific iSCSI Qualified Name (IQN) of your SBD target server. This IQN is essential for logging into the correct target.

    sudo iscsiadm -m discovery --type=st --portal=10.0.0.17:3260

    Expected Output:

    10.0.0.17:3260,1 iqn.2006-04.dbnw1.local:dbnw1
10.0.0.17:3260,1 iqn.2006-04.ascsnw1.local:ascsnw1
10.0.0.17:3260,1 iqn.2006-04.nfs.local:nfs

    Identify the IQN corresponding to your SBD device (e.g., iqn.2006-04.nfs.local:nfs).

  3. Log In to the iSCSI Device:
    Manually log into the identified iSCSI target. This establishes an iSCSI session, making the block device available to the operating system.

    sudo iscsiadm -m node -T iqn.2006-04.nfs.local:nfs --login --portal=10.0.0.17:3260

    Expected Output:

    Logging in to [iface: default, target: iqn.2006-04.nfs.local:nfs, portal: 10.0.0.17,3260]
Login to [iface: default, target: iqn.2006-04.nfs.local:nfs, portal: 10.0.0.17,3260] successful.

  4. Enable Automatic Login:
    To ensure persistence across reboots, configure the iSCSI initiator to automatically log into the target. This modifies the iSCSI node database entry for the target.

    sudo iscsiadm -m node -p 10.0.0.17:3260 -T iqn.2006-04.nfs.local:nfs --op=update --name=node.startup --value=automatic

  5. Validate iSCSI Device Availability:
    After logging in and enabling automatic startup, verify that the SBD iSCSI devices are now visible using lsscsi.

    sudo lsscsi

    Expected Output (with new SBD device):

    [0:0:0:0]    disk    Msft     Virtual Disk     1.0   /dev/sda
[1:0:1:0]    disk    Msft     Virtual Disk     1.0   /dev/sdb
[5:0:0:0]    cd/dvd  Msft     Virtual CD/ROM   1.0   /dev/sr0
[6:0:0:0]    disk    LIO-ORG  sbdnfs           4.0   /dev/sdc

  6. Repeat for All Devices and Nodes:
    Crucially, repeat steps 1-5 for all iSCSI devices configured for SBD on every cluster node. Consistency across all nodes is essential for reliable cluster operation.

Once all iSCSI devices are correctly discovered, logged in, and configured for automatic startup, the lsscsi and lsblk commands should clearly display the SBD disks. This confirms their availability to the SBD service.

Example of lsscsi and lsblk with SBD disks:

sudo lsscsi


[0:0:0:0]    disk    Msft     Virtual Disk     1.0   /dev/sda
[1:0:1:0]    disk    Msft     Virtual Disk     1.0   /dev/sdb
[5:0:0:0]    cd/dvd  Msft     Virtual CD/ROM   1.0   /dev/sr0
[6:0:0:0]    disk    LIO-ORG  sbdnfs           4.0   /dev/sdc
[7:0:0:0]    disk    LIO-ORG  sbdnfs           4.0   /dev/sdd
[8:0:0:0]    disk    LIO-ORG  sbdnfs           4.0   /dev/sde


sudo lsblk


NAME              MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda                 8:0    0   64G  0 disk
├─sda1              8:1    0  200M  0 part /boot/efi
├─sda2              8:2    0  500M  0 part /boot
├─sda3              8:3    0    1M  0 part
└─sda4              8:4    0 63.3G  0 part
    ├─rootvg-tmplv  253:0    0    2G  0 lvm  /tmp
    ├─rootvg-usrlv  253:1    0   10G  0 lvm  /usr
    ├─rootvg-homelv 253:2    0    1G  0 lvm  /home
    ├─rootvg-varlv  253:3    0    8G  0 lvm  /var
    └─rootvg-rootlv 253:4    0    2G  0 lvm  /
sdb                 8:16   0   16G  0 disk
└─sdb1              8:17   0   16G  0 part /mnt
sdc                 8:32   0   50M  0 disk
sdd                 8:48   0   50M  0 disk
sde                 8:64   0   50M  0 disk
sr0                11:0    1  628K  0 rom

Scenario 4: Node Fails to Rejoin Cluster After Being Fenced

A critical aspect of cluster management is the ability for a node to gracefully rejoin the cluster after an outage or intentional fencing. If the SBD slot designated for that node is not in a clean state, the node will be prevented from rejoining. This typically happens when a node was previously fenced, and the SBD device still holds a “reset” or “pending” status for that node, indicating an unclean shutdown.

Diagnosis Steps

  1. List SBD Device Status:
    First, ensure the SBD devices are visible on the system using lsscsi. Then, check the state of the SBD slots on each device using the sbd -d <device> list command. This will show the current status of each node’s slot on that specific SBD disk.

    sudo lsscsi

    Example Output:

    [0:0:0:0]    disk    Msft     Virtual Disk     1.0   /dev/sda
[1:0:1:0]    disk    Msft     Virtual Disk     1.0   /dev/sdb
[5:0:0:0]    cd/dvd  Msft     Virtual CD/ROM   1.0   /dev/sr0
[6:0:0:0]    disk    LIO-ORG  sbdnfs           4.0   /dev/sdc
[7:0:0:0]    disk    LIO-ORG  sbdnfs           4.0   /dev/sdd
[8:0:0:0]    disk    LIO-ORG  sbdnfs           4.0   /dev/sde

    Now, check the SBD slot status for each SBD device:

    sudo sbd -d /dev/sdc list


    0       node1   clear
1       node2   reset   node1


    sudo sbd -d /dev/sdd list


    0       node1   clear
1       node2   reset   node1


    sudo sbd -d /dev/sde list


    0       node1   clear
1       node2   reset   node1

    In this example, node2 is in a reset state, indicating it was fenced by node1 and its slot needs clearing.

Resolution: Clear the SBD Slot

Clearing the SBD slot informs the SBD mechanism that the previously fenced node has been acknowledged and can attempt to rejoin the cluster. This is particularly important when SBD_STARTMODE is set to clean.

  1. Validate SBD_STARTMODE Parameter:
    Examine the SBD_STARTMODE parameter in /etc/sysconfig/sbd. This setting dictates the behavior of the SBD daemon upon startup, specifically concerning fenced nodes.

    sudo grep -i SBD_STARTMODE /etc/sysconfig/sbd

    Expected Output:

    SBD_STARTMODE=clean

    * If SBD_STARTMODE=always, the node will attempt to rejoin the cluster even if it was previously fenced, disregarding the SBD slot state.
    * If SBD_STARTMODE=clean (recommended for safety), the node will only rejoin after its SBD slot has been manually cleared, preventing a fenced node from automatically rejoining without intervention.

  2. Clear the SBD Slot:
    Use the sbd message clear command to explicitly clear the SBD slot for the problematic node. This action updates the SBD device, marking the node’s slot as clean, thus allowing it to rejoin the cluster.

    # To clear the local node's slot on a specific device
sudo sbd -d <SBD_DEVICE> message LOCAL clear

# Alternatively, if you know the specific node name to clear
sudo sbd -d <DEVICE_NAME> message <NODENAME> clear

    Example:

    sudo sbd -d /dev/sdc message node2 clear
sudo sbd -d /dev/sdd message node2 clear
sudo sbd -d /dev/sde message node2 clear

    Note: Replace <SBD_DEVICE>, <DEVICE_NAME>, and <NODENAME> with your specific device and node details.

  3. Start Cluster Service:
    After clearing the SBD slots, attempt to start the cluster services on the node. If the service still fails to start, it might indicate persistent iSCSI session issues. In such cases, a full iSCSI session reset might be necessary.

    sudo systemctl start pacemaker

    If the cluster service fails again, perform a complete iSCSI session logout and login sequence:

    sudo iscsiadm -m node -u # Logs out from all iSCSI nodes
sudo iscsiadm -m node -l # Logs in to all iSCSI nodes configured for automatic startup

    Then, re-attempt to start the Pacemaker service.

Scenario 5: SBD Service Fails After Adding a New SBD Device

Adding new SBD devices to an existing Pacemaker cluster can sometimes lead to the SBD service failing to start or exhibiting unusual behavior. This often occurs because SBD, when integrated with Pacemaker, is managed by the cluster itself, and manual starts/stops might not properly integrate the new device into the cluster’s fencing mechanism. The cluster needs to be aware of and control all SBD devices.

Symptoms

  • SBD Service Status Errors: When checking the SBD service status (sudo systemctl status sbd), you observe errors such as “sbd failed; please check the logs” and “Failed to start sbd.service: Operation refused.” This indicates that the systemd service is unable to manage SBD effectively.
  • Failed Test Messages: Sending a test message to a node via an SBD device also results in the “sbd failed; please check the logs” error, confirming that SBD is not functioning correctly.
    sudo sbd -d /dev/disk/by-id/scsi-360014056eadbecfeca042d4a66b9d779 message node1 test

    Expected Output:
    sbd failed; please check the logs.
  • Log Entries: The /var/log/messages file will contain diagnostic information, often showing errors related to message delivery or recipient issues, such as:
    Mar  2 06:58:06 node1 sbd[11105]: /dev/disk/by-id/scsi-360014056eadbecfeca042d4a66b9d779:    error: slot_msg: slot_msg(): No recipient / cmd specified.
Mar  2 06:58:06 node1 sbd[11104]: warning: messenger: Process 11105 failed to deliver!
  • Blank sbd -d list Output: A critical symptom is a blank output from sbd -d <device> list. This suggests that SBD is not properly initialized or cannot communicate with the devices to read their status. Refer to Red Hat’s solutions for “SBD device list showing blank output” for more details.

Resolution: Restart the Pacemaker Cluster

When SBD is configured as a fencing device and is under Pacemaker’s control, direct manual manipulation of the sbd.service can lead to inconsistencies. To properly integrate a newly added SBD device, the entire Pacemaker cluster needs a coordinated restart. This ensures that the cluster rediscovers and takes control of all configured SBD devices, incorporating them into its fencing strategy.

  1. Stop and Restart All Cluster Services:
    Initiate a controlled stop and restart of the entire Pacemaker cluster across all nodes. This is the most reliable way to ensure that all cluster components, including SBD, are properly reinitialized and synchronized with the updated configuration.

    sudo pcs cluster stop --all
sudo pcs cluster start --all

  2. Verify SBD Service Status:
    After the cluster restart, check the status of the SBD service. It should now be running actively, indicating successful initialization under cluster control.

    sudo systemctl status sbd.service

    Expected Output:

    ● sbd.service - Shared-storage based fencing daemon
   Loaded: loaded (/usr/lib/systemd/system/sbd.service; enabled; vendor preset: disabled)
   Active: active (running)

  3. Validate SBD Device List:
    Finally, confirm that the sbd -d list command no longer produces a blank output and correctly displays the status of all configured SBD slots, including the newly added device. This indicates that SBD is fully operational and integrated with the cluster.

    sudo sbd -d /dev/disk/by-id/scsi-360014056eadbecfeca042d4a66b9d779 list

    Expected Output:

    0   node1   clear
1   node2   clear

General Troubleshooting Best Practices for RHEL Clusters

Beyond the specific scenarios, maintaining a robust troubleshooting approach for RHEL Pacemaker clusters is key. Always ensure:

  • Network Connectivity: Verify all network paths between cluster nodes and between nodes and iSCSI target servers are open and stable. Check firewall rules (firewalld or iptables).
  • System Logs: Regularly review /var/log/messages, journalctl -u sbd, journalctl -u pacemaker, and journalctl -u corosync for any errors or warnings.
  • Resource Management: Ensure nodes have sufficient CPU, memory, and I/O capacity, especially for iSCSI operations.
  • Watchdog Device: Confirm the watchdog device is functional and configured correctly, as SBD relies on it for system-level fencing.
  • Time Synchronization: Accurate time synchronization (NTP) across all cluster nodes is crucial for proper heartbeat and messaging.

Next Steps

If, after following this comprehensive troubleshooting guide, your SBD service and Pacemaker cluster continue to experience issues, further investigation is warranted. It is recommended to collect detailed system logs from your Red Hat Enterprise Linux systems. Generate an sos report which aggregates system information, configuration files, and logs into a single archive. This report is invaluable for advanced diagnostics.

For assistance with persistent or complex issues, gather your sos report and contact Red Hat Support or your cloud provider’s technical support team. Providing a complete sos report will significantly expedite the resolution process.

Have you encountered similar SBD issues in your RHEL Pacemaker clusters? Share your experiences, challenges, or successful resolutions in the comments below. Your insights can help others facing similar problems!

Post a Comment