Resolving SBD Service Failures in SUSE Pacemaker Clusters for Virtual Machines
This article is specifically designed for IT professionals and system administrators responsible for maintaining high availability within SUSE Enterprise Linux (SLES) Pacemaker clusters, particularly those deployed on Linux virtual machines. It meticulously outlines common scenarios where the STONITH Block Device (SBD) service fails to initialize or operate correctly. Furthermore, it provides comprehensive, actionable guidance for diagnosing these issues and implementing effective resolutions to restore cluster integrity and functionality.
Understanding the STONITH Block Device (SBD) Mechanism¶
The STONITH Block Device (SBD) serves as a critical fencing mechanism in Pacemaker clusters, essential for preventing split-brain scenarios and ensuring data consistency during node failures. At its core, an SBD device requires at least one additional virtual machine (VM) configured to function as an Internet Small Computer System Interface (iSCSI) target server. This dedicated server is responsible for provisioning the SBD device, which can be shared amongst various Pacemaker clusters to provide robust, shared-storage based fencing.
One significant advantage of adopting SBD for fencing is its operational familiarity. For organizations already leveraging SBD devices in on-premises deployments, integrating them into cloud environments like Microsoft Azure necessitates minimal changes to existing Pacemaker cluster management practices. For Pacemaker clusters hosted in Microsoft Azure that utilize SBD for storage protection, two primary setup options are available. These methods facilitate the secure and reliable operation of your high-availability solutions. These include configuring SBD with a dedicated iSCSI target server or integrating SBD with an Azure shared disk, each offering distinct architectural benefits.
Diagnosing SBD Service Issues¶
Accurately diagnosing the root cause of an SBD service failure is the foundational step in any troubleshooting process. The following methodical diagnostic sequence illustrates how to determine if a cluster startup or operational issue is indeed caused by a non-functional SBD service. This systematic approach ensures that you precisely identify the dependent service failure impacting your Pacemaker cluster’s health and availability.
Step 1: Check the Cluster Status
Initiate the diagnostic process by examining the overall status of your Pacemaker cluster. A common symptom of an SBD service failure is that the cluster appears unavailable or is reporting critical errors when queried.
sudo crm status
Expected Output for a Failed Cluster:
ERROR: status: crm_mon (rc=102): Error: cluster is not available on this node
This error message signifies that the
crm command-line utility cannot establish a connection with the cluster manager on the local node. This strongly suggests that a core cluster service, such as Pacemaker itself, is either not running or is severely compromised, warranting further investigation into its dependencies.
Step 2: Verify Pacemaker Service Status
Following the cluster status check, directly inspect the operational state of the Pacemaker service. Pacemaker is inherently dependent on several underlying services to function correctly, and its failure often indicates that one or more of these prerequisite services are not operational.
sudo systemctl status pacemaker
Expected Output for a Failed Pacemaker Service:
pacemaker.service - Pacemaker High Availability Cluster Manager
Loaded: loaded (/usr/lib/systemd/system/pacemaker.service; enabled; vendor preset: disabled)
Active: inactive (dead) since Thu 2024-08-01 04:56:39 UTC; 2min 39s ago
Docs: man:pacemakerd
https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html-single/Pacemaker_Explained/index.html
Aug 01 04:59:07 nfs-0 systemd[1]: Dependency failed for Pacemaker High Availability Cluster Manager.
Aug 01 04:59:07 nfs-0 systemd[1]: pacemaker.service: Job pacemaker.service/start failed with result 'dependency'.
The “inactive (dead)” status, coupled with the explicit “Dependency failed” message, clearly indicates that the Pacemaker service could not start successfully. This failure points to an issue with one of its essential dependencies, compelling a deeper dive into the service’s prerequisites to identify the specific culprit.
Step 3: List Pacemaker Service Dependencies
To pinpoint the exact dependency causing the Pacemaker service to fail, list its complete service dependency tree. This command reveals all services that must be active and healthy before the Pacemaker high-availability cluster manager can successfully launch.
sudo systemctl list-dependencies pacemaker
Expected Dependency Output:
pacemaker.service
● ├─corosync.service
● ├─dbus.service
● ├─sbd.service
● ├─system.slice
● ├─resource-agents-deps.target
● └─sysinit.target
This output distinctly shows
sbd.service as a direct and critical dependency of pacemaker.service, alongside other core services like corosync.service and dbus.service. This is a vital piece of information that guides the next step in isolating the problem.
Step 4: Check Status of Core Dependency Services
Proceed by individually checking the status of each identified core dependency. For instance, verify the status of Corosync, which is fundamental for inter-node communication and quorum management within the cluster.
sudo systemctl status corosync
Example Active Corosync Output:
corosync.service - Corosync Cluster Engine
Loaded: loaded (/usr/lib/systemd/system/corosync.service; disabled; vendor preset: disabled)
Active: active (running) since Thu 2024-08-01 04:49:15 UTC; 38s ago
Process: 23972 ExecStop=/usr/share/corosync/corosync stop (code=exited, status=0/SUCCESS)
Process: 24075 ExecStart=/usr/share/corosync/corosync start (code=exited, status=0/SUCCESS)
Main PID: 24094 (corosync)
Tasks: 2 (limit: 4096)
CGroup: /system.slice/corosync.service
└─24094 corosync
Aug 01 04:49:15 nfs-0 corosync[24094]: [TOTEM ] A new membership (10.0.0.6:32) was formed. Members joined: 2
Aug 01 04:49:15 nfs-0 corosync[24094]: [CPG ] downlist left_list: 0 received in state 2
Aug 01 04:49:15 nfs-0 corosync[24094]: [VOTEQ ] Waiting for all cluster members. Current votes: 1 expected_votes: 2
Aug 01 04:49:15 nfs-0 corosync[24094]: [QUORUM] This node is within the primary component and will provide service.
Aug 01 04:49:15 nfs-0 corosync[24094]: [QUORUM] Members[2]: 1 2
Aug 01 04:49:15 nfs-0 corosync[24094]: [MAIN ] Completed service synchronization, ready to provide service.
Aug 01 04:49:15 nfs-0 corosync[24075]: Starting Corosync Cluster Engine (corosync): [ OK ]
If the Corosync service is reported as active and running, this suggests that the problem likely does not reside with basic cluster communication. The next logical step, therefore, is to investigate the status of the SBD service itself, as it remains a critical dependency.
Step 5: Check SBD Service Status
This final, focused check on the SBD service is typically where the core issue behind Pacemaker’s dependency failure is revealed. The output will explicitly state whether SBD is running or has encountered an error.
sudo systemctl status sbd
Expected Output for a Failed SBD Service:
● sbd.service - Shared-storage based fencing daemon
Loaded: loaded (/usr/lib/systemd/system/sbd.service; enabled; vendor preset: disabled)
Active: failed (Result: timeout) since Thu 2024-08-01 04:59:07 UTC; 3min 3s ago
Docs: man:sbd(8)
Process: 25251 ExecStart=/usr/sbin/sbd $SBD_OPTS -p //run/sbd.pid watch (code=killed, signal=TERM)
Aug 01 04:57:37 nfs-0 systemd[1]: Starting Shared-storage based fencing daemon...
Aug 01 04:57:37 nfs-0 sbd[25251]: warning: open_any_device: Failed to open /dev/disk/by-id/scsi-360014056eadbecfeca042d4a66b9d779. Trying any o...ithin 120s
Aug 01 04:59:07 nfs-0 systemd[1]: sbd.service: Start operation timed out. Terminating.
Aug 01 04:59:07 nfs-0 systemd[1]: Failed to start Shared-storage based fencing daemon.
Aug 01 04:59:07 nfs-0 systemd[1]: sbd.service: Unit entered failed state.
Aug 01 04:59:07 nfs-0 systemd[1]: sbd.service: Failed with result 'timeout'.
This output conclusively confirms that the SBD service failed to start, indicated by the “failed (Result: timeout)” status and the explicit error message “Failed to start Shared-storage based fencing daemon.” This direct confirmation allows you to proceed confidently with troubleshooting the specific underlying causes of the SBD service failure.
Common Causes and Resolutions for SBD Service Failures¶
Understanding the common culprits behind SBD service failures is paramount for efficient troubleshooting and swift resolution. This section delves into various typical scenarios that lead to SBD service instability and provides detailed, step-by-step resolutions for each identified cause.
Cause 1: SBD Service Failed Due to iSCSI Failure¶
One of the most prevalent reasons for SBD service failure stems from underlying issues with iSCSI connectivity. If the Pacemaker service remains inoperative and the SBD service exhibits a failed state on both cluster nodes, instability within the iSCSI services is a highly probable cause. iSCSI services depend on the iSCSI Qualified Name (IQN) for seamless and secure communication between the initiator (your cluster node) and the target (the SBD device server). When these crucial iSCSI services are improperly configured, disabled, or encounter operational issues, the SBD disks become inaccessible, leading directly to the failure of both the SBD service and, consequently, the Pacemaker cluster.
Resolution:
To mitigate SBD failures related to iSCSI, it is essential to ensure that your iSCSI services are correctly configured, enabled, and actively running on all cluster nodes.
-
Validate Setup Configuration:
It is crucial to first verify that your Pacemaker cluster setup, particularly the iSCSI components, adheres strictly to the recommended configurations. Consult the official documentation for SUSE - set up Pacemaker on SUSE Linux Enterprise Server in Azure. Any deviations or misconfigurations can prevent iSCSI services from initializing properly, leading to SBD inaccessibility. -
Enter Cluster Maintenance Mode:
Before implementing any significant changes to the cluster’s core components, it is a best practice to place the entire cluster into maintenance mode. This action is critical for preventing unintended fencing actions, resource movements, or service disruptions while you perform troubleshooting and configuration adjustments.sudo crm configure property maintenance-mode=true -
Enable iSCSI Services:
Confirm that both theiscsid(iSCSI daemon, responsible for managing iSCSI connections) andiscsi(service for iSCSI login and scanning of devices) services are enabled to automatically start upon system boot. This ensures persistent and reliable connectivity to your SBD devices across reboots.sudo systemctl enable iscsisudo systemctl enable iscsid
Enabling these services is a proactive step that ensures the system consistently attempts to establish and maintain iSCSI connections whenever it becomes operational. -
Verify iSCSI Service Status:
After enabling the services, proceed to verify their current operational status. This step confirms that bothiscsidandiscsiservices are actively running as expected, a prerequisite for SBD functionality.sudo systemctl status iscsisudo systemctl status iscsid
Expected Output for Working iSCSI Services:iscsi.service - Login and scanning of iSCSI devices Loaded: loaded (/usr/lib/systemd/system/iscsi.service; enabled; vendor preset: enabled) Active: active (exited) since Thu 2024-08-01 04:18:51 UTC; 31min ago Docs: man:iscsiadm(8) man:iscsid(8) Main PID: 1823 (code=exited, status=0/SUCCESS) Tasks: 0 (limit: 4096) CGroup: /system.slice/iscsi.service Aug 01 04:18:51 nfs-0 systemd[1]: Starting Login and scanning of iSCSI devices... Aug 01 04:18:51 nfs-0 iscsiadm[1823]: Logging in to [iface: default, target: iqn.2006-04.nfs.local:nfs, portal: 10.0.0.17,3260] Aug 01 04:18:51 nfs-0 iscsiadm[1823]: Logging in to [iface: default, target: iqn.2006-04.nfs.local:nfs, portal: 10.0.0.18,3260] Aug 01 04:18:51 nfs-0 iscsiadm[1823]: Logging in to [iface: default, target: iqn.2006-04.nfs.local:nfs, portal: 10.0.0.19,3260] Aug 01 04:18:51 nfs-0 systemd[1]: Started Login and scanning of iSCSI devices.
It is important to note thatiscsi.servicecommonly showsActive: active (exited)as it completes its login and scanning tasks then terminates. Conversely,iscsid.serviceshould consistently showActive: active (running)as it acts as a daemon. -
Exit Cluster Maintenance Mode:
Once you have meticulously verified that the iSCSI services are correctly configured, enabled, and running without issues, it is safe to remove the cluster from maintenance mode. This action allows the cluster to resume its normal operational state and triggers Pacemaker to attempt a restart, ideally with the SBD service now fully functional and accessible.sudo crm configure property maintenance-mode=false
This final step is crucial for the cluster to re-evaluate its current state, initiate necessary services, and bring up high-availability resources as required by your configuration.
Cause 2: SBD Configuration Issues¶
Incorrect or malformed configurations within the SBD setup files can severely impair the service’s functionality. This category of issues includes common mistakes such as specifying incorrect paths to SBD devices, misnaming devices, or introducing subtle syntax errors within the configuration files. Even minor discrepancies in these critical files can prevent the SBD service from starting reliably or operating as intended.
Resolution 1: Verify SBD Configuration File (/etc/sysconfig/sbd)
The primary SBD configuration file, /etc/sysconfig/sbd, must contain the recommended parameters and the precise paths to all SBD devices. Pay particular attention to the SBD_DEVICE parameter, which is crucial for defining the block devices used by SBD.
SBD_PACEMAKER=yes
SBD_STARTMODE=always
SBD_DELAY_START=no
SBD_WATCHDOG_DEV=/dev/watchdog
SBD_WATCHDOG_TIMEOUT=5
SBD_TIMEOUT_ACTION=flush,reboot
SBD_MOVE_TO_ROOT_CGROUP=auto
SBD_OPTS=
SBD_DEVICE="/dev/disk/by-id/scsi-xxxxxxxxxxxxxxxxxx;/dev/disk/by-id/scsi-xxxxxxxxxxxxxxxxx;/dev/disk/by-id/scsi-xxxxxxxxxxxxxxxxxx"
Ensure that the
SBD_DEVICE parameter accurately lists the full /dev/disk/by-id/ paths for all your SBD devices, with each path separated by a semicolon. Utilizing by-id paths is critically important for ensuring persistent device naming, which remains consistent across system reboots. Additionally, confirm that SBD_PACEMAKER=yes is correctly set, signifying SBD’s proper integration with the Pacemaker cluster manager.
Resolution 2: Verify STONITH Resource Configuration
Beyond the SBD configuration file, the STONITH resource itself within your Pacemaker cluster’s configuration must be correctly defined and aligned with your SBD setup. This configuration explicitly dictates how the cluster interacts with and utilizes the SBD fencing mechanism.
sudo crm configure show
Expected STONITH Resource Output (Example):
primitive stonith-sbd stonith:external/sbd \
params pcmk_delay_max=30s \
op monitor interval="600" timeout="15"
stonith-enabled=true \
stonith-timeout=144 \
Crucially, confirm that the
stonith-sbd primitive is explicitly defined using stonith:external/sbd and that stonith-enabled=true. The various params and op monitor values should be meticulously aligned with your specific cluster’s requirements, guaranteeing appropriate fencing behavior and proactive monitoring. Any deviation, missing parameters, or incorrect values in this configuration can lead to unpredictable behavior or outright failure of the fencing mechanism.
Cause 3: iSCSI Devices Aren’t Available on Cluster Nodes¶
A frequent and challenging issue arises when the iSCSI-backed SBD disks, despite being seemingly correctly configured, are simply not detected or accessible on one or more cluster nodes. This can occur due to various reasons, including the iSCSI initiator failing to log into its target, or a disruption in network connectivity to the iSCSI target server.
When you execute standard commands such as lsscsi or lsblk, the SBD disks will be noticeably absent from the output, which is a clear indicator that they are not recognized or visible by the operating system.
sudo lsscsi
Example Output (Missing SBD Disks):
[0:0:0:0] disk Msft Virtual Disk 1.0 /dev/sda
[0:0:0:1] disk Msft Virtual Disk 1.0 /dev/sdb
[1:0:0:0] disk Msft Virtual Disk 1.0 /dev/sdc
[1:0:0:1] disk Msft Virtual Disk 1.0 /dev/sdd
sudo lsblk
Example Output (Missing SBD Disks):
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 100G 0 disk
├─sda1 8:1 0 2M 0 part
├─sda2 8:2 0 512M 0 part /boot/efi
├─sda3 8:3 0 1G 0 part /boot
└─sda4 8:4 0 98.5G 0 part /
sdb 8:16 0 14G 0 disk
└─sdb1 8:17 0 14G 0 part /mnt
sdc 8:32 0 100G 0 disk
└─sdc1 8:33 0 100G 0 part
└─vg--NW1--NFS-NW1 254:0 0 100G 0 lvm
sdd 8:48 0 100G 0 disk
└─sdd1 8:49 0 100G 0 part
└─vg--NW2--NFS-NW2 254:1 0 100G 0 lvm
Observe the distinct absence of the smaller SBD disks (typically sized between 50MB and 100MB) in both command outputs. This indicates a fundamental issue with disk discovery or iSCSI session establishment.
Resolution:
To successfully resolve iSCSI device availability issues, you must ensure proper iSCSI initiator-target communication and effective disk discovery on all cluster nodes. This involves a sequence of commands to establish and persist the iSCSI connections.
-
Verify Initiator Name:
First, confirm that the iSCSI initiator name is correctly configured on each cluster node. This unique name is used by the iSCSI target to identify and authenticate the connecting client.sudo cat /etc/iscsi/initiatorname.iscsi
Node 1 Example Output:
InitiatorName=iqn.2006-04.nfs-0.local:nfs-0
Node 2 Example Output:
sudo cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2006-04.nfs-1.local:nfs-1
It is imperative that each node possesses a unique and accurately formatted IQN to prevent conflicts and ensure proper iSCSI identification. -
List Configured SBD Devices:
Double-check the specific paths to the SBD devices as they are defined within your SBD configuration file. These defined paths must precisely match the actual device identifiers once those devices are successfully discovered by the operating system.sudo grep SBD_DEVICE /etc/sysconfig/sbd
Example Output:
# SBD_DEVICE specifies the devices to use for exchanging sbd messages SBD_DEVICE="/dev/disk/by-id/scsi-360014056eadbecfeca042d4a66b9d779;/dev/disk/by-id/scsi-36001405cbac988092e448059d25d1a4a;/dev/disk/by-id/scsi-36001405a29a443e4a6e4ceeae822e5eb"
Verify that thesescsi-xxxxxxxxIDs correspond exactly to your anticipated SBD disks. Any mismatch will prevent SBD from accessing them. -
Test SBD Device Accessibility:
Attempt to list messages on one of the SBD devices explicitly mentioned in your SBD configuration file. This direct test reveals if the SBD utility can successfully interact with the underlying block device.sudo /usr/sbin/sbd -d /dev/disk/by-id/scsi-360014056eadbecfeca042d4a66b9d779 list
Expected Error Output:
== disk /dev/disk/by-id/scsi-360014056eadbecfeca042d4a66b9d779 unreadable! sbd failed; please check the logs.
This error confirms that the SBD service is unable to read from the specified device, indicating a fundamental issue with disk accessibility or iSCSI connectivity. -
Connect to iSCSI Devices:
To resolve the disk accessibility issue, you must manually discover and then log into the iSCSI target. This critical process establishes the necessary network connection between your cluster node (as the iSCSI initiator) and the SBD storage (as the iSCSI target). Execute these commands meticulously on all cluster nodes.Discover iSCSI Targets:
sudo iscsiadm -m discovery
Output showing available targets (example):
10.0.0.17:3260 via sendtargets 10.0.0.18:3260 via sendtargets 10.0.0.19:3260 via sendtargetsDiscover Targets by Type and Portal:
sudo iscsiadm -m discovery --type=st --portal=10.0.0.17:3260
Output showing specific IQN for the target:
10.0.0.17:3260,1 iqn.2006-04.dbnw1.local:dbnw1 10.0.0.17:3260,1 iqn.2006-04.ascsnw1.local:ascsnw1 10.0.0.17:3260,1 iqn.2006-04.nfs.local:nfs
Carefully identify the target name (e.g.,iqn.2006-04.nfs.local:nfs) that precisely corresponds to your SBD devices.Log into the iSCSI Target:
sudo iscsiadm -m node -T iqn.2006-04.nfs.local:nfs --login --portal=10.0.0.17:3260
Expected Successful Login Output:
Logging in to [iface: default, target: iqn.2006-04.nfs.local:nfs, portal: 10.0.0.17,3260] Login to [iface: default, target: iqn.2006-04.nfs.local:nfs, portal: 10.0.0.17,3260] successful.Set Node Startup to Automatic (Persistent Login):
sudo iscsiadm -m node -p 10.0.0.17:3260 -T iqn.2006-04.nfs.local:nfs --op=update --name=node.startup --value=automatic
This command is crucial for ensuring that the iSCSI login session persists across system reboots, which is fundamental for maintaining continuous cluster stability and SBD access.Verify Disk Detection:
After successfully logging into the iSCSI target, utilize thelsscsicommand to confirm that the SBD devices are now correctly visible and registered by the operating system.sudo lsscsi
Expected Output (with newly detected SBD disk /dev/sde):
[0:0:0:0] disk Msft Virtual Disk 1.0 /dev/sda [0:0:0:1] disk Msft Virtual Disk 1.0 /dev/sdb [1:0:0:0] disk Msft Virtual Disk 1.0 /dev/sdc [1:0:0:1] disk Msft Virtual Disk 1.0 /dev/sdd [2:0:0:0] disk LIO-ORG sbdnfs 4.0 /dev/sde
Remember to repeat the entire set ofiscsiadmcommands for all iSCSI portals and targets that collectively provide your SBD devices. Ensure this entire sequence of operations is executed identically on all nodes within the cluster to establish comprehensive and uniform connectivity. -
Confirm All SBD Devices are Detected:
After meticulously performing the iSCSI login steps for all relevant targets on all cluster nodes, re-confirm that all your SBD devices are now consistently listed by bothlsscsiandlsblk. This final check ensures complete visibility and readiness for the SBD service.sudo lsscsi
Final Expected Output (All SBD Disks Visible):
[0:0:0:0] disk Msft Virtual Disk 1.0 /dev/sda [0:0:0:1] disk Msft Virtual Disk 1.0 /dev/sdb [1:0:0:0] disk Msft Virtual Disk 1.0 /dev/sdc [1:0:0:1] disk Msft Virtual Disk 1.0 /dev/sdd [2:0:0:0] disk LIO-ORG sbdnfs 4.0 /dev/sde [3:0:0:0] disk LIO-ORG sbdnfs 4.0 /dev/sdf [4:0:0:0] disk LIO-ORG sbdnfs 4.0 /dev/sdgsudo lsblk
Final Expected Output (All SBD Disks Visible):
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 100G 0 disk ├─sda1 8:1 0 2M 0 part ├─sda2 8:2 0 512M 0 part /boot/efi ├─sda3 8:3 0 1G 0 part /boot └─sda4 8:4 0 98.5G 0 part / sdb 8:16 0 14G 0 disk └─sdb1 8:17 0 14G 0 part /mnt sdc 8:32 0 100G 0 disk └─sdc1 8:33 0 100G 0 part └─vg--NW1--NFS-NW1 254:0 0 100G 0 lvm sdd 8:48 0 100G 0 disk └─sdd1 8:49 0 100G 0 part └─vg--NW2--NFS-NW2 254:1 0 100G 0 lvm sde 8:64 0 50M 0 disk sdf 8:80 0 50M 0 disk sdg 8:96 0 50M 0 disk
Once all SBD devices are consistently and correctly visible across all cluster nodes, the SBD service should be able to initiate successfully, restoring fencing capabilities to your Pacemaker cluster.
Cause 4: Node Doesn’t Rejoin Cluster After Fencing¶
Occasionally, a node that has undergone a fencing process fails to successfully rejoin the cluster. This scenario frequently arises when the SBD slot designated for that specific node is not in a “clean” state, meaning it still retains a lingering fencing message. In such cases, the SBD service on the affected node might display a failed status, while other nodes in the cluster could be in a “pending” state, awaiting the problematic node’s reintegration.
Resolution 1: Check SBD Slot Status
Verify the status of each node’s SBD slot across your SBD devices. A “reset” status specifically indicates that a fencing message is currently present for that node, which actively prevents it from rejoining the cluster until addressed.
Begin by identifying your SBD devices through lsscsi and listing the persistent /dev/disk/by-id/ paths:
sudo lsscsi
Example Output (identifying SBD devices):
[0:0:0:0] disk Msft Virtual Disk 1.0 /dev/sda
[0:0:0:1] disk Msft Virtual Disk 1.0 /dev/sdb
[1:0:0:0] disk Msft Virtual Disk 1.0 /dev/sdc
[1:0:0:1] disk Msft Virtual Disk 1.0 /dev/sdd
[2:0:0:0] disk LIO-ORG sbdnfs 4.0 /dev/sde
[3:0:0:0] disk LIO-ORG sbdnfs 4.0 /dev/sdf
[4:0:0:0] disk LIO-ORG sbdnfs 4.0 /dev/sdg
And to confirm persistent device names:
sudo ls -l /dev/disk/by-id/scsi-*
Now, meticulously check the SBD slot status using one of the SBD devices, for example,
/dev/sde:
sudo sbd -d /dev/sde list
Example Output (showing a “reset” state for a node):
0 nfs-0 clear
1 nfs-1 reset nfs-0
In this particular example,
nfs-1 is clearly in a reset state. This indicates that it was fenced by nfs-0 and is currently unable to rejoin the cluster until this specific state is explicitly cleared.
Resolution 2: Address SBD_STARTMODE and Clear Node Slot
The SBD_STARTMODE parameter, configured in /etc/sysconfig/sbd, governs how a node behaves when attempting to rejoin the cluster following a fencing event. If this parameter is set to clean, the node will only be allowed to rejoin after its SBD slot has been manually cleared. Conversely, if set to always, the node will attempt to rejoin irrespective of its previous fencing state.
First, inspect the current SBD_STARTMODE setting:
sudo grep -i SBD_STARTMODE /etc/sysconfig/sbd
Example Output:
SBD_STARTMODE=clean
If
SBD_STARTMODE is configured as clean, the observed behavior is entirely expected. SBD has detected a fencing message within the node’s dedicated slot and is intentionally preventing it from joining the cluster until this issue is manually resolved.
To clear the node’s SBD slot and facilitate its rejoining of the cluster, follow these precise steps:
-
Clear the Fencing Message:
Clear the specific fencing message for the affected node. This action can be performed either from the fenced node itself (by using theLOCALkeyword) or from any other active node within the cluster by explicitly specifying the node’s name.From the fenced node:
sudo sbd -d <SBD_DEVICE> message LOCAL clear
Replace<SBD_DEVICE>with the actual path to your SBD device, e.g.,/dev/sde.From any cluster node (specifying the target node name):
sudo sbd -d <SBD_DEVICE> message <NODENAME> clear
For instance, to clear the SBD slot fornfs-1using/dev/sde:
sudo sbd -d /dev/sde message nfs-1 clear
This command writes a “clear” message to the specified node’s SBD slot, signaling that it is now safe for the node to attempt to rejoin the cluster. -
Restart Clustering Services and Re-establish iSCSI:
After successfully clearing the SBD slot, proceed to attempt to start the clustering service. If the service still fails to initialize, it is possible that the underlying iSCSI sessions have been disrupted. To rectify this, re-establish the iSCSI connections by first logging out and then logging back into all iSCSI targets.sudo iscsiadm -m node -u
This command gracefully logs out of all currently active iSCSI sessions associated with the node.sudo iscsiadm -m node -l
Subsequently, this command attempts to log back into all automatically configured iSCSI sessions, refreshing the connections. This two-step process of logging out and then back in often resolves latent disk visibility issues that prevent the SBD service from starting correctly.
Cause 5: SBD Service Doesn’t Start After You Add New SBD Device¶
When you introduce a new SBD device to an existing cluster configuration or make modifications to the paths of existing SBD devices, you might encounter a generic sbd failed; please check the logs error message. This error suggests that the SBD daemon is unable to properly interact with the newly added or modified block devices.
If you attempt to send a test message to one of the SBD devices, you may observe the following error output:
sudo sbd -d /dev/disk/by-id/scsi-360014056eadbecfeca042d4a66b9d779 message node1 test
Expected Error Output:
sbd failed; please check the logs.
Further inspection of the system logs, specifically
/var/log/messages, will often reveal more detailed diagnostic information, such as:
/var/log/messages
Mar 2 06:58:06 node1 sbd[11105]: /dev/disk/by-id/scsi-360014056eadbecfeca042d4a66b9d779: error: slot_msg: slot_msg(): No recipient / cmd specified.
Mar 2 06:58:06 node1 sbd[11104]: warning: messenger: Process 11105 failed to deliver!
This specific error message, particularly “No recipient / cmd specified,” strongly suggests that SBD is struggling to properly manage messages on the device. This usually occurs because the SBD daemon has not fully initialized with the updated configuration under the control of the cluster.
Resolution:
It is critical to remember that the SBD service, as an integral fencing mechanism for Pacemaker, must have its operation and device management orchestrated solely by the cluster. Therefore, manual starts or stops of the sbd.service are generally discouraged and can lead to inconsistent states. When new SBD devices are added or existing configurations are altered, the Pacemaker cluster requires a complete restart to effectively register these changes and bring the SBD service under its proper and authoritative control.
-
Restart Cluster Services on All Nodes:
Perform a graceful stop followed by a start of the entire Pacemaker cluster services on all cluster nodes. This comprehensive restart ensures that Pacemaker re-reads its entire configuration, including all SBD device definitions, and correctly initializes all related services in a synchronized manner.sudo crm cluster stop sudo crm cluster start
Execute these commands sequentially on each node. Ensure that services on one node have fully stopped before proceeding to the next, and then similarly, ensure they are fully started across the cluster. This orchestrated process facilitates a clean and synchronized restart of the entire high-availability stack. -
Verify SBD Service Status:
After the cluster services have been systematically restarted on all nodes, verify the SBD service status to confirm that it is now running correctly and without errors.sudo systemctl status sbd.service
Expected Active Output:
● sbd.service - Shared-storage based fencing daemon Loaded: loaded (/usr/lib/systemd/system/sbd.service; enabled; vendor preset: disabled) Active: active (running)
Anactive (running)status signifies successful initialization of the SBD daemon and its readiness to manage fencing messages. -
Confirm SBD Device List Output:
Finally, confirm that SBD can properly list the status of all nodes on the SBD devices, reflecting the desired “clear” states for each node. This validates that SBD is now correctly communicating with the devices.sudo sbd -d /dev/disk/by-id/scsi-360014056eadbecfeca042d4a66b9d779 list
Expected Clear Output:
0 node1 clear 1 node2 clear
This output verifies that SBD is effectively communicating with the shared devices and correctly managing the fencing states of your cluster nodes, indicating a successful resolution.
Cause 6: SBD Doesn’t Start After SLES 12 to SLES 15 Upgrade¶
Upgrading a SUSE Linux Enterprise Server (SLES) from version 12 to 15 can introduce significant changes that affect iSCSI client information and related software packages. Specifically, SLES 15 deprecates the targetcli-fb package, replacing it with newer Python 3 or Python 2 based versions, such as python3-targetcli-fb or python2-targetcli-fb. This package transition frequently results in system service configurations, including those critical for iSCSI operations, being inadvertently reset to their default values (disabled or stopped). Consequently, established connections to remote iSCSI-based SBD devices fail, directly preventing the SBD service from starting post-upgrade.
Resolution:
The solution to this post-upgrade issue involves manually re-enabling and starting the targetcli service. This service is fundamentally responsible for managing iSCSI targets and initiators, ensuring that your iSCSI connections are re-established and fully functional after the operating system upgrade.
mermaid
graph TD
A[SLES 12 to SLES 15 Upgrade] --> B{iSCSI Client Info Missing?};
B -- Yes --> C[targetcli-fb replaced with python3-targetcli-fb/python2-targetcli-fb];
C --> D[System services reset to disabled/stopped];
D --> E[SBD Service Failure];
E --> F[Manually Enable & Start targetcli Service];
F --> G[iSCSI Connections Re-established];
G --> H[SBD Service Starts Successfully];
Figure: Flowchart illustrating the SBD failure resolution after a SLES upgrade.
For more comprehensive guidance and to access the official SUSE support article pertaining to this specific scenario, please refer to the SUSE Knowledge Base article titled SBD Failure After SLES15 Upgrade. This resource provides detailed commands and contextual information vital for resolving this upgrade-related challenge.
Next Steps and Further Assistance¶
Upon diligently attempting the resolutions outlined in this article, your SUSE Pacemaker cluster’s SBD service should ideally be fully operational. However, should you continue to require further assistance, encounter persistent issues, or wish to seek expert support, please follow the instructions below to open a formal support request. Providing comprehensive diagnostic logs is paramount for accelerating the troubleshooting process.
When you submit your support request, it is highly recommended that you attach the following essential diagnostic logs:
supportconfig: This is a powerful utility provided by SUSE, designed to collect extensive system configuration and diagnostic information from your SLES environment.hb_report: A specialized tool specific to Pacemaker that gathers detailed and comprehensive cluster-related logs, crucial for analyzing cluster behavior.
These detailed logs furnish invaluable insights into your system and cluster’s state, significantly expediting the troubleshooting and resolution process by support engineers.
For any additional questions, clarifications, or if you require direct help, you are encouraged to create a support request through the Azure portal’s help and support blade. Alternatively, you can engage with the broader Azure community support forums, where experienced users and experts may offer insights. Your feedback is also immensely valuable; please consider submitting product feedback to the Azure feedback community to help improve services and documentation.
Post a Comment