Troubleshooting Node Removal from Windows Server Failover Clusters

Table of Contents

This document addresses issues where nodes are unexpectedly removed from an active Windows Server Failover Cluster membership. This problem can lead to disruptions in service availability and requires careful investigation to identify and resolve the underlying cause. Understanding the mechanisms behind cluster node communication and failure detection is crucial for effective troubleshooting. This article will guide you through the symptoms, potential causes, and recommended steps to diagnose and rectify these node removal incidents.

Symptoms¶

When a node is removed from a failover cluster, a specific event is typically logged in the system event log. This event serves as a primary indicator of the issue and provides valuable context for further investigation. You will observe events similar to the following example being logged within the system event logs of all remaining cluster nodes, excluding the node that was removed from the cluster membership.

This event, specifically Event ID 1135, indicates that a node within the cluster has determined that another node is no longer responsive or reachable. The node detecting the failure then disseminates this information to all other active nodes in the cluster. Upon receiving this notification, each node will terminate its heartbeat connections with the node that has been marked as down. This disconnection is a critical step in maintaining cluster stability and preventing potential data corruption or inconsistencies that could arise from communication issues.

What Causes a Node to Be Marked Down¶

Windows Server Failover Clusters rely on a robust communication mechanism between all participating nodes. This communication occurs over designated networks configured to permit cluster network traffic. Nodes within the cluster continuously exchange “heartbeat” packets across these networks. These heartbeats are essential for each node to monitor the health and availability of other nodes in the cluster.

Consider a simple two-node cluster scenario, consisting of nodes NodeA and NodeB. NodeA will periodically send heartbeat packets to NodeB, and similarly, NodeB will send heartbeats to NodeA. Upon receiving a heartbeat packet, the receiving node is expected to send a response back to the originating node. This two-way communication confirms that both the network path and the remote node are operational.

If a node fails to receive a response to its heartbeat packets within a defined timeframe, it is considered a missed heartbeat. Each node maintains a count of missed heartbeats for every other node in the cluster. By default, a cluster node tolerates a limited number of consecutive heartbeat failures before declaring a network route or a node as down. This default threshold is set to five consecutive failures within a five-second interval.

For instance, if NodeA sends heartbeat requests to NodeB and does not receive responses for five consecutive attempts within five seconds, NodeA will deem the communication path to NodeB as down. If other communication paths to NodeB are still functioning, NodeB may remain an active cluster member. However, if all communication routes to NodeB are considered down by NodeA and potentially other nodes, NodeB will be removed from the active failover cluster membership. This removal is accompanied by the logging of Event ID 1135, as described earlier. On the affected node, NodeB in this example, the Cluster Service will typically terminate and then automatically restart in an attempt to rejoin the cluster.

It’s important to note that the specifics of how the cluster handles network partitioning in scenarios with three or more nodes can be complex. For a deeper dive into this topic, you can refer to resources that specifically discuss “Partitioned Cluster Networks.”

Known Causes for Heartbeat Process Failures¶

Now that we understand the heartbeat mechanism and how node failures are detected, let’s explore common reasons why this process might fail, leading to unexpected node removals.

1. Actual Network Hardware Failures: Physical network infrastructure issues are a primary suspect in heartbeat failures. If network packets are lost due to faulty cables, malfunctioning network interface cards (NICs), or problems with switches or routers along the communication path, heartbeats will fail. To diagnose this, capturing network traces from both the node initiating the heartbeat and the node expected to respond is highly recommended. Analyzing these traces can reveal packet loss or network congestion issues occurring between the nodes.

2. Network Profile Flapping: In Windows environments, network connections are assigned profiles, such as “Domain,” “Private,” or “Public.” If a network connection’s profile unexpectedly and repeatedly switches between these categories (a phenomenon known as “profile flapping”), network input/output (I/O) operations can be temporarily blocked. This blockage can disrupt heartbeat communication. To investigate this, examine the Network Profile Operational log in the Event Viewer. Navigate to Applications and Services Logs\Microsoft\Windows\NetworkProfile\Operational and review the events logged around the time of the Event ID 1135 occurrence. Look for events indicating network profile changes on the node mentioned in the event log. If profile flapping is detected, further investigation into the cause of these profile changes is necessary. Microsoft provides resources and hotfixes to address known issues related to network location profile changes.

3. Disabled IPv6 Firewall Rules: Failover Clustering in modern Windows Server environments prioritizes IPv6 for heartbeat communication. If IPv6 is enabled on the cluster servers, but critical inbound and outbound firewall rules related to IPv6 Neighbor Discovery are disabled in the Windows Firewall, heartbeat communication can be disrupted. Specifically, ensure that the following rules are enabled for both inbound and outbound traffic in the Windows Firewall with Advanced Security:

   • Core Networking - Neighbor Discovery Advertisement
   • Core Networking - Neighbor Discovery Solicitation
     Disabling these rules can prevent nodes from discovering and communicating with each other over IPv6, leading to heartbeat failures.

4. Antivirus Software Interference: Aggressive or improperly configured antivirus software can sometimes interfere with network communication, including the UDP traffic used for cluster heartbeats. Antivirus software might mistakenly identify heartbeat packets as suspicious or malicious and block them. If you suspect antivirus interference, a temporary test can be performed by disabling or uninstalling the antivirus software on the cluster nodes. Caution: Disabling antivirus software significantly reduces system security and should only be done in a controlled test environment and for a minimal duration. If disabling antivirus resolves the issue, reconfigure the antivirus software to exclude cluster communication traffic or consult the antivirus vendor for recommended configurations in a clustered environment.

5. Network Latency: Even if network packets are not entirely lost, excessive latency or delays in network communication can cause heartbeat failures. If the round-trip time for heartbeat packets exceeds the default timeout thresholds, nodes might be falsely marked as down. Network latency can be caused by network congestion, overloaded network devices, or long physical distances between cluster nodes in multi-site deployments. Tools like ping and pathping can be used to measure network latency between cluster nodes. If high latency is observed, investigate potential network bottlenecks or consider adjusting cluster heartbeat timeout settings (with caution, as discussed later).

6. Firewall or Router Misconfigurations Blocking UDP Port 3343: As mentioned, Failover Clustering defaults to IPv6 and utilizes UDP port 3343 for heartbeat communication. Firewalls, routers, or switches situated between cluster nodes must be correctly configured to permit UDP traffic on port 3343. If these network devices are blocking or filtering this traffic, heartbeats will fail. Verify the configurations of all intervening network devices to ensure that UDP port 3343 is open and allows communication between cluster nodes.

7. IPsec Security Policy Refreshes: IPsec (Internet Protocol Security) policies can enhance network security, but their refresh mechanisms can sometimes disrupt cluster communication. During an IPsec Group Policy update, Windows Firewall with Advanced Security (WFAS) may tear down existing IPsec Security Associations (SAs). While these SAs are being renegotiated, network connectivity can be temporarily blocked. If authentication with Active Directory during SA renegotiation experiences delays, this network blockage can extend long enough to interrupt cluster heartbeats. If heartbeats are missed during this period, nodes might be incorrectly marked as down due to the default 5-second threshold. Investigate IPsec policy refresh intervals and authentication processes if this is suspected as a cause.

8. Outdated Network Card Drivers and Firmware: Using outdated or incompatible network card drivers or firmware can lead to various network issues, including packet loss and communication instability, which can manifest as heartbeat failures. Similarly, misconfigurations of the network card itself or the connected network switch can also contribute to heartbeat loss. Ensure that network cards are using the latest drivers and firmware recommended by the manufacturer. Verify network card and switch configurations for any errors or inconsistencies.

9. Packet Loss on Modern Network Cards and Virtual Network Cards: Modern network cards, particularly in virtualized environments, might experience packet loss due to various factors, such as insufficient receive buffers or server performance limitations. To monitor packet loss, use Performance Monitor and add the counter “Network Interface\Packets Received Discarded.” This counter accumulates dropped packets since the last server reboot. A consistently high value for this counter suggests potential packet loss issues. Low receive buffer settings on the network card or insufficient server resources to handle inbound network traffic can cause packet drops. Network card manufacturers may provide settings to adjust receive buffer sizes in the network adapter’s properties. Consult the manufacturer’s documentation for recommended values and configuration instructions. In VMware environments, specific VMware Knowledge Base articles provide guidance on diagnosing and addressing packet loss in virtual network adapters, including recommended settings adjustments.

These are some of the most frequently encountered causes of Event ID 1135 and node removals from failover clusters. However, other less common factors might also be at play. The primary goal of this troubleshooting guide is to provide insight into the heartbeat process and offer a structured approach to investigate and identify the root cause of node removal issues.

Understanding Heartbeat Parameters¶

To address persistent node removal issues, some administrators might consider adjusting the cluster’s heartbeat timeout parameters. These parameters control the sensitivity of the cluster’s failure detection mechanism. The key parameters and their default and adjustable ranges are summarized below:

Parameter	Default (milliseconds)	Range (milliseconds)
SameSubnetDelay	1000	250-2000
CrossSubnetDelay	1000	250-4000
SameSubnetThreshold	5	3-10
CrossSubnetThreshold	5	3-10

Important Note: While increasing these timeout values might temporarily mask the symptoms of node removal and reduce the frequency of Event ID 1135, it is crucial to understand that this approach does not address the underlying problem. It merely makes the cluster less sensitive to network issues. Increasing these values to their maximum limits should be considered a temporary workaround, not a permanent solution.

The best practice is always to thoroughly investigate and identify the root cause of the heartbeat failures and implement a proper fix. Increasing timeout values should only be considered in specific scenarios, such as multi-site clusters where nodes are geographically dispersed and unavoidable network latency exists. In such cases, carefully increasing timeout values might be necessary to accommodate the inherent network characteristics of the environment, but even then, thorough network optimization and monitoring are still essential.

Ignoring the root cause and solely relying on increased timeout values can lead to delayed failure detection. A cluster that is slow to detect failures might experience prolonged periods of degraded performance or increased risk of data inconsistencies in more severe failure scenarios. Therefore, prioritize identifying and resolving the actual network or system issues that are causing heartbeat failures rather than simply masking the symptoms with increased timeout values.

If you are facing persistent node removal issues in your Windows Server Failover Cluster, systematically investigate the potential causes outlined in this article. Analyze network configurations, hardware health, software configurations, and system performance to pinpoint the root of the problem. Addressing the root cause will ensure a stable and reliable cluster environment and prevent future unexpected node removals.

If you have further questions or insights regarding troubleshooting node removal from Windows Server Failover Clusters, please feel free to share your comments and experiences below! Your contributions can help others in the community facing similar challenges.