Optimize Windows Server Failover Cluster: Mastering Network Threshold Tuning

Table of Contents

This article provides solutions for adjusting the network threshold settings within a Windows Server Failover Cluster. Modifying these thresholds, specifically by increasing their values, can significantly reduce a cluster’s susceptibility to temporary network disruptions and latency. By implementing these adjustments, administrators can enhance the stability and resilience of their failover clusters, particularly in environments where network variability is a factor. Understanding and strategically tuning these network thresholds is crucial for maintaining optimal cluster performance and minimizing unnecessary failovers.

Symptom¶

In environments utilizing Windows Failover Cluster nodes, it is generally advisable to transition the cluster’s monitoring configuration to a more relaxed state than the default settings. The out-of-the-box cluster settings are often designed to be quite stringent, which can inadvertently lead to unwarranted service outages, especially in modern, dynamic network environments. These default configurations are primarily optimized for highly controlled, on-premises network infrastructures. They often fail to adequately account for the potential for induced latency that is inherent in multitenant environments, such as those found in cloud platforms like Microsoft Azure Infrastructure-as-a-Service (IaaS).

Windows Server Failover Clustering employs continuous monitoring of network connectivity and the overall health of each node within the cluster. This constant vigilance is essential for ensuring high availability and fault tolerance. Should a node become unreachable over the network, the cluster initiates a recovery process. This process is designed to restore service availability by transferring applications and services to another operational node within the cluster. However, even minor latency in communication between cluster nodes can trigger false positives in this monitoring system, leading to issues. A common manifestation of such network latency issues is the generation of error 1135 in the system event log.

Error 1135 (system event log)

This error specifically indicates that a cluster node, for example, “Node 1,” has been removed from active failover cluster membership. The underlying cause, as indicated by the error message, could be the unexpected cessation of the Cluster service on the affected node. More frequently, however, this error points to a loss of communication between the node in question and the other active nodes participating in the failover cluster. When this error occurs, it is recommended to initially run the “Validate a Configuration” wizard. This diagnostic tool is designed to thoroughly examine the network configuration of the cluster and identify potential misconfigurations or issues. If the error persists despite network validation, further investigation into hardware or software problems related to the network adapters on the problematic node is warranted. Additionally, it is crucial to examine other network components that facilitate communication with the node, such as network hubs, switches, or bridges, for potential failures.

The following is an example extracted from a Cluster.log file, illustrating the sequence of events leading to a node being marked as unreachable and removed from the cluster due to network communication issues:

0000ab34.00004e64::2014/06/10-07:54:34.099 DBG   [NETFTAPI] Signaled NetftRemoteUnreachable event, local address 10.xx.x.xxx:3343 remote address 10.x.xx.xx:3343
0000ab34.00004b38::2014/06/10-07:54:34.099 INFO  [IM] got event: Remote endpoint 10.xx.xx.xxx:~3343~ unreachable from 10.xx.x.xx:~3343~
0000ab34.00004b38::2014/06/10-07:54:34.099 INFO  [IM] Marking Route from 10.xxx.xxx.xxxx:~3343~ to 10.xxx.xx.xxxx:~3343~ as down
0000ab34.00004b38::2014/06/10-07:54:34.099 INFO  [NDP] Checking to see if all routes for route (virtual) local fexx::xxx:5dxx:xxxx:3xxx:~0~ to remote xxx::cxxx:xxxd:xxx:dxxx:~0~ are down
0000ab34.00004b38::2014/06/10-07:54:34.099 INFO  [NDP] All routes for route (virtual) local fxxx::xxxx:5xxx:xxxx:3xxx:~0~ to remote fexx::xxxx:xxxx:xxxx:xxxx:~0~ are down
0000ab34.00007328::2014/06/10-07:54:34.099 INFO  [CORE] Node 8: executing node 12 failed handlers on a dedicated thread
0000ab34.00007328::2014/06/10-07:54:34.099 INFO  [NODE] Node 8: Cleaning up connections for n12.
0000ab34.00007328::2014/06/10-07:54:34.099 INFO  [Nodename] Clearing 0 unsent and 15 unacknowledged messages.
0000ab34.00007328::2014/06/10-07:54:34.099 INFO  [NODE] Node 8: n12 node object is closing its connections
0000ab34.00008b68::2014/06/10-07:54:34.099 INFO  [DCM] HandleNetftRemoteRouteChange
0000ab34.00004b38::2014/06/10-07:54:34.099 INFO  [IM] Route history 1: Old: 05.936, Message: Response, Route sequence: 150415, Received sequence: 150415, Heartbeats counter/threshold: 5/5, Error: Success, NtStatus: 0 Timestamp: 2014/06/10-07:54:28.000, Ticks since last sending: 4
0000ab34.00007328::2014/06/10-07:54:34.099 INFO  [NODE] Node 8: closing n12 node object channels
0000ab34.00004b38::2014/06/10-07:54:34.099 INFO  [IM] Route history 2: Old: 06.434, Message: Request, Route sequence: 150414, Received sequence: 150402, Heartbeats counter/threshold: 5/5, Error: Success, NtStatus: 0 Timestamp: 2014/06/10-07:54:27.665, Ticks since last sending: 36
0000ab34.0000a8ac::2014/06/10-07:54:34.099 INFO  [DCM] HandleRequest: dcm/netftRouteChange
0000ab34.00004b38::2014/06/10-07:54:34.099 INFO  [IM] Route history 3: Old: 06.934, Message: Response, Route sequence: 150414, Received sequence: 150414, Heartbeats counter/threshold: 5/5, Error: Success, NtStatus: 0 Timestamp: 2014/06/10-07:54:27.165, Ticks since last sending: 4
0000ab34.00004b38::2014/06/10-07:54:34.099 INFO  [IM] Route history 4: Old: 07.434, Message: Request, Route sequence: 150413, Received sequence: 150401, Heartbeats counter/threshold: 5/5, Error: Success, NtStatus: 0 Timestamp: 2014/06/10-07:54:26.664, Ticks since last sending: 36

This log excerpt demonstrates the cluster’s internal communication monitoring. The [NETFTAPI] Signaled NetftRemoteUnreachable event and subsequent [IM] got event: Remote endpoint unreachable messages clearly indicate the detection of a communication breakdown between cluster nodes. The logs further detail the process of marking routes as down and initiating recovery actions, ultimately leading to the removal of the node from the active cluster membership. The Route history entries provide a granular view of heartbeat exchanges, showing timestamps, sequence numbers, and error status for recent communication attempts. These log details are invaluable for diagnosing network-related issues within the failover cluster.

0000ab34.00007328::2014/06/10-07:54:34.100 INFO    <realLocal>10.xxx.xx.xxx:~3343~</realLocal>
0000ab34.00007328::2014/06/10-07:54:34.100 INFO    <realRemote>10.xxx.xx.xxx:~3343~</realRemote>
0000ab34.00007328::2014/06/10-07:54:34.100 INFO    <virtualLocal>fexx::xxxx:xxxx:xxxx:xxxx:~0~</virtualLocal>
0000ab34.00007328::2014/06/10-07:54:34.100 INFO    <virtualRemote>fexx::xxxx:xxxx:xxxx:xxxx:xxxx:~0~</virtualRemote>
0000ab34.00007328::2014/06/10-07:54:34.100 INFO    <Delay>1000</Delay>
0000ab34.00007328::2014/06/10-07:54:34.100 INFO    <Threshold>5</Threshold>
0000ab34.00007328::2014/06/10-07:54:34.100 INFO    <Priority>140481</Priority>
0000ab34.00007328::2014/06/10-07:54:34.100 INFO    <Attributes>2147483649</Attributes>
0000ab34.00007328::2014/06/10-07:54:34.100 INFO  </struct mscs::FaultTolerantRoute>
0000ab34.00007328::2014/06/10-07:54:34.100 INFO   removed

This subsequent log snippet provides further context, specifically highlighting the network configuration parameters in play during the communication failure. The <Delay>1000</Delay> tag indicates a heartbeat delay of 1000 milliseconds (1 second), while <Threshold>5</Threshold> signifies that a node is considered unreachable after 5 consecutive missed heartbeats. These settings, while suitable for some environments, can be overly sensitive in networks with even minor transient latency.

0000ab34.0000a7c0::2014/06/10-07:54:38.433 ERR   [QUORUM] Node 8: Lost quorum (3 4 5 6 7 8)
0000ab34.0000a7c0::2014/06/10-07:54:38.433 ERR   [QUORUM] Node 8: goingAway: 0, core.IsServiceShutdown: 0
0000ab34.0000a7c0::2014/06/10-07:54:38.433 ERR   lost quorum (status = 5925)

Finally, this log segment reveals the ultimate consequence of the repeated communication failures: the cluster loses quorum. The [QUORUM] Node 8: Lost quorum (3 4 5 6 7 8) message indicates that Node 8, along with other nodes, has lost its quorum, which is essential for the cluster to function correctly. Quorum loss signifies a critical state where the cluster can no longer reliably maintain its operational integrity and may lead to service disruptions across the entire clustered environment. This further emphasizes the importance of addressing the underlying network communication issues and adjusting network thresholds appropriately.

Cause¶

The behavior of Windows Server Failover Clusters in detecting and reacting to network connectivity issues is governed by two primary configuration settings. These settings, Delay and Threshold, dictate the sensitivity of the cluster’s network monitoring and are crucial in determining how the cluster responds to potential network disruptions. Understanding these settings is paramount for optimizing cluster performance and stability, especially in environments susceptible to network latency or transient issues.

Delay - This parameter defines the interval at which cluster heartbeat messages are transmitted between nodes within the cluster. The delay value represents the duration, measured in seconds or milliseconds, that the cluster waits before sending the subsequent heartbeat. It’s important to note that the heartbeat delay can be configured differently based on the network topology. Specifically, a cluster can have distinct delay settings for communication between nodes residing on the same subnet (SameSubnetDelay) and for communication between nodes located on different subnets (CrossSubnetDelay). This differentiation allows for finer control over network monitoring based on expected network characteristics within and across subnets. Shorter delay values result in more frequent heartbeat transmissions, increasing the responsiveness of the cluster to network changes but also potentially increasing network traffic. Conversely, longer delay values reduce network traffic but may make the cluster slower to react to network issues.

Threshold - This setting determines the number of consecutive heartbeat messages that can be missed before the cluster initiates a recovery action. The threshold is expressed as a count of missed heartbeats. Similar to the Delay setting, the Threshold can also be configured differently for nodes on the same subnet (SameSubnetThreshold) and nodes on different subnets (CrossSubnetThreshold). This allows administrators to tailor the cluster’s fault tolerance based on the anticipated reliability of network links within and across subnets. A lower threshold value makes the cluster more sensitive to network disruptions, triggering recovery actions more quickly upon detecting missed heartbeats. A higher threshold value makes the cluster more tolerant of transient network issues, delaying recovery actions until a larger number of heartbeats are missed. Choosing the appropriate threshold is a balance between rapid fault detection and avoiding unnecessary failovers due to temporary network glitches.

By default, Windows Server Failover Cluster configurations are pre-set with specific values for these network thresholds. The SameSubnetThreshold is typically set to 10, while the SameSubnetDelay is configured to 1000 milliseconds (or 1 second). These default values imply that if network connectivity monitoring fails to detect heartbeats for a duration equivalent to 10 times the SameSubnetDelay (in this case, 10 seconds), the failover threshold is considered to be reached. This threshold breach triggers a recovery process, leading to the node experiencing communication issues being removed from the active cluster membership. This removal subsequently initiates the relocation of resources and workloads from the affected node to another available and healthy node within the cluster. As a consequence of this process, cluster errors are typically logged, and prominently among them is cluster error 1135, as previously described. This default configuration, while robust in well-maintained, low-latency networks, can be overly aggressive in environments with even minor network variability, potentially leading to false positive failovers.

To illustrate the default behavior, consider a scenario where transient network latency causes a node to miss a few consecutive heartbeats, but the network connectivity is quickly restored. With the default SameSubnetThreshold of 10 and SameSubnetDelay of 1000ms, even a 10-second period of communication disruption, regardless of its transient nature, will trigger a node removal. This can be problematic in environments where short-lived network hiccups are not uncommon, as it can lead to unnecessary failovers and potential service interruptions. Therefore, understanding and adjusting these network thresholds is critical for tailoring the failover cluster’s behavior to the specific characteristics and reliability of the underlying network infrastructure.

Resolution¶

To effectively address the issues arising from overly sensitive default network configurations in Windows Server Failover Clusters, the recommended solution is to relax the cluster’s network configuration settings. This relaxation primarily involves increasing the values of the Delay and Threshold parameters, making the cluster less reactive to transient network hiccups and more tolerant of minor communication latency. By adjusting these settings, administrators can significantly reduce the likelihood of false positive failovers and enhance the overall stability of the cluster, especially in environments where network variability is a concern.

For detailed guidance and best practices on adjusting these crucial network parameters, refer to the Microsoft documentation on “Heartbeat and threshold”. This resource provides comprehensive information on understanding and configuring these settings to optimize failover cluster behavior in various network environments, including cloud deployments and environments with inherent network latency. It is essential to consult this documentation to gain a deeper understanding of the implications of modifying these settings and to determine the most appropriate values for your specific cluster environment.

When considering adjusting these settings, it is crucial to understand the trade-offs involved. Increasing the Delay and Threshold values will make the cluster less sensitive to transient network issues, reducing the chances of false positive failovers. However, it will also mean that the cluster will take longer to detect genuine node failures. Therefore, the optimal values for Delay and Threshold will depend on the specific characteristics of your network and the desired balance between fault tolerance and responsiveness. In environments with known network latency or potential for transient disruptions, increasing these values is generally recommended. Conversely, in highly stable and reliable networks, the default settings may be sufficient, or only minor adjustments may be necessary.

Practical Steps for Adjusting Network Thresholds:

1. Assess your Network Environment: Before making any changes, thoroughly evaluate the characteristics of your network. Consider factors such as network latency, potential for transient disruptions, and overall network reliability. Environments with higher latency or less predictable network behavior will generally benefit more from increased Delay and Threshold values.

2. Identify Current Settings: Use PowerShell cmdlets or the Failover Cluster Manager interface to determine the current values of SameSubnetDelay, SameSubnetThreshold, CrossSubnetDelay, and CrossSubnetThreshold for your cluster. This will provide a baseline for your adjustments. PowerShell cmdlets such as Get-ClusterNetwork can be used to retrieve these settings.

3. Plan Adjustments: Based on your network assessment, determine the desired degree of relaxation for the network thresholds. A common starting point for environments with moderate network variability is to double or triple the default values. For example, you might consider increasing SameSubnetThreshold from 10 to 20 or 30, and SameSubnetDelay from 1000ms to 2000ms or 3000ms. It is crucial to adjust these settings incrementally and monitor the cluster’s behavior after each adjustment.

4. Implement Changes: Use PowerShell cmdlets like Set-ClusterParameter to modify the Delay and Threshold values. For instance, to increase the SameSubnetThreshold to 20, you would use a command similar to: (Get-ClusterNetwork "Cluster Network 1").SetSameSubnetThreshold(20). Replace “Cluster Network 1” with the actual name of your cluster network. Similar commands can be used to adjust other threshold and delay parameters.

5. Monitor and Test: After implementing the changes, closely monitor the cluster’s performance and stability. Pay attention to system event logs and cluster logs for any errors or warnings. Conduct thorough testing, including simulated network disruptions, to ensure that the adjusted settings provide the desired level of fault tolerance without introducing new issues. Observe how the cluster responds to both transient network hiccups and simulated node failures to validate the effectiveness of the changes.

6. Iterate and Refine: Network tuning is often an iterative process. Based on your monitoring and testing results, you may need to further refine the Delay and Threshold values to achieve the optimal balance for your environment. Continue to monitor and test after each adjustment until you are satisfied with the cluster’s stability and responsiveness.

By carefully considering your network environment and strategically adjusting the Delay and Threshold settings, you can significantly enhance the resilience of your Windows Server Failover Cluster and minimize the occurrence of error 1135 and other network-related issues. Remember to always test changes in a non-production environment first before implementing them in production.

References¶

For comprehensive information and further reading on tuning Windows Cluster network configuration settings, consult the following resource:

Tuning Failover Cluster Network Thresholds.

---

Feel free to share your experiences with tuning failover cluster network thresholds or ask any questions in the comments below!