Boost Windows Server Performance: Troubleshooting TCP/IP Issues

Table of Contents

Measuring Transmission Control Protocol/Internet Protocol (TCP/IP) performance involves comparing throughput between identical endpoints. Ideally, this comparison should be made using machines with similar hardware configurations, traversing the same network path, and running the same Operating System (OS). However, real-world performance is influenced by numerous factors that can act as bottlenecks. These often include the capabilities and state of the underlying network infrastructure, inherent characteristics of the TCP protocol itself, and the transmission rate of storage Input/Output (IOs).

Effective performance tuning is crucial for achieving optimal network throughput. It involves configuring the endpoints appropriately to leverage their full potential. Detailed guides on network subsystem performance tuning are available to help set the best parameters for network interface cards (NICs) and other related components on Windows Server.

TCP receive window autotuning is a significant feature designed to improve performance, particularly on networks experiencing high latency. This feature allows applications to dynamically scale the TCP receive window size. A larger window enables more data segments to be in transit before an acknowledgment is required, thereby utilizing available bandwidth more effectively on high-latency links. For instance, if autotuning is set to ‘normal’ and the application provides sufficient buffer space, the OS can quickly increase the TCP window to its maximum limit. Conversely, in low-latency network environments, the benefits of aggressive window scaling are less pronounced, as fewer segments are typically in flight simultaneously.

For networks characterized by high latency where the TCP window doesn’t immediately scale to its maximum, Windows implements sophisticated algorithms. These algorithms, such as CUBIC, NewReno, and Compound TCP, work to estimate the bandwidth-delay product (BDP) of the connection. By understanding the BDP, the algorithms can appropriately adjust the TCP window size to maximize throughput without overwhelming the network or causing excessive packet loss. Windows dynamically assigns one of these congestion control algorithms to each TCP socket created, optimizing performance based on network conditions.

Current TCP settings on Windows systems, including Windows 10 and Windows Server versions, are largely predefined for common scenarios. Administrators can inspect these settings using the Get-NetTCPSettings PowerShell cmdlet. To gain insight into active TCP connections, including details like local and remote IP addresses and ports, as well as connection state, the Get-NetTCPConnection cmdlet is invaluable. These tools provide visibility into how TCP is configured and behaving on a given system.

Enhancing TCP/IP Throughput: Practical Tips¶

Achieving high TCP/IP throughput requires a systematic approach that addresses potential performance inhibitors at multiple layers of the network stack. Several key areas should be examined and optimized:

• Underlying Network Health: Ensure there are no fundamental issues within the network infrastructure itself. Factors like packet loss, excessive jitter, or high latency on the physical or link layer will significantly degrade TCP performance, as the protocol will interpret these as congestion and slow down.
• NIC Advanced Properties: Configure the advanced properties of the network interface cards for performance. Features such as Jumbo frames (if supported end-to-end), Receive Side Scaling (RSS) or Virtual Machine Queue (VMQ) for distributing load across CPU cores, various offload features (like Checksum Offload, Large Send Offload - LSO), and Receive Segment Coalescing (RSC) can dramatically improve performance. These should generally be enabled unless specific network compatibility issues arise or during targeted troubleshooting.
• TCP Autotuning Level: Verify that the TCP receive window autotuning level is configured appropriately, typically set to normal for most use cases unless there’s a specific reason to restrict it. This allows the OS to dynamically size the receive window based on network conditions and application buffer availability.
• System Resource Monitoring: Utilize performance monitoring tools, such as Windows Performance Monitor (Perfmon), to check for potential bottlenecks outside the network layer. High CPU utilization or slow storage IOs can limit the rate at which data can be processed or consumed by applications, thereby capping effective network throughput.
• Security Feature Selection: Evaluate security requirements carefully. Implement security features, such as IPsec, based strictly on actual organizational needs. Overly aggressive or unnecessary security processing can add significant overhead, consuming CPU resources and reducing throughput.
• Baseline Creation: Establish a performance baseline before making configuration changes or experiencing issues. A baseline represents the expected performance under normal conditions and is essential for comparing results after tuning or during troubleshooting to identify if performance has improved or degraded.

Testing Tool for TCP Throughput¶

To accurately measure the maximum achievable throughput for a specific hardware configuration, performance tuning is necessary, and verifying the absence of underlying network problems like packet loss is critical. Dedicated tools are required for stress- testing the network layer and quantifying performance.

Two widely-used tools from Microsoft for this purpose are NTttcp.exe and ctsTraffic.exe. Both tools are designed to generate and receive TCP traffic at high rates, bypassing application-layer complexities to measure the raw capability of the network stack and underlying hardware. These tools typically operate in a client-server model, allowing measurement of both upload (push) and download (pull) scenarios.

These tools provide detailed statistics on throughput, CPU utilization, and other relevant metrics. Referring to documentation on Performance Tools for Network Workloads can offer guidance on selecting the appropriate tool and understanding its features.

Bottlenecks for TCP Throughput: Common Pitfalls¶

Certain activities, while useful for deep analysis, can significantly skew performance results during TCP throughput tests. Avoid using generic network monitoring tools or capturing detailed network packet logs (like using Wireshark or Microsoft Network Monitor) during high-throughput testing.

Network Driver Interface Specification (NDIS) monitoring filters, used by these tools, introduce processing overhead. Each packet captured adds delay for both the sender and receiver. This process consumes significant CPU resources and generates numerous storage IOs as packets are written to disk. The very act of capturing can reduce the network performance being measured, potentially causing the TCP protocol to falsely detect packet loss or congestion due to the monitoring-induced delays, leading it to unnecessarily reduce its transmission rate as part of its congestion control mechanisms.

Adding security layers introduces processing costs and can become a significant performance bottleneck. Protocols like Internet Protocol Security (IPsec) require encryption, decryption, and integrity checking, all of which demand CPU cycles. When comparing data protection methods, using IPsec in integrity-only mode (Authentication Header - AH) generally incurs less processing overhead than modes providing both integrity and confidentiality (Encapsulating Security Payload - ESP, especially with encryption). Furthermore, third-party security software, such as firewalls or intrusion prevention systems, can add substantial latency and processing demands on packet handling paths, potentially slowing down network output considerably.

If tested performance falls short of expectations or baseline figures, collecting targeted logs using Windows Performance Monitor and focusing on Network-Related Performance Counters is recommended. These counters provide insights into various aspects of network activity without the heavy overhead of packet capture. Key counters include bytes sent/received, packet rates, errors, discards, and protocol-specific statistics.

Beyond the TCP layer, slow performance might stem from issues in upper-layer protocols. File system protocols like Server Message Block (SMB) or Network File System (NFS) are common examples. These protocols rely on the underlying TCP transport but also add their own processing requirements, consuming CPU resources and depending heavily on disk IO performance. Slowness at this layer could be attributed to inefficient protocol implementation, poor configuration, or underlying issues like a faulty driver, a high Deferred Procedure Call (DPC) queue delaying critical processing, or simply slow disk IO operations preventing data from being read or written fast enough.

Diagnosing high DPC activity can be challenging, often requiring advanced analysis using tools like Xperf or Windows Performance Recorder (WPR) to capture detailed CPU usage profiles. Identifying disk-related performance issues is comparatively easier using tools like Performance Monitor to check counters such as Disk Queue Length, Average Disk Sec/Transfer, and Disk Bytes/Sec. Detailed guides on examining and tuning disk performance are available.

When using testing applications like ctsTraffic or NTttcp, adjusting parameters like the number of threads and buffer sizes is possible to maximize throughput. While this helps determine the theoretical maximum hardware capability, it might not accurately reflect real-world application performance. Actual applications are constrained by their design, including the number of threads they use for network operations and the buffer sizes configured for API calls. Furthermore, application-layer protocols like SMB or CIFS have their own built-in buffering, caching, and optimization mechanisms. If performance is poor for a specific application compared to the established baseline, collaboration with an application specialist may be necessary to identify the bottleneck within the application layer or its interaction with the OS and network stack.

How to Create a Baseline¶

Creating a performance baseline is fundamental to effective network troubleshooting. It provides a point of comparison to evaluate current performance, identify deviations, and measure the impact of configuration changes. The baseline should ideally be established early in the system’s deployment phase, under normal operating conditions, to accurately represent typical performance characteristics.

A robust performance baseline encompasses several critical factors:

• Source and Destination Networks: Clearly identify the specific network segments, subnets, or even physical locations involved in the traffic flow being measured.
• Latency and Hop Count: Measure the network latency (Round Trip Time - RTT) and the number of network hops between the source and destination endpoints. These metrics are fundamental determinants of TCP performance, especially RTT, which directly influences the effectiveness of window scaling and congestion control.
• Processor/Interface Capability and Configuration: Document the CPU specifications (model, core count, speed) and the network interface card details (model, speed, driver version) and their configuration settings (e.g., link speed, duplex, advanced settings like RSS, offloads). Ensure similar hardware is used when comparing across systems.
• Time Frame of Tests: Record when the tests were conducted (e.g., during working hours, off-peak hours, peak load periods). Network congestion varies throughout the day, and tests conducted at different times can yield vastly different results.
• OS Versions: Note the specific versions and build numbers of the operating system running on both the source and destination machines. OS versions can have different network stack implementations and default settings.
• Throughput Direction: Measure both “pull” (data transferred from server to client) and “push” (data transferred from client to server) throughput, as performance can differ depending on the direction due to asymmetrical network paths or endpoint processing capabilities.

Important Note: When creating baselines for comparison across different servers or environments, strive to use server models with similar processing power and network adapter configurations (number and type of NICs) to minimize variations due to hardware capabilities.

During testing, pay attention to resource utilization, particularly CPU distribution across cores, especially if RSS is enabled. Aim for balanced CPU utilization. Also, monitor the number of simultaneous TCP transport sessions created by the testing tools. Tools like ctsTraffic often perform best utilizing a number of connections equivalent to the number of RSS queues configured on the NIC (commonly 4 or 8), as this can help distribute processing load effectively across CPU cores.

Here are the steps for measuring throughput using the ctsTraffic tool and creating a baseline record:

1. Download the ctsTraffic tool: Obtain the latest version of the tool from a trusted source, such as Microsoft’s GitHub repository.

2. Understand ctsTraffic Parameters: Familiarize yourself with the tool’s command-line parameters. Key parameters include:

   Parameter	Description
   -listen:<IP/*>	Used on the server side. Specifies the IP address to bind to for listening. Using * binds to all available IP addresses on the machine. Example: -listen:*.
   -target:<IP>	Used on the client side. Specifies the IP address of the server where ctsTraffic is listening. Example: -target:192.168.1.10.
   -pattern:pull	Used on both client and server for a data pull test (data sent from server to client). Default is push (client to server). Do not use this for push tests.
   -connections:<num>	Specifies the number of simultaneous TCP connections the client will establish. A common value is 8, often matching the number of RSS queues. Example: -connections:8.
   -iterations:<num>	Multiplies the number of connections specified by -connections. For example, -iterations:10 with -connections:8 attempts 80 connections in total. If not specified, the client runs connections until a default limit (e.g., 1000).
   -statusfilename:<file>	Saves basic test summary (console verbosity level 1 output) to a .txt file, often compatible with Excel for simple charting.
   -connectionfilename:<file>	Saves verbose socket-level details, including individual connection timings and errors, to a .csv file. Useful for in-depth troubleshooting.
   -consoleverbosity:<0-3>	Controls the amount of output displayed on the console during the test. 1 provides a summary, higher numbers provide more detail. Example: -consoleverbosity:1.

3. Open Resource Monitor: On the machine receiving the data (client for pull, server for push), open Resource Monitor to observe CPU utilization per core during the test. This helps verify RSS is distributing load effectively.

4. Start ctsTraffic on the Server: Open a command prompt or PowerShell window and run the server-side command. Include -pattern:pull if testing pull throughput.
   

   Ctstraffic.exe -listen:* -consoleverbosity:1 -pattern:pull
   

   
   (Remove -pattern:pull for push tests).
5. Start ctsTraffic on the Client: Open a separate command prompt or PowerShell window on the client machine and run the client-side command. Ensure the -target IP is correct and include -pattern:pull if performing a pull test. Adjust -connections and -iterations as needed for your test scenario.
   

   Ctstraffic.exe -target:192.168.1.10 -consoleverbosity:1 -pattern:pull -connections:8 -iterations:10
   

   
   (Replace 192.168.1.10 with the server’s IP, remove -pattern:pull for push tests).
6. Monitor CPU Utilization: While the test runs, observe the CPU utilization in Resource Monitor on the receiving machine. Confirm that multiple CPU cores are actively engaged, ideally with balanced load. If not, troubleshoot RSS configuration or NIC driver issues.

7. Calculate Throughput: Once the test completes, the ctsTraffic output on the client side will provide total bytes transferred and the test duration. Use these values to calculate the throughput, typically expressed in bits per second (Gb/s or Mb/s). Remember that 1 byte = 8 bits.

   

   For example, if the output shows Total Bytes=85,899,349,200 and Test Duration=36.678 seconds, the calculation is:
   (85,899,349,200 Bytes / 36.678 seconds) * 8 bits/Byte = 18,735,885,097.34 bits/second.
   To convert to Gigabits per second: 18,735,885,097.34 / (1024^3 / 8 * 1000 * 1000) or simply / 1,000,000,000 for Gigabits (decimal) = approximately 18.74 Gb/s (decimal).

Document these results meticulously as your baseline for the specific test scenario (source/destination, hardware, time, push/pull, parameters used).

Next Steps¶

After establishing a performance baseline and understanding potential bottlenecks, continue monitoring and tuning. Regularly compare current performance against your baseline. Investigate any significant deviations using performance counters and targeted troubleshooting based on the potential bottlenecks identified. Remember that optimizing network performance is an ongoing process.

Have you encountered specific TCP/IP performance issues on Windows Server? What tools and techniques have you found most effective in troubleshooting and optimizing throughput? Share your experiences and insights in the comments below!