NUMA and Processor Affinity Issues in IIS: Understanding Performance Bottlenecks

Table of Contents

This article addresses a specific performance issue encountered in Internet Information Services (IIS) 10.0 running on hardware utilizing Non-Uniform Memory Access (NUMA) architecture. The problem arises when the IIS processor affinity setting, which is intended to bind worker processes to specific CPUs, fails to function correctly because a separate IIS optimization feature for NUMA is enabled. Understanding the interplay between NUMA, processor affinity, and IIS thread pool management is crucial for resolving this conflict and optimizing web server performance on modern hardware.

Understanding NUMA Architecture¶

Modern servers often employ Non-Uniform Memory Access (NUMA) architecture to improve scalability and performance beyond the limits of traditional Uniform Memory Access (UMA) systems. In a UMA system, all CPUs share access to a single, global memory pool, and memory access time is roughly uniform for all processors, regardless of where the data is located in memory. This model works well for a limited number of processors.

However, as the number of CPU cores increases, the bus connecting processors to memory becomes a bottleneck. To overcome this, NUMA divides the system’s physical memory into sections, with each section physically closer to a specific group of processors, forming what are called NUMA nodes. Each NUMA node typically consists of one or more processors and a local memory bank.

Processors within a node can access their local memory very quickly. Accessing memory located in a different NUMA node (remote memory) is still possible, but it incurs a higher latency due to communication overhead across the inter-node interconnect. This difference in access time is where the term “non-uniform” comes from. Operating systems and applications designed to be NUMA-aware can optimize performance by attempting to schedule threads on processors within the node that holds the data they frequently access, thereby minimizing costly remote memory accesses.

The adoption of NUMA architecture allows servers to scale to a significantly higher number of processors and larger amounts of memory. However, it also introduces complexity. Software, including web servers like IIS, must be designed or configured to operate efficiently in this environment. Poor NUMA awareness can lead to performance degradation, as processes or threads might frequently access remote memory, negating the benefits of the architecture.

Processor Affinity in Computing Systems¶

Processor affinity is a scheduling policy that binds a process or thread to a specific CPU or a set of CPUs within a multiprocessor system. This binding prevents the operating system’s scheduler from migrating the process or thread to other processors. Affinity can be configured at the operating system level for general processes or at the application level if the application provides such controls.

The primary motivations for using processor affinity include improving cache performance and reducing context switching overhead. When a thread runs on a CPU, it populates that CPU’s caches (L1, L2, L3) with frequently used instructions and data. If the thread is then migrated to a different CPU, the cached data is no longer readily available, and the new CPU’s caches must be repopulated, leading to potential performance stalls (cache misses). Binding a thread to a CPU ensures that it benefits from the data and instructions already in that CPU’s cache.

Furthermore, migrating a thread between CPUs involves saving the thread’s state on the current CPU and restoring it on the new CPU, which is a process called context switching. While necessary, frequent context switching incurs overhead. Processor affinity can reduce this overhead by preventing unnecessary migrations, although modern operating system schedulers are generally quite efficient at minimizing detrimental context switches.

In specific scenarios, such as dedicating certain processes to particular CPUs to ensure predictable performance or isolate workloads, processor affinity can be a valuable tuning tool. However, improperly applied affinity can also reduce performance by preventing the scheduler from optimally distributing load across available processors, potentially leaving some CPUs idle while others are overloaded.

IIS Processor Affinity Configuration¶

Internet Information Services (IIS) provides configuration settings that allow administrators to apply processor affinity to application pool worker processes (w3wp.exe). This capability enables administrators to specify that a particular application pool, which handles requests for one or more websites, should only run on a designated subset of the server’s processors.

The relevant settings are configured within the application pool’s advanced settings. The smpAffinitized property is a boolean flag that enables or disables processor affinity for the worker process. If smpAffinitized is set to true, the system will attempt to bind the worker process to the CPUs specified by the smpProcessorAffinityMask and smpProcessorAffinityMask2 properties.

These affinity mask properties use bitmasks to indicate which processors are enabled. Each bit in the mask corresponds to a logical processor on the system. If a bit is set to 1, the corresponding processor is included in the affinity mask; if it’s 0, the processor is excluded. smpProcessorAffinityMask is a 32-bit mask, and smpProcessorAffinityMask2 is used for systems with more than 32 processors, extending the mask to 64 bits. By setting these masks, an administrator can precisely control which CPUs an IIS worker process is allowed to run on.

For instance, on a system with 8 logical processors, setting smpProcessorAffinityMask to 0x0F (binary 00001111) would attempt to bind the worker process to processors 0, 1, 2, and 3. Setting it to 0x01 (binary 00000001) would attempt to bind it only to processor 0. This fine-grained control is intended to help manage resource allocation and potentially improve cache locality for specific application pools.

Introducing IIS Thread Pool Ideal CPU Optimization for NUMA¶

With the advent of IIS 10.0, a new performance optimization feature was introduced specifically for servers utilizing NUMA architecture. This feature, known as “IIS Thread Pool Ideal CPU Optimization for NUMA hardware,” aims to improve the efficiency of IIS worker processes by making the thread pool more NUMA-aware.

The core idea behind this optimization is to distribute the work handled by the IIS thread pool more effectively across the available NUMA nodes and processors. Instead of potentially concentrating activity on a few processors or nodes, leading to imbalanced load and increased remote memory access, the optimization attempts to leverage the full capacity of the system by intelligently distributing incoming requests and thread processing across all available logical processors, taking into account the NUMA topology.

By default, this IIS Thread Pool Ideal CPU Optimization is enabled in IIS 10.0. Its goal is to maximize overall throughput and system utilization on NUMA hardware by ensuring that threads are scheduled on processors that are considered “ideal” based on the NUMA topology, often favouring local memory access and balancing load across nodes. This approach is generally beneficial for aggregate web server performance on high-core-count, NUMA-based systems.

When this optimization is active, the IIS thread pool’s internal logic attempts to steer threads towards ideal CPUs across the entire system, considering the NUMA structure. This is a form of system-wide resource management aimed at optimizing performance from a global perspective, rather than restricting a specific process to a subset of resources.

The Conflict: Affinity vs. Optimization¶

The root cause of the issue where IIS processor affinity fails to work on NUMA hardware in IIS 10.0 stems from a direct conflict between the explicit instruction provided by the processor affinity settings (smpAffinitized, smpProcessorAffinityMask) and the implicit behavior driven by the IIS Thread Pool Ideal CPU Optimization for NUMA.

Processor affinity dictates that a worker process must run only on a specific, limited set of CPUs. It tells the operating system scheduler to restrict the process’s execution to these designated processors, ignoring all others for that particular process. This is a directive to constrain the process’s CPU usage.

Conversely, the IIS Thread Pool Ideal CPU Optimization for NUMA, when enabled, attempts to leverage all available NUMA nodes and their associated processors to distribute thread execution optimally and balance the load across the entire system. This is an internal IIS mechanism that guides its thread scheduling towards using the most suitable CPU from the perspective of NUMA performance, often implying a preference for spreading activity across the system to utilize local node resources and avoid congestion.

When both settings are active simultaneously for an application pool on IIS 10.0 NUMA hardware, they issue contradictory instructions. Processor affinity says, “Stay on these few CPUs.” The NUMA optimization says, “Use the ideal CPUs, which might be anywhere across the system, to balance load and improve NUMA performance.” In this specific conflict scenario observed in IIS 10.0, the NUMA optimization feature takes precedence over the processor affinity settings.

As a result, even if smpAffinitized is set to true and affinity masks are configured to restrict a worker process to, for example, CPUs in only one NUMA node or a specific set of cores, the IIS thread pool, guided by the Ideal CPU Optimization, will continue to schedule threads for that worker process on processors across multiple NUMA nodes or outside the specified affinity mask. The explicit affinity setting is effectively ignored or overridden by the internal optimization logic.

This conflict means that administrators attempting to use processor affinity for specific performance tuning or resource partitioning purposes on IIS 10.0 on NUMA hardware will find that their affinity settings have no effect. The worker process will behave as if affinity is disabled, distributing its load potentially across all available processors as determined by the optimization feature.

Symptoms of the Conflict¶

The primary symptom observed when encountering this issue is the failure of IIS 10.0 processor affinity settings to effectively constrain worker processes (w3wp.exe) on NUMA hardware. Despite having configured smpAffinitized="true" and set appropriate smpProcessorAffinityMask and potentially smpProcessorAffinityMask2 values for an application pool, monitoring the system’s process activity reveals that the corresponding worker process is executing on CPUs outside the specified affinity mask.

Administrators might use tools like Task Manager, Resource Monitor, or more advanced performance monitoring tools to observe the CPU usage of the w3wp.exe process. When affinity is working correctly, the process’s threads should only be seen utilizing the CPUs indicated by the mask. However, in this problematic scenario, the worker process’s activity is distributed across a wider range of processors, often spanning multiple NUMA nodes, as if no affinity mask were applied at all.

This lack of effective affinity can manifest in various ways, depending on why affinity was desired in the first place. If affinity was set to improve cache locality for a specific application, the application might experience unexpected cache misses and potentially lower performance than anticipated. If affinity was used to isolate a demanding application pool to a subset of processors to prevent it from impacting other workloads, that isolation might fail, leading to resource contention issues across the system.

Furthermore, while the IIS Thread Pool Ideal CPU Optimization is generally beneficial for overall system throughput, its interaction overriding affinity might not align with specific tuning goals where confining a workload is more important than general distribution. Debugging performance issues becomes harder when the configured affinity settings are not respected by the system.

Workaround¶

Fortunately, there is a workaround available to address this conflict and allow processor affinity to function as intended on IIS 10.0 NUMA systems. The workaround involves disabling the IIS Thread Pool Ideal CPU Optimization feature that is causing the conflict. Since this optimization is enabled by default in IIS 10.0, explicitly disabling it removes the conflicting behavior, allowing the processor affinity settings to take effect.

The IIS Thread Pool Ideal CPU Optimization feature is controlled by a registry setting. By modifying this registry value, you can turn off the optimization.

The registry key path is:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\InetInfo\Parameters

Within this key, the specific value that controls the optimization is ThreadPoolUseIdealCpu. By default, on IIS 10.0 NUMA systems, this value is set to 1, indicating that the Ideal CPU Optimization is enabled. To disable it, you need to change the value of ThreadPoolUseIdealCpu from 1 to 0.

Here are the steps to apply the workaround:

1. Open the Registry Editor (regedit.exe) with administrative privileges.
2. Navigate to the path HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\InetInfo\Parameters.
3. Look for the DWORD (32-bit) Value named ThreadPoolUseIdealCpu.
4. If the value exists and is set to 1, double-click it and change the ‘Value data’ from 1 to 0.
5. If the value does not exist, you may need to create it as a new DWORD (32-bit) Value and set its data to 0.
6. Close the Registry Editor.
7. For the change to take effect, you typically need to restart the IIS Admin Service or, more reliably, restart the server. Restarting the server ensures all relevant IIS processes and services pick up the new registry setting.

Once the ThreadPoolUseIdealCpu value is set to 0 and IIS services are restarted, the IIS Thread Pool Ideal CPU Optimization for NUMA will be disabled. With this optimization out of the way, the processor affinity settings (smpAffinitized, smpProcessorAffinityMask, smpProcessorAffinityMask2) configured for your application pools should now be correctly respected by the operating system scheduler, allowing you to bind worker processes to the desired subset of processors.

It is important to note that disabling the ThreadPoolUseIdealCpu setting disables the specific NUMA optimization feature that conflicts with affinity. While this allows affinity to work, it might also mean that the IIS thread pool will not leverage the NUMA architecture for load balancing in the way the optimization intended. Therefore, this workaround should be used when processor affinity is a specific requirement, and the administrator is willing to potentially forgo the default NUMA optimization benefits for the affected application pools. Testing performance after applying the workaround is recommended to ensure it meets the desired outcome.

Resolution¶

The issue where IIS 10.0 processor affinity conflicts with the NUMA Thread Pool Ideal CPU Optimization is recognized as a bug in certain releases of IIS 10.0. Microsoft has addressed this conflict in later versions and updates to IIS 10.0.

The definitive resolution for this problem is to upgrade your installation of Internet Information Services to a version where this specific issue has been fixed. The original article mentions that the fix is available in later versions of IIS 10.0, such as “IIS 10.0 RS3” (referring to the Windows Server/Client release associated with that version, like Windows Server 2016/Windows 10 Anniversary Update and later).

Upgrading IIS, typically by applying the latest Windows Server cumulative updates or installing a newer Windows Server version that includes an updated IIS release, will incorporate the necessary code changes to correctly handle both the NUMA optimization and processor affinity settings. In fixed versions, IIS is designed to either prioritize affinity correctly when configured or manage the thread pool in a way that respects both the overall NUMA architecture and the specific affinity constraints placed on a worker process.

By upgrading, you should no longer need to apply the registry workaround (ThreadPoolUseIdealCpu=0) to make processor affinity work. This means you can potentially benefit from the default IIS NUMA optimizations for application pools where affinity is not set, while also having the ability to apply processor affinity to specific application pools when needed for particular tuning or isolation purposes.

It is always recommended to keep your server operating system and components like IIS up-to-date with the latest patches and cumulative updates. This ensures you have fixes for known issues, including performance-related bugs and security vulnerabilities. Before applying major upgrades in a production environment, it is advisable to test the upgrade on a staging or development system to confirm compatibility and verify that the issue is resolved and no new problems are introduced.

Implementing the resolution via upgrade provides a more robust solution than relying solely on the workaround, as it represents the intended behavior of IIS on modern NUMA hardware.

Performance Considerations and Best Practices¶

When working with IIS on NUMA systems and considering settings like processor affinity and thread pool optimization, it’s crucial to understand the potential performance implications and apply best practices.

Processor affinity, while a powerful tool for specific scenarios, is not a universal performance booster. It can be beneficial for applications with high cache locality requirements or when dedicating CPU resources is necessary. However, in many cases, the default operating system and application (like IIS) scheduling, which aims to balance load dynamically across all available CPUs, can provide better overall throughput. Binding a process to a small number of CPUs on a many-core system can lead to underutilization of available processing power if that process becomes a bottleneck.

The IIS Thread Pool Ideal CPU Optimization for NUMA in IIS 10.0 was introduced precisely because the default dynamic scheduling was enhanced to be NUMA-aware. This optimization aims to improve performance by intelligently distributing work across the NUMA nodes to minimize remote memory access latency and balance CPU load from a system perspective. For most general-purpose IIS workloads on NUMA hardware, leaving this optimization enabled (which is the default) is likely to yield better overall performance and scalability.

Therefore, processor affinity in IIS should be considered a tuning knob for specific, advanced scenarios rather than a default configuration. Use it only when you have a clear reason and have verified through testing that it provides a measurable benefit for a particular application pool. Examples might include:

• Isolating a Resource-Intensive Application: If one application pool consistently consumes a large amount of CPU, you might attempt to bind it to a specific set of cores to prevent it from impacting other application pools.
• Troubleshooting Cache-Related Issues: For applications known to benefit significantly from CPU cache residency, testing affinity might reveal performance improvements.
• Licensing Restrictions: In rare cases, software licenses tied to a specific number of CPU cores might necessitate limiting a process’s visibility to cores.

Before applying processor affinity or disabling the NUMA optimization via the registry workaround, it is strongly recommended to:

1. Understand your workload: Identify which application pools are experiencing performance issues or require specific resource management.
2. Establish a performance baseline: Measure key performance indicators (CPU usage, request latency, throughput, error rates) before making any configuration changes.
3. Apply changes methodically: Implement the workaround or affinity settings on a test system first.
4. Monitor and test: After applying changes, rigorously monitor performance metrics to confirm that the desired improvement was achieved and that no negative side effects were introduced elsewhere in the system.

Disabling the ThreadPoolUseIdealCpu optimization via the registry should be seen as a temporary workaround until you can upgrade IIS to a version that resolves the conflict properly. Once upgraded, re-evaluate whether processor affinity is still necessary or if the improved default NUMA handling in the newer IIS version provides sufficient performance gains.

Consider using tools like perfmon or Visual Studio’s performance profiler to analyze CPU usage per core, cache misses, and NUMA memory access patterns (if the system provides such counters) to make informed decisions about affinity and other performance tuning settings.

What has been your experience with IIS processor affinity on NUMA hardware? Have you encountered similar conflicts or found other effective tuning strategies? Share your thoughts in the comments below.