Resolve High Memory Usage in ReFS on Windows Server: A Comprehensive Fix

Table of Contents

Introduction¶

This document provides a detailed solution to address concerns regarding excessive memory consumption and related performance degradation when utilizing the Resilient File System (ReFS) on Windows Server environments. ReFS, designed for maximum data availability, can in certain scenarios exhibit higher memory usage compared to traditional file systems like NTFS. This article will guide you through understanding the underlying causes of this behavior and implementing effective resolutions to optimize your system’s performance.

Applicable Systems¶

The solutions and information detailed in this article are applicable to the following Microsoft Windows operating systems:

• Windows 10 (all editions)
• Windows Server 2016
• Windows Server 2019
• Windows Server, version 1903
• Windows Server, version 1909

This guidance is particularly relevant for administrators and IT professionals managing servers running these operating systems and utilizing ReFS for critical data storage.

Symptoms of High Memory Usage¶

You may encounter the following symptoms if your Windows Server system is experiencing high memory usage related to ReFS:

• Performance Degradation: The server may become sluggish and unresponsive. Applications and services hosted on the server may exhibit slower performance, impacting user experience and business operations.
• High RAM Utilization: Monitoring tools such as Task Manager or Resource Monitor will indicate consistently high RAM usage, often nearing or reaching maximum capacity. This can lead to system instability and potential crashes.
• Memory Pressure: The system may experience memory pressure, where available RAM is insufficient to meet demands. This can trigger excessive paging to disk, further slowing down the system.
• Application Errors: Some applications may encounter errors related to memory allocation failures or timeouts due to insufficient available memory.
• Slow File Operations: File operations on ReFS volumes, such as copying, moving, or accessing files, may become noticeably slower than expected.

These symptoms can collectively disrupt server operations and require prompt attention to restore optimal performance.

Root Cause: ReFS Allocate-on-Write Semantics¶

The primary driver behind the observed high memory usage in ReFS lies in its fundamental design principle: allocate-on-write semantics for metadata updates. Unlike traditional file systems that often employ write-in-place updates, ReFS adopts a more resilient approach to ensure data integrity.

Understanding Allocate-on-Write:

In a write-in-place system, when metadata needs to be updated (for example, when a file is created, modified, or deleted), the file system directly modifies the existing metadata structures on disk. This approach can be faster but introduces a vulnerability: if a power failure or system crash occurs during the write operation, the metadata can become corrupted or inconsistent, leading to data loss or file system errors. This scenario is known as a “torn write”.

ReFS, to mitigate the risk of torn writes and enhance data resiliency, implements allocate-on-write. Instead of modifying metadata in place, ReFS always allocates new storage space for updated metadata and writes the changes to these newly allocated regions. The old metadata remains untouched until the new metadata is successfully written and committed. This ensures that even in the event of an interruption, ReFS can always revert to the previous, consistent state of the metadata, guaranteeing file system integrity.

Impact on Memory Usage:

While allocate-on-write provides superior data protection, it inherently leads to increased I/O operations and potentially higher memory usage. Here’s why:

1. Increased Metadata I/O: Allocate-on-write necessitates writing metadata updates to new locations on the storage volume. This results in more metadata I/O operations compared to write-in-place systems, which might update metadata in the same location.

2. Block Caching for Metadata: ReFS employs a block caching mechanism to cache its metadata in RAM for faster access. While caching is generally beneficial, ReFS’s block caching logic for metadata is not as resource-efficient as the file caching logic used for file data. This means that ReFS metadata caching might consume more RAM for the same amount of cached data compared to file data caching.

3. Lazy Unmapping and Active Working Set Growth: ReFS utilizes the cache manager to manage metadata streams. The cache manager lazily unmaps inactive views of metadata from memory. In certain workload scenarios, this lazy unmapping can lead to a gradual increase in the active working set of the server, meaning the amount of RAM actively used by ReFS metadata grows over time. This expansion of the working set can eventually create memory pressure, especially on systems with limited RAM or under heavy load.

Combined Effect:

The combination of ReFS’s allocate-on-write semantics and its block caching strategy for metadata can result in larger metadata streams and increased memory consumption. In environments with intensive metadata operations, such as virtualized environments or database servers heavily relying on file system metadata, this effect can become more pronounced and lead to the high memory usage symptoms previously described.

Resolution: Cumulative Update and Tunable Parameters¶

Microsoft addressed the high memory usage issue in ReFS through cumulative update 4013429, released on March 14, 2017. This update introduces three tunable registry parameters that allow administrators to fine-tune ReFS’s memory management behavior and mitigate excessive memory consumption.

Obtaining the Update:

Cumulative update 4013429 is available through Windows Update. It is highly recommended to install this update to benefit from the performance improvements and memory management enhancements it provides for ReFS.

Alternatively, you can download the update directly from the Microsoft Update Catalog by searching for KB4013429.

Tunable Registry Parameters:

The update introduces the following three registry parameters, which can be configured to adjust ReFS’s memory management strategies:

• RefsEnableLargeWorkingSetTrim: Controls aggressive trimming of the ReFS metadata working set.
• RefsNumberOfChunksToTrim: Adjusts the granularity of memory unmapping during namespace cycling.
• RefsEnableInlineTrim: Enables inline MM trim during metadata page unmapping, the most aggressive option.

These parameters are optional and can be used individually or in combination, as they address different aspects of ReFS memory management.

Configuring Tunable Parameters: Step-by-Step Guide¶

To implement the resolution, you need to configure the tunable registry parameters. Caution: Incorrectly modifying the registry can cause serious problems. Always back up the registry before making any changes.

The registry keys for these parameters are located in the following subkey:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem

You will need to create new REG_DWORD values within this subkey if they don’t already exist.

Option 1: Complete MM Unmap at Checkpoint (RefsEnableLargeWorkingSetTrim)¶

This option instructs ReFS to attempt a complete unmapping of all metadata streams at every checkpoint. A checkpoint in ReFS is a point in time where the file system’s metadata is consistently written to disk. By forcing a complete unmap at each checkpoint, this option aims to aggressively reclaim memory used by metadata.

Mechanism:

When RefsEnableLargeWorkingSetTrim is set to 1, ReFS will initiate a process to unmap all metadata streams from memory during each checkpoint operation. This operation will only be fully effective if the ReFS volume is relatively idle and does not have actively mapped metadata pages. If there are active metadata pages, they cannot be unmapped immediately and will remain in memory until they become inactive.

Registry Setting:

1. Open Registry Editor (regedit.exe).
2. Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem.
3. Right-click in the right pane, select New -> DWORD (32-bit) Value.
4. Name the new value: RefsEnableLargeWorkingSetTrim.
5. Double-click RefsEnableLargeWorkingSetTrim and set the Value data to 1.
6. Click OK.
7. Restart the system for the changes to take effect.

When to use:

This option is a good starting point and is generally safe to apply. It is most effective in scenarios where ReFS volumes experience periods of low activity, allowing for more complete memory reclamation during checkpoints.

Option 2: Adjusting Unmap Granularity (RefsNumberOfChunksToTrim)¶

ReFS employs a lazy memory management logic for unmapping metadata. When ReFS cycles through its namespace to perform unmapping, it does so in chunks or segments of virtual address space. The RefsNumberOfChunksToTrim parameter controls the size of these chunks, effectively adjusting the granularity of the unmapping process.

Mechanism:

The amount of virtual address space unmapped is determined by the following formula:

• Volume size > 10 TB: RefsNumberOfChunksToTrim * 128 MB
• Volume size < 10 TB: RefsNumberOfChunksToTrim * 64 MB

The default value for RefsNumberOfChunksToTrim is 4 if the parameter is not explicitly set or set to 0. Increasing the value of RefsNumberOfChunksToTrim will increase the amount of memory unmapped in each cycle, potentially leading to more aggressive memory reclamation.

Registry Setting:

1. Open Registry Editor (regedit.exe).
2. Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem.
3. Right-click in the right pane, select New -> DWORD (32-bit) Value.
4. Name the new value: RefsNumberOfChunksToTrim.
5. Double-click RefsNumberOfChunksToTrim and set the Value data to a desired number (e.g., 8, 16, 32). Start with a moderate increase from the default.
6. Click OK.
7. Restart the system for the changes to take effect.

When to use:

This option is useful when Option 1 alone is insufficient. Experiment with different values for RefsNumberOfChunksToTrim. Higher values will lead to more aggressive unmapping but could potentially introduce a slight performance overhead if active metadata is frequently unmapped and then needs to be re-mapped. Start with values like 8 or 16 and gradually increase if needed.

Option 3: Inline MM Trim (RefsEnableInlineTrim)¶

This is the most aggressive memory management option. When enabled, ReFS sends down an MM (Memory Manager) trim command inline while it unmaps its metadata pages. MM trim is a more forceful operation that directly signals to the memory manager to release physical memory associated with the unmapped virtual address space.

Mechanism:

By setting RefsEnableInlineTrim to 1, ReFS instructs the memory manager to immediately release physical memory when metadata pages are unmapped. This can lead to the most significant reduction in memory usage among the three options.

Registry Setting:

1. Open Registry Editor (regedit.exe).
2. Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem.
3. Right-click in the right pane, select New -> DWORD (32-bit) Value.
4. Name the new value: RefsEnableInlineTrim.
5. Double-click RefsEnableInlineTrim and set the Value data to 1.
6. Click OK.
7. Restart the system for the changes to take effect.

When to use and Considerations:

This option should be used with caution and as a last resort if Options 1 and 2 do not provide satisfactory results. While it can aggressively reduce memory usage, it carries a potential performance risk, especially on high-performance storage media like SSDs or NVMe drives.

Performance Regression Risk:

Inline MM trim can introduce performance regression because it might trigger more frequent and potentially more expensive memory management operations. On high-performance storage, the overhead of these operations might become noticeable, especially under heavy I/O load. Therefore, carefully monitor system performance after enabling RefsEnableInlineTrim. If you observe performance degradation, consider reverting to less aggressive options or adjusting RefsNumberOfChunksToTrim instead.

Recommendation for Parameter Configuration¶

The recommended approach to configuring these parameters is to start with the least aggressive option and progressively move to more aggressive options if needed:

1. First, try setting RefsEnableLargeWorkingSetTrim = 1. This is generally the safest option and can be effective in many scenarios. Monitor memory usage after applying this setting and restarting the server.

2. If RefsEnableLargeWorkingSetTrim alone doesn’t produce the desired result, try adjusting RefsNumberOfChunksToTrim. Start by setting it to a value like 8 or 16, and then experiment with higher values (e.g., 32, 64) if necessary. Monitor memory usage and system performance after each adjustment.

3. Only as a last resort, if the above options are insufficient, consider setting RefsEnableInlineTrim = 1. Carefully monitor system performance after enabling this option for any signs of regression. If performance degradation is observed, consider reverting to a less aggressive configuration.

Summary Table of Registry Parameters:

Parameter	Value Type	Description	Recommendation
RefsEnableLargeWorkingSetTrim	REG_DWORD	Enables complete MM unmap of metadata streams at each checkpoint.	Start with this option. Generally safe and effective for idle volumes.
RefsNumberOfChunksToTrim	REG_DWORD	Adjusts the granularity of memory unmapping. Higher values = more aggressive unmapping.	Experiment with values like 8, 16, 32. Balance memory reduction with potential performance overhead. Default is 4.
RefsEnableInlineTrim	REG_DWORD	Enables inline MM trim during metadata page unmapping. Most aggressive option.	Use as a last resort. Monitor for performance regression, especially on high-performance storage.

Deeper Dive: Understanding ReFS Allocate-on-Write¶

As previously discussed, ReFS’s allocate-on-write approach is a cornerstone of its design, aimed at maximizing data integrity and resilience. Let’s delve deeper into the rationale and benefits of this technique.

Resilience to Torn Writes:

The primary motivation behind allocate-on-write is to eliminate the risk of torn writes, a significant vulnerability in write-in-place file systems. Torn writes occur when a write operation to metadata is interrupted, typically due to a power failure or unexpected system dismount. If a write is only partially completed in a write-in-place system, the metadata can become corrupted and inconsistent. This inconsistency can lead to file system errors, data loss, and even file system unmountability.

How Allocate-on-Write Prevents Torn Writes:

Allocate-on-write elegantly solves the torn write problem. By always writing metadata updates to newly allocated regions, ReFS ensures that the old, consistent metadata copy remains intact until the new metadata is fully and successfully written. In the event of an interruption during the write process, ReFS can simply revert to the previous, consistent metadata copy. This guarantees metadata consistency and prevents data corruption caused by torn writes.

Benefits of Allocate-on-Write for Data Integrity:

Beyond preventing torn writes, allocate-on-write contributes to overall data integrity in several ways:

• Metadata Checksumming: ReFS employs checksumming for metadata. With allocate-on-write, when metadata is updated, both the new metadata and its checksum are written atomically to the new location. This further enhances data integrity by detecting and preventing silent data corruption.
• Copy-on-Write Semantics: Allocate-on-write is closely related to copy-on-write semantics. This approach not only protects against torn writes but also facilitates features like snapshots and data deduplication, which rely on the ability to efficiently create consistent copies of data.

Trade-offs and Memory Management:

While allocate-on-write offers significant advantages in terms of data integrity and resilience, it comes with trade-offs, primarily in terms of I/O overhead and potential memory usage. As discussed earlier, the increased metadata I/O and ReFS’s metadata caching strategy can contribute to higher memory consumption in certain scenarios. The tunable registry parameters introduced in cumulative update 4013429 are designed to address these trade-offs and allow administrators to balance data integrity with memory efficiency based on their specific workload requirements.

Conclusion¶

High memory usage in ReFS on Windows Server, while sometimes a concern, is often a consequence of its robust allocate-on-write design, which prioritizes data integrity and resilience. By understanding the underlying causes and implementing the resolutions outlined in this article, specifically by applying cumulative update 4013429 and carefully configuring the tunable registry parameters, administrators can effectively mitigate excessive memory consumption and optimize ReFS performance. Remember to monitor your system after making changes to ensure optimal balance between memory efficiency and data integrity.

We encourage you to share your experiences and questions in the comments below. Have you encountered high memory usage with ReFS? What configuration strategies have you found most effective? Your insights can be valuable to the community!