Home backup and recovery it security server administration windows server

Windows Server 2016: Beware Backup Errors Post-Update – Third-Party Compatibility Issues

Updated: • 0 • min read
Table of Contents

Windows Server 2016 update issue

Windows Server 2016 provides a robust platform for critical business operations, offering significant improvements in various areas, including storage, networking, and security. Maintaining the system with regular updates is essential for security patches and performance enhancements. However, applying these updates can sometimes introduce unforeseen compatibility challenges, particularly with third-party software. A notable issue that administrators have encountered pertains to backup operations failing or reporting errors after installing servicing updates, specifically affecting third-party backup applications relying on the Volume Shadow Copy Service (VSS).

This document explores a specific scenario where servicing stack updates in Windows Server 2016 (Version 1607) can lead to VSS System Writer errors, causing third-party backup software to report files as missing. We will delve into the technical details of this problem, explain its root cause, and provide a practical solution to restore proper backup functionality. Understanding the interaction between Windows updates, VSS, and backup applications is crucial for maintaining data integrity and ensuring business continuity.

Understanding the Impact of Servicing Updates

Windows Server 2016, like its predecessors, relies on a component-based servicing architecture. Servicing stack updates are a critical type of update that prepare the operating system to receive future updates, including cumulative updates, security updates, and other fixes. These updates modify the underlying servicing components of the operating system, which are responsible for installing and managing other software updates.

A common consequence of applying a servicing stack update is the restructuring or modification of files within the C:\Windows\servicing directory. This directory contains important files related to the servicing process, including manifests, logs, and state files. In the specific case observed, a servicing stack update applied to Windows Server 2016 version 1607 (build 14393.693) caused a significant change in how certain files were handled within this directory structure.

The Servicing Directory Change

Prior to the specific update, certain servicing state files, notably amd64_installed and x86_installed, resided within a version-specific subfolder under C:\Windows\servicing\Version, such as C:\Windows\servicing\Version\10.0.14393.0. These files likely track the installation state of components for different architectures (AMD64 for 64-bit, x86 for 32-bit, though Server 2016 is primarily 64-bit).

After installing a servicing stack update (for example, KB 4013418), the system creates a new version folder, reflecting the updated build number, such as C:\Windows\servicing\Version\10.0.14393.1051. The updated versions or equivalents of the amd64_installed and x86_installed files are then saved into this new folder. Crucially, the older-version folder (10.0.14393.0 in this example) is subsequently emptied, and the original amd64_installed and x86_installed files are removed from their previous location.

This change in file location, while seemingly innocuous from an operating system perspective, creates a discrepancy that impacts applications relying on snapshots of the system state taken before the files were moved or deleted. This is where the Volume Shadow Copy Service (VSS) and its writers come into play.

The Role of VSS and the System Writer

The Volume Shadow Copy Service (VSS) is a technology in Microsoft Windows that allows backup applications and other management tools to create consistent, point-in-time copies (shadow copies) of volumes. This is essential for backing up files that are in use, ensuring that the copied data is in a coherent state. VSS coordinates with applications that have VSS “writers” installed. These writers are components that ensure application data is prepared for snapshotting (e.g., flushing buffers, quiescing databases).

The System Writer is a built-in VSS writer responsible for coordinating the backup of critical operating system components and state information. This includes files related to the system boot process, the registry, and potentially system servicing state. When a backup application requests a VSS snapshot, the VSS service instructs all registered writers, including the System Writer, to prepare their data. The System Writer identifies and lists the specific files and registry keys it needs to include in the snapshot metadata to ensure a restorable system state.

The Metadata Mismatch

The issue arises because the System Writer, when first initialized or snapshotting the system state before the servicing update modifies the servicing directory, includes the original paths to the amd64_installed and x86_installed files (e.g., C:\Windows\servicing\Version\10.0.14393.0\amd64_installed) in its metadata. This metadata is then stored as part of the VSS snapshot information.

When a third-party backup application performs a backup operation using a VSS snapshot created after the servicing update has been applied and the original files/folder are removed, it consults the VSS snapshot metadata. The metadata, however, still points to the old, non-existent path for these two files. The backup application, attempting to locate and back up these files from the shadow copy volume using the paths specified in the metadata, fails to find them. This results in “file not found” warnings or errors reported by the backup software.

The screenshot below conceptually illustrates the interaction flow, highlighting where the metadata discrepancy occurs.

mermaid graph TD A[Third-Party Backup Software] --> B{Request VSS Snapshot}; B --> C[VSS Service]; C --> D[VSS Writers (System Writer)]; D -- Pre-Update State --> E[System Writer Metadata (Includes old paths: \...\10.0.14393.0\files)]; E -- Stored in Snapshot --> F[VSS Snapshot]; A -- Backup from Snapshot --> G[Access Shadow Copy Volume]; G -- Consults Metadata --> F; F -- Provides old paths --> G; G -- Attempts to read files at old paths --> H[C:\Windows\servicing\Version\10.0.14393.0]; H -- After Update: Folder is Empty --> I{Files Not Found!}; I --> A -- Reports Warnings --> J[Backup Log (ANS4251W)]; D -- Post-Update State --> K[System Writer Metadata (Includes new paths: \...\10.0.14393.1051\files)]; K -- Future Backups? --> A;
Conceptual diagram showing VSS interaction and the point of failure due to outdated metadata.

This issue is particularly noticeable with third-party backup solutions because they often perform detailed file-level checks based on the VSS metadata provided by the writers. In contrast, native Windows Server Backup might handle System State backups differently, potentially not strictly relying on the same granular file lists from the VSS writer metadata for these specific files, or perhaps it uses a different method for accessing or verifying these particular system components.

Diagnosing the Problem with diskshadow

Administrators can confirm this specific VSS metadata issue using the built-in diskshadow utility. diskshadow is a command-line tool that exposes the functionality of the VSS service. By listing the details of the VSS writers, one can inspect the files that each writer is attempting to back up as part of its component list.

The process involves:
1. Opening a command prompt as administrator.
2. Running diskshadow /l output.txt to start the utility and log output.
3. At the diskshadow prompt, typing list writers detailed.
4. Waiting for the command to complete (this can take a few minutes).
5. Typing exit to leave diskshadow.
6. Opening the output.txt file in a text editor.

When examining the output for the “System Writer”, you would look for the section detailing the files and components managed by this writer. In the affected scenario, you would find entries similar to these within the System Writer’s file list, even though the files were moved or deleted:

- File List:
  Path: \\?\globalroot\device\harddiskvolumeshadowcopyX\windows\servicing\version\10.0.14393.0
  Files:
    amd64_installed
    x86_installed

(Note: harddiskvolumeshadowcopyX represents the specific shadow copy device path used during the list writers detailed command execution, which changes each time).

This diskshadow output confirms that the System Writer’s registered component list, captured within the VSS metadata available via the snapshot interface, still includes references to files located in the old 10.0.14393.0 path, even though those files no longer exist there on the live file system (or the shadow copy derived from it after the update).

The backup software warnings, such as the example provided:

Date/TimeANS4251W System Writer file '\\?\globalroot\device\harddiskvolumeshadowcopy3\windows\servicing\version\10.0.14393.0\amd64_installed': not found.
Date/TimeANS4251W System Writer file '\\?\globalroot\device\harddiskvolumeshadowcopy3\windows\servicing\version\10.0.14393.0\x86_installed': not found.

directly correlate with the file paths listed in the outdated System Writer metadata revealed by diskshadow. The backup application is essentially reporting that it cannot find the files that the VSS metadata told it should be present at that specific location within the snapshot.

The Resolution: Creating Placeholder Files

Fortunately, the solution to this specific issue is straightforward and non-disruptive. The VSS System Writer’s metadata expects the presence of files at the specified paths. The content of these files is not the issue; their mere existence at the expected location is what satisfies the check performed by the backup software based on the VSS metadata.

The resolution involves manually recreating the folder structure and the two specific files that the System Writer metadata is still referencing. These files do not need to contain any specific data; empty files are sufficient.

The steps are as follows:

  1. Open Command Prompt or PowerShell as an administrator.

  2. Create the specific version directory if it no longer exists. The path is C:\Windows\servicing\Version\10.0.14393.0.

    • In Command Prompt:
      mkdir C:\Windows\servicing\Version\10.0.14393.0
    • In PowerShell:
      New-Item -Path "C:\Windows\servicing\Version\10.0.14393.0" -ItemType Directory -Force

      (The -Force parameter in PowerShell ensures the directory is created even if parent directories don’t exist, though C:\Windows\servicing\Version should already exist).

  3. Create the two placeholder files within the newly created or existing 10.0.14393.0 directory. The files should be named amd64_installed and x86_installed.

    • In Command Prompt:
      type nul > C:\Windows\servicing\Version\10.0.14393.0\amd64_installed
type nul > C:\Windows\servicing\Version\10.0.14393.0\x86_installed
    • In PowerShell:
      New-Item -Path "C:\Windows\servicing\Version\10.0.14393.0\amd64_installed" -ItemType File -Force
New-Item -Path "C:\Windows\servicing\Version\10.0.14393.0\x86_installed" -ItemType File -Force

      (Again, -Force ensures creation even if the target path didn’t exist, but we created it in step 2).

After executing these commands, verify that the files C:\Windows\servicing\Version\10.0.14393.0\amd64_installed and C:\Windows\servicing\Version\10.0.14393.0\x86_installed now exist and are likely 0 KB in size.

Verifying the Fix

To confirm that the issue is resolved, you can perform the following checks:

  1. Run diskshadow again: Repeat the diskshadow /l output.txt and list writers detailed steps. Examine the output for the System Writer. The entries for amd64_installed and x86_installed should still appear under the 10.0.14393.0 path, but their presence on the file system (even as placeholders) means they can now be located during a backup process that relies on this outdated metadata.
  2. Run a backup job: Execute a backup job using your third-party backup application that previously reported the warnings. Monitor the backup logs. The ANS4251W warnings or equivalent “file not found” messages for these specific files should no longer appear.

If the warnings persist after creating the files, double-check that the files were created at the exact paths specified by the backup error message and the diskshadow output. Ensure there are no typos in the folder or file names.

Why Windows Server Backup Isn’t Affected

It is worth noting why the native Windows Server Backup (WSB) tool does not exhibit this behavior. While WSB also utilizes VSS for System State backups, its mechanism for backing up system components might differ from generic file-level backups performed by some third-party tools based strictly on VSS writer file lists. WSB might have a more integrated or hardcoded understanding of system components, or its System State backup process might not perform granular file-existence checks for these specific servicing files in the same manner as third-party agents. This highlights that the issue lies specifically in how some third-party tools interpret and use the potentially outdated file path metadata provided by the VSS System Writer post-update.

Broader Implications and Best Practices

While the resolution for this specific issue is simple, it underscores potential complexities that can arise when applying operating system updates, especially servicing stack updates that modify core system directories. Compatibility between operating system components, VSS, and third-party applications like backup software is critical.

Managing Updates and Backups

  • Testing: Always test significant operating system updates, including servicing stack updates, in a lab or staging environment before deploying them to production servers, especially those running critical applications and third-party agents (like backup software).
  • Vendor Communication: Stay informed about compatibility notes and recommended update sequences from your third-party backup software vendor. They may issue patches or specific guidance related to Windows updates.
  • Monitoring: After applying any update, monitor system event logs (Application, System, and specifically the VSS event logs), as well as backup job logs, for any new errors or warnings. Early detection of issues like this can prevent incomplete backups or restore failures.
  • VSS Health: Regularly check the health of VSS writers using vssadmin list writers. Ensure all necessary writers are stable and without errors. Issues with VSS writers are a common cause of backup problems.
  • Documentation: Document the steps taken to resolve issues like this. Having a record can be invaluable if the issue recurs after future updates or on other servers.

Potential Future Considerations

It is possible that future Windows Server 2016 servicing updates might address the root cause of this problem by ensuring the System Writer metadata is properly updated or no longer includes references to files in old servicing version directories. However, until such an update is confirmed, the placeholder file resolution remains the effective workaround. Administrators should be aware that this issue could potentially reappear with future servicing stack updates if the underlying mechanism isn’t permanently altered.

Conclusion

The appearance of “file not found” warnings in third-party backup applications after installing servicing updates on Windows Server 2016, specifically related to files within the C:\Windows\servicing directory, is a known issue stemming from a discrepancy between the VSS System Writer metadata and the actual file system state post-update. The simple act of creating empty placeholder files at the locations referenced by the outdated VSS metadata effectively resolves these warnings by satisfying the backup software’s file check.

While this workaround is effective, it highlights the intricate dependencies within a server environment and the importance of careful planning, testing, and monitoring when applying system updates. By understanding the role of VSS and its writers, and being prepared to diagnose and address metadata inconsistencies, administrators can ensure the continued reliability of their backup strategies.

Have you encountered this specific issue or similar VSS-related problems after applying Windows Server updates? How did you diagnose and resolve it? Share your experiences and insights in the comments below to help others in the community navigate these complex scenarios.

Post a Comment