Troubleshooting: RemoteFX USB Scanner Redirection Fails in Windows Server

Table of Contents

This article addresses a specific issue encountered when using RemoteFX USB redirection in a Windows Server environment, where a redirected USB scanner fails to open or initiate a scan. This problem is primarily linked to how the scanner’s driver interacts with the Windows session architecture in a multi-user remote desktop scenario. Understanding the underlying technical details is crucial for diagnosing and resolving this issue effectively.

Symptoms of the Issue¶

Users may encounter this problem when attempting to use a USB scanner redirected via RemoteFX from a client computer to a Windows Server running Remote Desktop Services (RDS). The typical scenario involves a client machine, such as one running Windows 8.1 Pro x64, connecting to a server like Windows Server 2012 or Windows Server 2012 R2 Datacenter configured with the Remote Desktop Session Host role and the Desktop Experience feature.

After successfully establishing an RDP session with RemoteFX USB redirection enabled and verifying the scanner appears in the server’s “Devices and Printers”, attempting to initiate a scan from the server session results in failure. Specifically, when right-clicking the scanner icon and selecting “Start scan” or using scanner software, the scan process does not begin. Crucially, no explicit error message is displayed to the user, leaving them unaware of the underlying cause of the failure. The device appears present but is functionally inaccessible.

Technical Explanation: The Role of Session Isolation¶

The root cause of this issue lies in the design of RemoteFX USB redirection in Windows Server 2012 and later versions, particularly in comparison to its implementation in client operating systems like Windows 7 SP1. In a multi-user environment like RDS on Windows Server, devices redirected via RemoteFX USB redirection are assigned to the specific user session they were redirected into. This mechanism provides essential session isolation, ensuring that a user in one RDP session cannot access devices redirected into another user’s session, enhancing security and privacy.

Services and device drivers traditionally often run within a dedicated, non-interactive session known as “Session 0”. This session is isolated from the interactive user sessions (Session 1, Session 2, etc.) where users log in and run applications. When a scanner driver, potentially running as part of a service in Session 0, or user-mode components interacting with such a service, attempts to access the redirected scanner device (which resides in the user’s specific interactive session), the Windows I/O Manager enforces this session isolation.

The mechanism by which applications or services open handles to devices is the CreateFile function. By default, the I/O Manager in multi-session Windows operating systems will deny access to devices residing in a user session if the calling process (like a service in Session 0) does not explicitly indicate that it is aware of and capable of handling sessions. This is where the FILE_FLAG_SESSION_AWARE flag becomes critical.

When a driver or associated service calls CreateFile to access the redirected scanner device without setting the FILE_FLAG_SESSION_AWARE flag, the system interprets this as an attempt by a non-session-aware process (likely running outside the target user session) to access a session-specific device. Consequently, the CreateFile call fails internally with an E_ACCESSDENIED error code (0x80070005). Although this specific error code is not typically shown to the end-user, it is the underlying reason why the scanner operation fails silently.

Resolution for the RemoteFX Scanner Issue¶

The primary and most effective resolution for this problem involves updating the scanner’s driver. Since the issue stems from the driver’s implementation of device access using the CreateFile function without the necessary session awareness flag, the fix must come from the driver vendor.

For Scanner Manufacturers/Driver Developers: The fix requires modifying the driver or any associated service components that interact directly with the scanner hardware. When calling the CreateFile function to open a handle to the redirected USB scanner device, the dwFlagsAndAttributes parameter must include the FILE_FLAG_SESSION_AWARE flag. This explicitly signals to the Windows I/O Manager that the process is capable of handling device access in a multi-session environment. Implementing this change requires recompiling and releasing an updated driver package.

For End-Users and System Administrators: The recommended course of action is to obtain and install the latest version of the scanner driver provided by the scanner manufacturer. It is essential to ensure that the driver version specifically addresses compatibility with Remote Desktop Services and multi-session environments like Windows Server 2012 R2 and later. Contacting the scanner manufacturer’s support or checking their website for driver updates compatible with Windows Server RDS is the necessary step. Generic drivers provided by Microsoft or included with the operating system may not have this specific fix implemented if the original vendor driver did not incorporate it.

Prerequisites for RemoteFX USB Redirection¶

To successfully utilize RemoteFX USB redirection in the first place, ensuring the environment meets the basic prerequisites is vital. While not the cause of this specific scanning issue, incorrect setup can prevent redirection entirely.

On the client machine (e.g., Windows 8.1):
- The client operating system must support RemoteFX USB redirection (Windows 7 SP1 Enterprise/Ultimate, Windows 8/8.1 Enterprise/Pro x64, Windows 10 Enterprise/Pro x64).
- The RDP client version must support RemoteFX (included with compatible OS versions).
- RemoteFX must be enabled, which typically requires a DirectX 11-capable GPU on the client, though some basic redirection might work without it for non-graphics-intensive devices.
- The Group Policy setting “Allow RDP redirection of other supported RemoteFX USB devices from this computer” must be enabled under Computer Configuration\Administrative Templates\Windows Components\Remote Desktop Services\Remote Desktop Connection Client\RemoteFX USB Device Redirection.

On the server machine (e.g., Windows Server 2012 R2):
- The server must have the Remote Desktop Session Host role service installed.
- The Desktop Experience feature should be installed (often required for full multimedia and device compatibility, including WIA/TWAIN scanning).
- The appropriate drivers for the USB device (in this case, the scanner) must be installed on the server within the RDP session environment. This is often done after redirecting the device for the first time, allowing Windows Update or provided driver packages to install the necessary software in the user’s session context.

Ensuring these prerequisites are met is a foundational step. If redirection fails entirely, these settings and hardware requirements should be checked before troubleshooting device-specific issues like the scanner access problem.

Expanding on Session Isolation and CreateFile¶

The concept of session isolation in Windows Server RDS is a critical security and stability feature. In older Windows versions (pre-Vista/Server 2008), Session 0 was also the session where the first user logged in. This architecture posed security risks because services running in Session 0 could potentially be exploited to gain access to user data or run malicious code in a privileged context.

With the introduction of Session 0 isolation, services were moved to a non-interactive session distinct from user sessions (Session 1, 2, etc., assigned dynamically). This separation prevents direct interaction between services and user applications and data without explicit mechanisms. Device access falls under this. When a device is redirected into a user’s session (e.g., Session 3), only processes running within Session 3 are normally allowed to open handles to it.

However, many device drivers and their associated user-mode components (often services) historically ran in Session 0. For these components to interact with devices in a user session, they need a way to indicate to the operating system that they are “session-aware” and have internal logic to handle device access requests originating from different user sessions. The FILE_FLAG_SESSION_AWARE flag serves this purpose. By setting this flag when calling CreateFile, the driver or service tells the I/O Manager, “I know this device might be in a different session, and I have the logic to handle that safely and correctly, including validating requests come from the device’s owning session.” If the flag is missing, the system assumes the older, non-session-aware behavior and denies access as a protective measure.

This design pattern requires driver developers to update their code to be compatible with modern multi-session environments. Older drivers written without consideration for Session 0 isolation and multi-user RDS may fail in unexpected ways, as seen with the scanner issue.

Troubleshooting Steps (Beyond the Driver)¶

While the driver update is the definitive fix for this specific issue, if you are troubleshooting RemoteFX USB redirection, other factors might be involved if the driver fix doesn’t immediately work or if other devices fail to redirect.

1. Verify Group Policy Settings: Double-check that the “Allow RDP redirection…” GPO is correctly applied on the client machine and is indeed enabled. Run gpresult /r on the client to confirm policy application.
2. Check Device Manager: On the server within the RDP session, open Device Manager. See if the redirected scanner appears correctly. Check for any warning triangles or errors indicating a driver problem on the server side. Even if the driver is installed, there might be installation issues within that specific session.
3. Test with Other USB Devices: Try redirecting a simpler USB device (like a basic USB drive or keyboard) to see if RemoteFX USB redirection is fundamentally working. If simpler devices redirect successfully, it further points towards a device-specific or driver-specific issue with the scanner.
4. Review Server Roles and Features: Confirm that the Remote Desktop Session Host role and Desktop Experience feature are correctly installed and configured on the server.
5. Check RDP Client Settings: Ensure the RDP client on the Windows 8.1 machine is configured to redirect “Other supported RemoteFX USB devices”. This setting is usually found under the “Local Resources” tab, within the “More…” options under “Local devices and resources”.
6. Antivirus/Security Software: Occasionally, security software on either the client or server can interfere with device redirection. Temporarily disabling such software for testing purposes (in a controlled environment) might help rule this out, though this should be done cautiously.

If after verifying these steps and installing an updated, session-aware driver the issue persists, gathering detailed logs (e.g., RDP event logs, system event logs, driver logs if available) would be the next step for deeper analysis.

Comparing RemoteFX USB Redirection Types¶

RemoteFX provides different levels of device redirection:

• High-Level Device Redirection: This is the most common type for many devices (like printers, smart cards). The client OS uses a virtual channel to send commands to a virtual device driver on the server. The server OS then processes these commands. This works well for standard device types but might not expose all features of complex or unique devices.
• Port Redirection: Redirects serial or parallel ports.
• Drive Redirection: Redirects client drives as mapped network drives.
• RemoteFX USB Redirection: This method virtualizes the USB port itself. This allows the server OS to interact with the USB device at a lower level, often enabling the use of the native device drivers installed on the server. This is particularly useful for devices like scanners, webcams, and specialized industrial devices that require full driver functionality and features. This is the type affected by the issue described in this article.

RemoteFX USB redirection is generally preferred for devices that don’t work well with high-level redirection or require specific driver features. However, as this issue demonstrates, it also introduces complexities related to device driver compatibility in a multi-session environment.

Driver Development Considerations for Multi-Session Environments¶

For driver developers targeting Windows Server RDS environments, ensuring compatibility requires specific coding practices. Beyond setting the FILE_FLAG_SESSION_AWARE flag, drivers and associated services should:

• Properly manage device handles, ensuring handles opened in one session are not misused by processes in another.
• Implement robust session change notifications if needed to track device availability as users connect and disconnect.
• Avoid assumptions about the session context in which their code is running, especially for components that might run in Session 0.
• Test thoroughly in actual RDS environments with multiple concurrent sessions.

These practices are essential for creating stable and reliable drivers for devices intended for use with RemoteFX USB redirection on Windows Server.

The Criticality of Updated Drivers¶

This troubleshooting scenario highlights a common challenge in remote desktop environments: the reliance on third-party device drivers being designed and tested for such complex setups. While Microsoft provides the RemoteFX framework and session isolation features, the responsibility for ensuring a device’s driver correctly interacts with these features lies with the device manufacturer. For devices like scanners, which often rely on user-mode services and specific driver interfaces (like TWAIN or WIA), a driver update addressing session awareness is not merely a performance tweak but a fundamental requirement for functionality in a multi-session RDS environment.

If the scanner manufacturer confirms they do not have a driver compatible with Windows Server RDS or multi-session environments that includes the necessary FILE_FLAG_SESSION_AWARE fix, the specific scanner model may simply be incompatible with RemoteFX USB redirection on these platforms. In such cases, exploring alternative methods (if available and supported by the device) or using a scanner model known to be compatible with RDS might be necessary.

Conclusion¶

The failure of a RemoteFX USB redirected scanner to open or scan in Windows Server 2012 R2 (and similar multi-session environments) is typically caused by the scanner’s driver not correctly handling session isolation. The driver attempts to access the device, which resides in the user’s isolated session, from a potentially non-session-aware context (like a service in Session 0) without using the FILE_FLAG_SESSION_AWARE flag in the CreateFile function. This leads to an E_ACCESSDENIED error and prevents the scanner from functioning.

The definitive fix requires an updated driver from the scanner manufacturer that has been modified to set the FILE_FLAG_SESSION_AWARE flag and correctly handle device access in a multi-session RDS environment. Users experiencing this issue should contact their scanner vendor for the latest, compatible driver. Ensuring all RemoteFX and RDS prerequisites are met on both client and server is also important for overall redirection functionality.

Have you encountered this specific issue with RemoteFX USB scanner redirection? What steps did you take, and did obtaining an updated driver from the manufacturer resolve the problem for you? Share your experiences and insights in the comments below.