Troubleshooting VMware Driver Installation Errors on Windows Server 2008 R2 SP1

Table of Contents

Virtual machines running on VMware vSphere, specifically those using the Windows Server 2008 R2 Service Pack 1 operating system, may encounter difficulties during driver installation or updates. This common issue prevents newly added hardware, virtual devices, or updated components from functioning correctly within the guest operating system. The problem typically manifests when attempting to install or update device drivers through the standard Windows Device Manager interface.

Investigation into such failures often points towards a specific configuration conflict involving the Telnet Server service. When this service is configured in a non-default manner regarding its logon account and interaction settings, it can inadvertently interfere with critical system processes. This interference can disrupt operations necessary for the successful installation of device drivers, leading to errors and unresolved hardware in the virtual machine environment. Understanding the underlying cause and implementing the correct configuration adjustments is crucial for resolving this driver installation impediment.

Symptoms of Driver Installation Failure¶

When this particular issue occurs, the primary symptom is the inability to successfully install or update drivers for devices within the virtual machine. This is most often attempted via the Windows Device Manager. The process will fail, and the user will be presented with a generic error message indicating a problem occurred during installation.

The specific error message displayed is:

Windows found driver software for your device but encountered an error while attempting to install it.

While this message is unspecific, further investigation into system logs can provide a clearer picture of the failure point. The setupapi.dev.log file, located in the %windir%\inf directory, is a critical resource for diagnosing device installation issues. Analysis of this log file reveals more detailed information about the driver installation process and the specific error encountered by the system’s setup components.

Within the setupapi.dev.log, entries resembling the following pattern are indicative of this problem:

dvi: {Plug and Play Service: Device Install for PCI\VEN_15AD&DEV_07B0&SUBSYS_07B015AD&REV_01\FF565000B37984FE00}
ump: Creating Install Process: DrvInst.exe <DateTime>
ump: Server install process exited with code 0xc0000142 <DateTime>
ump: {Plug and Play Service: Device Install exit(c0000142)}
ndv: Device Install failed for new device...installing NULL driver.
dvi: {Plug and Play Service: Device Install for PCI\VEN_15AD&DEV_07B0&SUBSYS_07B015AD&REV_01\FF565000B37984FE00}

The key piece of information here is the exit code 0xc0000142. This specific error code, STATUS_DLL_INIT_FAILED, indicates that a dynamic-link library (DLL) initialization routine failed. In the context of device installation via DrvInst.exe, this suggests that a critical component or dependency required by the driver installation process could not be properly initialized or accessed, likely due to permission issues or conflicts within the system’s session environment.

Beyond the setupapi.dev.log, reviewing the Windows Event Viewer can also yield valuable clues. Checking the System and Application logs for errors or warnings timestamped around the time of the failed driver installation attempt might reveal related service control errors, permission denied messages, or crashes associated with system processes. Together, the error message, the specific log entries in setupapi.dev.log with the 0xc0000142 code, and potential Event Viewer entries solidify the symptoms pointing towards this specific configuration-related issue.

Cause: Service Account Configuration Conflict¶

The root cause of this driver installation failure stems from an incorrect or non-standard configuration of the Telnet Server service (TlntSvr) within the Windows Server 2008 R2 SP1 guest operating system. Specifically, the issue arises when the Telnet Server service is configured to log on using the Local System account AND the option “Allow service to interact with desktop” is not selected. This configuration deviates from the service’s default settings and creates an environment that conflicts with other essential system processes.

Understanding Service Logon Accounts¶

Windows services run under specific security principals called service accounts. Different service accounts have varying levels of permissions and capabilities.
* Local Service: This account has minimal privileges on the local computer and presents anonymous credentials on the network. It’s suitable for services that don’t need to interact with system components at a high level or access network resources under a specific identity. This is the default account for the Telnet Server service.
* Network Service: Similar to Local Service, but presents the computer’s credentials on the network.
* Local System: This is a highly privileged account. It has extensive privileges on the local computer, acting as the computer’s identity on the network. It has the SE_TCB_NAME privilege (“Act as part of the operating system”), giving it significant power. Services running as Local System typically reside in Session 0, a non-interactive session.

The Role of Sessions and Window Stations¶

In Windows, interactive user sessions run in sessions other than Session 0 (e.g., Session 1, 2, etc., depending on the console or RDP connection). System processes and services running without the ability to interact with the desktop are typically isolated in Session 0. A “window station” is a secure object that contains a clipboard, a set of atoms, and a group of desktop objects. Services running in Session 0 operate within a specific window station (often Service-0x...).

The “Allow service to interact with desktop” option, when checked for the Local System account, allows the service to run in the interactive user’s session (Session 1, 2, etc.) and potentially display a user interface. Without this option checked, a service running as Local System is confined to Session 0.

How the Conflict Occurs¶

The Telnet Server service, during its initialization phase, performs certain permission modifications related to the window station in which it is running. When configured to run as Local System without desktop interaction, it operates within a window station in Session 0. The specific permission changes it makes can inadvertently affect other processes also running within Session 0 under the Local System account that also do not interact with the desktop.

One such critical process is the driver installation process (DrvInst.exe), which is invoked by the Plug and Play service (often running as Local System in Session 0) to install new device drivers. This process relies on specific permissions and environmental conditions within its window station to initialize correctly and access necessary resources (like DLLs or configuration files). If the Telnet Server service has altered the permissions or state of the shared window station in Session 0 in an incompatible way, the driver installation process (DrvInst.exe) can fail its initialization routine, leading to the 0xc0000142 error code seen in the logs. This security-related isolation mechanism, intended to protect Session 0, backfires when one service (Telnet Server) modifies the shared session environment in a way that negatively impacts another service (driver installation).

Workaround to Resolve the Issue¶

The recommended and most straightforward workaround for this issue is to revert the Telnet Server service configuration to its default, secure settings. By changing the service’s logon account, you prevent it from causing the permission conflicts in Session 0 that interfere with driver installation.

Step 1: Revert Telnet Server Service Logon¶

The primary solution is to configure the Telnet Server service (TlntSvr) to use its default logon account, which is Local Service. This account has lower privileges and operates differently, avoiding the Session 0 window station conflict experienced with the Local System account configuration.

To change the service logon configuration, follow these steps:

1. Open the Services Microsoft Management Console (MMC). You can do this by typing services.msc in the Run dialog box (Windows Key + R) or by navigating through Server Manager -> Configuration -> Services.
2. Locate the “Telnet Server” service in the list.
3. Right-click the “Telnet Server” service and select “Properties”.
4. Go to the “Log On” tab.
5. Select the “Local Service account” radio button.
6. Ensure that the “Allow service to interact with desktop” checkbox is unchecked. (While this option caused the issue when combined with Local System, it should also be unchecked for Local Service as this account generally cannot interact with the desktop).
7. Click “Apply” and then “OK”.
8. Restart the Telnet Server service. Even better, restart the entire virtual machine to ensure all related processes are reset.

After applying this change and restarting the VM, attempt the driver installation again via Device Manager. In most cases, this action alone will resolve the 0xc0000142 error and allow drivers to install successfully.

Step 2: Reinstall VMware Tools (If Necessary)¶

If changing the Telnet Server service logon does not immediately resolve the driver installation problem, or if you notice that crucial VMware virtual hardware drivers (which are part of VMware Tools) are the ones failing to install or update, the issue might be compounded by a corrupted or problematic installation of VMware Tools itself. In this scenario, a forced removal and reinstallation of VMware Tools can often clear up underlying issues preventing driver installation.

You might encounter difficulties uninstalling VMware Tools via standard methods (Add/Remove Programs) if the dependency on core system components is corrupted. VMware provides a command-line method for a forced cleanup during installation.

Here are the steps to perform a forced cleanup and reinstallation of VMware Tools:

1. Obtain the setup files for the latest compatible version of VMware Tools for your specific VMware product (vSphere, Workstation, etc.) and the guest OS (Windows Server 2008 R2 SP1). This is typically available by mounting the VMware Tools installer ISO from the vSphere client or hypervisor interface (VM -> Guest -> Install/Upgrade VMware Tools).
2. Open an elevated Command Prompt window. Click Start, type cmd in the search box, right-click “Command Prompt” and select “Run as administrator”.
3. Navigate to the directory containing the VMware Tools setup files. If you mounted the ISO, this will be the CD-ROM drive letter (e.g., D:\). Type the drive letter followed by a colon and press Enter (e.g., D:).

4. Execute the setup command with the /c switch to force cleanup of previous installations.

   • For 32-bit Windows Server 2008 R2 SP1: Type setup /c and press Enter.
   • For 64-bit Windows Server 2008 R2 SP1: Type setup64 /c and press Enter.

   The /c switch instructs the installer to perform a comprehensive cleanup, removing registry entries and files associated with a previous VMware Tools installation even if the standard uninstaller fails.
   5. Allow the cleanup process to complete. This may take a few minutes and might require a reboot.
   6. Once the cleanup is finished (and after a reboot if prompted), proceed to install the latest version of VMware Tools normally. Mount the ISO again if needed, and run setup.exe (or setup64.exe) without any switches.
   7. Follow the on-screen prompts to complete the installation. Choose a “Typical” or “Complete” installation.
   8. Perform a final restart of the virtual machine after the VMware Tools installation is finished.

After successfully reinstalling VMware Tools, verify that the Telnet Server service logon setting is still correctly configured for the Local Service account as per Step 1. Then, attempt to install the troublesome drivers again. The combination of the correct Telnet service configuration and a fresh VMware Tools installation should resolve the issue.

Verifying Driver Installation¶

Once you have applied the workarounds, it is essential to verify that the drivers have been installed correctly.

1. Open Device Manager. You can do this by typing devmgmt.msc in the Run dialog box (Windows Key + R) or via Server Manager -> Diagnostics -> Device Manager.
2. Check the device that previously had the installation issue. It should no longer appear with a yellow exclamation mark or as an “Unknown device”.
3. Right-click the device and select “Properties”.
4. On the “General” tab, the “Device status” should read “This device is working properly.”
5. On the “Driver” tab, you can verify the driver provider, date, and version to ensure the correct driver is installed.

If the device still shows issues, double-check the Telnet Server service configuration and ensure VMware Tools were installed correctly and the VM was rebooted afterwards. Review the setupapi.dev.log and Event Viewer again for any new errors that might point to a different underlying problem.

Advanced Troubleshooting Considerations¶

While the Telnet Server service configuration is the specific cause addressed here, other factors can sometimes impede driver installation in a virtual machine. If the above steps do not work, consider these additional points:

• VMware Compatibility: Ensure the virtual machine’s hardware version is compatible with the version of VMware Tools you are trying to install and the version of the hypervisor you are running.
• Windows Updates: Verify that the Windows Server 2008 R2 SP1 guest OS has all recommended and critical updates installed. Missing updates can sometimes lead to dependency issues for driver installations.
• Driver Source: Ensure you are using the correct and reliable source for the drivers (e.g., VMware Tools for virtual hardware, or vendor-provided drivers for specific pass-through devices).
• Security Software: Temporarily disabling antivirus or other security software could help rule out interference, though this should be done with caution and only for troubleshooting.
• System File Checker: Run sfc /scannow from an elevated command prompt to check for and repair corrupted Windows system files that might affect driver installation.

Data Collection for Microsoft Support¶

If the issue persists after attempting the workarounds and reviewing advanced considerations, and you require assistance from Microsoft Support, they will likely need diagnostic information from your system. The recommended tool for collecting relevant data for deployment-related issues on Windows is the Troubleshooting Scripts (TSS) tool.

You can gather information by following the steps mentioned in the Microsoft documentation page titled “Gather information by using TSS for deployment-related issues”. Be prepared to provide the collected logs to the support engineer to facilitate their investigation.

Third-Party Information Disclaimer¶

This article discusses products manufactured by companies independent of Microsoft. While this information is provided to help resolve a specific compatibility issue observed with these third-party products, Microsoft makes no warranty, implied or otherwise, regarding the performance or reliability of these non-Microsoft products. Users should refer to the documentation and support resources provided by the respective third-party vendors (like VMware) for detailed information about their products.

Resolving driver installation issues in virtual environments requires careful attention to configuration details and system logs. By understanding the interaction between Windows services, logon accounts, and system sessions, administrators can effectively diagnose and fix problems like the one caused by the misconfigured Telnet Server service on Windows Server 2008 R2 SP1 VMs in VMware.

Have you encountered this specific issue, or similar driver installation problems in virtual machines? Share your experiences or alternative solutions in the comments below!