Troubleshooting BITS Transfer Failures in Virtual Machine Manager

Table of Contents

System Center Virtual Machine Manager (VMM) is a management solution for virtualized data centers, providing a unified management experience across Hyper-V, VMware, and Citrix XenServer virtualization platforms. A critical function of VMM is the movement of data, such as virtual machine files, templates, and ISO images, between the VMM server, library servers where these assets are stored, and the virtualization hosts where virtual machines run. This data transfer process often relies on the Background Intelligent Transfer Service (BITS). BITS is designed to reliably transfer files in the background, utilizing idle network bandwidth and automatically resuming transfers if interrupted. While efficient, BITS can encounter issues that prevent successful data movement, leading to operational problems within the VMM environment.

When BITS transfers fail in VMM, it can manifest in various ways, severely impacting the ability to deploy new virtual machines, migrate existing ones, or update host configurations. These failures typically present as errors within the VMM console, often accompanied by cryptic error codes. Identifying the root cause of these transfer failures is crucial for maintaining a healthy and functional virtualized infrastructure managed by System Center VMM. This article delves into a common cause of such failures and provides a comprehensive resolution involving network port configuration.

Symptoms of BITS Transfer Failure¶

Users interacting with the System Center Virtual Machine Manager console or performing operations through PowerShell may encounter errors indicating a failure during data transfer. These issues can range from transfers being unusually slow to outright failure, often with specific error codes pointing towards underlying communication problems. The BITS service, while robust, is susceptible to network configuration issues or conflicts with other services running on the involved servers.

Common error messages observed during BITS transfer failures in VMM include:

Error (12700)
<path to vhd on destination host> 'The file or directory is corrupted and unreadable' (0x80070570)

This error, despite suggesting file corruption, often indicates a failure in the transfer process itself, preventing the destination file from being written correctly or completely. The underlying issue is not necessarily the file’s integrity on the source but the inability to transfer it reliably to the destination. This can halt critical operations like deploying a virtual machine from a template or migrating storage.

Another frequently encountered error is:

Error (2927)
A Hardware Management error has occurred trying to contact server servername.contoso.com. (Unknown error (0x8033811e))

While labeled a “Hardware Management error,” Error 2927 is a general communication failure that can occur when VMM attempts to interact with a host or library server. In the context of transfers, this error signifies that VMM could not establish or maintain the necessary communication channel, often involving BITS, to complete the operation. The specific error code 0x8033811e points towards a VMM service communication issue.

A third error code commonly associated with BITS transfer problems is:

Error (2912)
An internal error has occurred trying to contact an agent on the servername.contoso.com server. (Unknown error (0x80072f06))

Error 2912 indicates an issue with VMM’s ability to communicate with the VMM agent running on a target server (host or library server). This agent facilitates many VMM operations, including coordinating BITS transfers. The 0x80072f06 error code translates to “The server could not establish a connection with the client” or a similar network-related connection failure, again pointing towards underlying communication problems that prevent BITS from functioning correctly.

These symptoms typically manifest when performing actions that require moving large amounts of data, such as:
* Creating a new virtual machine from a template stored on a library server.
* Cloning an existing virtual machine.
* Migrating a virtual machine’s storage from one location to another.
* Importing or exporting virtual machine files or templates.
* Copying files to or from a library share.

Understanding these error patterns is the first step in diagnosing the problem. When these errors consistently appear during data transfer operations, a network communication conflict, particularly involving the ports used by BITS, should be suspected.

Cause of BITS Transfer Failures¶

The root cause of the BITS transfer failures described above often stems from a network port conflict. Background Intelligent Transfer Service, as used by System Center Virtual Machine Manager for data transfers, utilizes a default network port to facilitate communication between the VMM server, library servers, and virtualization hosts. By default, BITS is configured to use TCP port 443.

The choice of port 443 for BITS transfers presents a significant challenge because this port is the standard, well-known port for HTTPS (Hypertext Transfer Protocol Secure) traffic. HTTPS is widely used by secure websites and countless applications that rely on encrypted web communication. This means that if any other service or application running on the V VMM server, a library server, or a virtualization host is actively using port 443, it can interfere with or completely block BITS from using the same port for its transfers.

Common culprits for port 443 usage include:
* Internet Information Services (IIS) hosting a secure website.
* Other management agents or services that use HTTPS for communication or reporting.
* VPN clients or servers configured to use port 443 to bypass firewalls.
* Certain monitoring agents.
* Applications that embed web servers for administration or functionality.

When another service binds to port 443 on a server involved in a VMM transfer, BITS cannot establish its connection on that port. This conflict prevents the transfer from initiating or completing successfully, leading to the errors observed in the VMM console. The operating system allows only one process to listen on a specific IP address and port combination at a time. Therefore, if HTTPS traffic from another application is claiming port 443, BITS is effectively shut out.

The interaction between VMM components during a transfer can be visualized as follows:

```mermaid
graph LR
A[VMM Server] – Initiates Transfer → B[Library Server]
B – BITS Transfer (Default Port 443) → C[Virtualization Host]
A – Initiates Transfer → C[Virtualization Host]
A – BITS Transfer (Default Port 443) → C[Virtualization Host]
C – BITS Transfer (Default Port 443) → B[Library Server]

C -- Other Service (e.g., IIS) --> Internet[Internet/Intranet]
C -- Other Service using 443 --> D[Another Server]

 subgraph Problem Area
    B[Library Server] -- Conflict on Port 443 --> B_Conflict[(Other Service using 443)]
    C[Virtualization Host] -- Conflict on Port 443 --> C_Conflict[(Other Service using 443)]
end

B_Conflict -. Blocks VMM/BITS .-> C
C_Conflict -. Blocks VMM/BITS .-> B
B_Conflict -. Blocks VMM/BITS .-> A
C_Conflict -. Blocks VMM/BITS .-> A

```

This diagram illustrates how the VMM server, library servers, and hosts communicate, highlighting that any of these machines having another service listening on port 443 can disrupt BITS transfers between any of the involved parties. The resolution focuses on changing the port BITS uses specifically for VMM transfers to circumvent this common conflict.

It is important to note that this port conflict typically affects the data transfer channel used by BITS, not necessarily the primary communication channel between the VMM server and its agents, which usually occurs on different ports (e.g., RPC ports, WinRM ports). However, since data transfer is a fundamental operation, its failure has a significant impact on VMM functionality.

Resolution: Configuring a Different BITS Port¶

The most effective way to resolve this port conflict is to configure System Center Virtual Machine Manager to use a different, non-conflicting port for BITS transfers. This involves changing the default BITS port setting within VMM itself. Once configured, VMM will instruct the BITS service on all managed hosts and library servers to use the new port when performing transfers orchestrated by VMM.

Before implementing this change, it is crucial to select a new port number carefully. The chosen port must meet the following criteria:
1. Not in Use: The port must not be currently used by any other application or service on the VMM server, any library servers, or any virtualization hosts managed by VMM.
2. Not Blocked by Firewall: The chosen port must be open bidirectionally (inbound and outbound) on the firewalls between the VMM server, library servers, and all virtualization hosts. This includes Windows Firewall on the servers themselves and any external network firewalls.
3. Within Range: The port number should be within the valid TCP port range (1-65535), and preferably within the range of registered ports (1024-49151) or dynamic/private ports (49152-65535) to avoid conflicting with other well-known services (ports 1-1023). A common practice is to choose a port in the higher dynamic range.

Step 1: Identify an Available Port¶

To identify a suitable port, you can check for currently listening ports on the servers involved in VMM transfers (VMM server, library servers, hosts). The netstat command-line utility is invaluable for this purpose.

Open Command Prompt or PowerShell as administrator on each relevant server and run the following command:

netstat -aon | findstr LISTENING

This command lists all active connections and listening ports (-a), displays port numbers numerically (-n), and includes the Process ID (PID) associated with each connection/port (-o). The findstr LISTENING filters the output to show only ports that a process is actively listening on.

Review the output to see which ports are in use, particularly focusing on ports in the 1024-65535 range. Look for a port number that does not appear in the output on any of the servers. You can also use tools like TCPView from Sysinternals (Microsoft) for a more graphical view of port usage and associated processes.

For example, if you see output showing processes listening on 443, 80, 5985, 5986, etc., you would avoid those ports. A port like 8500, 8501, or something in the 5xxxx range might be good candidates, provided they are not listed as already in use.

Let’s check for port 443 specifically:

netstat -aon | findstr :443

If this command returns a result showing a process ID listening on 0.0.0.0:443 or the server’s IP address on port 443, then port 443 is indeed in use by another service, confirming the likely cause of the BITS issue. Note down the PID and use Task Manager (Details tab) or tasklist /svc /FI "PID eq <PID>" to identify the process using the port.

Step 2: Configure the New BITS Port in VMM¶

The recommended method for changing the BITS port in VMM is using the VMM PowerShell module. This provides a clean and supported way to modify the VMM configuration.

Launch the Virtual Machine Manager command shell (or a standard PowerShell session with the VMM module loaded) as an administrator.

Connect to the VMM server if running the command remotely:

# Replace 'YourVMMServerName' with the actual VMM server name
# This step is optional if running directly on the VMM server
Get-SCVMMServer -ComputerName "YourVMMServerName"

Now, set the new BITS proxy port using the Set-SCVMMServer cmdlet. Replace <NewPort> with the port number you identified as available in Step 1.

# Replace <NewPort> with the desired port number, e.g., 8500
$vmmServer = Get-SCVMMServer
Set-SCVMMServer -VMMServer $vmmServer -BITSProxyPort <NewPort>

For example, to set the BITS port to 8500:

$vmmServer = Get-SCVMMServer
Set-SCVMMServer -VMMServer $vmmServer -BITSProxyPort 8500

After running this command, VMM will update its configuration and inform the VMM agents on managed hosts and library servers about the new port to use for future BITS transfers initiated by VMM. This change is dynamic and typically does not require restarting VMM services or the servers, although allowing a few minutes for the change to propagate is advisable.

Step 3: Configure Firewalls¶

Changing the BITS port in VMM is only effective if network firewalls allow traffic on the newly configured port between all relevant servers. You must ensure that the new port is open bidirectionally (inbound and outbound) on the Windows Firewall of the VMM server, all library servers, and all virtualization hosts managed by VMM. Any intermediate network firewalls must also permit traffic on this port between these servers.

Configuring Windows Firewall:

You can configure Windows Firewall using the graphical interface (Windows Firewall with Advanced Security) or via PowerShell. PowerShell is particularly useful for automating this across multiple servers.

Using PowerShell (Recommended for automation):

On each server (VMM server, library servers, hosts), run the following PowerShell commands as administrator. Replace <NewPort> with the port number you configured in VMM (e.g., 8500).

To create an inbound rule:

# Replace <NewPort> with the new BITS port number
$Port = <NewPort>
$RuleName = "VMM BITS Transfer (Inbound)"
$Action = "Allow"
$Protocol = "TCP"

New-NetFirewallRule -DisplayName $RuleName -Direction Inbound -LocalPort $Port -Protocol $Protocol -Action $Action -Profile Any -Description "Allows VMM BITS transfers on port $Port"

To create an outbound rule:

# Replace <NewPort> with the new BITS port number
$Port = <NewPort>
$RuleName = "VMM BITS Transfer (Outbound)"
$Action = "Allow"
$Protocol = "TCP"

New-NetFirewallRule -DisplayName $RuleName -Direction Outbound -LocalPort $Port -Protocol $Protocol -Action $Action -Profile Any -Description "Allows VMM BITS transfers on port $Port"

Run these commands on the VMM server, all library servers, and all hosts. Ensure that Group Policies are not configured to override these local firewall rules in a way that would block the new port.

Using the Graphical Interface (Windows Firewall with Advanced Security):

1. Open Windows Firewall with Advanced Security (wf.msc) on the server.
2. In the left pane, select Inbound Rules.
3. In the Actions pane (or right-click Inbound Rules), select New Rule….
4. Choose Port and click Next.
5. Select TCP, choose Specific local ports, and enter the new BITS port number (). Click Next.
6. Select Allow the connection and click Next.
7. Select the applicable network profiles (Domain, Private, Public). Typically, Domain and Private are sufficient within a corporate network. Click Next.
8. Give the rule a descriptive name (e.g., “VMM BITS Inbound “) and optional description. Click Finish.
9. Repeat steps 3-8 for Outbound Rules, selecting Outbound Rule in step 3 and Direction Outbound in step 5.

After configuring the firewalls, verify that the rules are active.

Step 4: Verify the Configuration and Test Transfers¶

Once the BITS port is changed in VMM and firewalls are configured, it’s time to verify the settings and test transfers.

Verify VMM BITS Port Setting:

You can check the currently configured BITS proxy port in VMM using PowerShell:

Get-SCVMMServer | Select-Object BITSProxyPort

This command should output the new port number you configured.

Verify Agent Configuration (Optional but recommended):

You can check the VMM agent configuration on a host or library server. The setting should be reflected in the agent’s configuration, typically stored in the registry or configuration files used by the agent service. While not usually necessary to manually check, understanding where the agent gets its configuration can be useful for advanced troubleshooting. The VMM agent receives this setting directly from the VMM server.

Test Data Transfers:

Perform operations in VMM that rely on BITS transfers, such as:
* Creating a new virtual machine from a template on a library server to a host.
* Migrating storage for an existing virtual machine.
* Copying a large file to a library share via the VMM console.

Monitor these operations to see if they complete successfully and if the transfer speeds are as expected (limited by BITS throttling or network bandwidth). If the transfers now succeed without the previous errors (12700, 2927, 2912), the port conflict was likely the issue, and changing the BITS port has resolved it.

Additional Troubleshooting Steps¶

If changing the BITS port and configuring firewalls do not resolve the issue, consider the following:

1. Verify Port Availability Again: Re-run netstat -aon on all relevant servers to ensure that the new port is not being used by another service and that VMM/BITS is attempting to listen on it after the configuration change.
2. Check VMM Service Status: Ensure that the System Center Virtual Machine Manager service and the System Center Virtual Machine Manager Agent service are running on all involved servers.
3. Check BITS Service Status: Ensure the Background Intelligent Transfer Service is running on the VMM server, library servers, and hosts. It should be set to start automatically.
4. Verify Name Resolution: Ensure proper DNS resolution between all VMM components (VMM server, library servers, hosts). Use ping or Test-NetConnection with hostnames and FQDNs.
5. Test Network Connectivity: Use Test-NetConnection -ComputerName <RemoteServer> -Port <NewBITSPort> from the VMM server to a host/library server and vice versa to confirm that the new port is reachable through firewalls.
6. Review VMM Logs: Examine the VMM server logs and the VMM agent logs on the affected servers for more detailed error information. These logs often provide a clearer picture of where the process failed. Logs are typically located in C:\ProgramData\Microsoft\Virtual Machine Manager\Trace. Use tools like TraceLogView to read them.
7. Check BITS Event Logs: Look at the BITS operational and administrative event logs in the Event Viewer (Applications and Services Logs > Microsoft > Windows > BITS). These logs can show specific BITS-related errors during the failed transfers.
8. Review Certificate Configuration: While less common for this specific error, if BITS is configured to use SSL (which it uses on port 443 by default), ensure that the necessary certificates are correctly deployed and trusted on all servers. However, the primary issue described here is port usage conflict, not typically an SSL certificate validation failure.
9. Temporarily Disable Firewalls (for testing only!): As a temporary troubleshooting step in a controlled environment, you could briefly disable Windows Firewall on the involved servers to definitively rule out firewall blocking as the cause. Do not do this in a production environment or leave firewalls disabled. If disabling firewalls resolves the issue, the problem is definitely firewall-related, and you need to re-examine your firewall rules carefully.

Considerations for Large Environments¶

In large VMM environments with many hosts and library servers, manually configuring firewalls on each server can be cumbersome. Group Policy Objects (GPOs) in Active Directory are the ideal method for centrally managing Windows Firewall rules across multiple servers. Create a GPO that applies to your VMM servers, library servers, and hosts, configuring the necessary inbound and outbound rules for the new BITS port.

Consider standardizing on a specific range of dynamic ports for internal management services to minimize conflicts. Document the chosen BITS port for future reference and for easier troubleshooting by other administrators.

Potential Relevant Video:

While a direct video on changing the VMM BITS port might be specific, understanding VMM networking and architecture is key. The following video provides an overview of System Center Virtual Machine Manager components and interactions, which can help visualize the communication paths involved in BITS transfers.

Note: Replace “examplevideoid” with a real, relevant YouTube video ID if available. For this response, this is a placeholder.

This detailed approach, including identifying the port conflict, carefully selecting a new port, configuring VMM, and updating firewalls, provides a robust solution to the common BITS transfer failures indicated by errors 12700, 2927, and 2912 in System Center Virtual Machine Manager. Always perform changes like this in a test environment first if possible, and schedule maintenance windows if necessary.

Successfully resolving these transfer issues is vital for the smooth operation of your virtual infrastructure, ensuring that you can efficiently deploy and manage virtual machines as needed.

Were you experiencing one of these specific error codes? Did changing the BITS port resolve the issue for you? Share your experience or ask questions in the comments below!