Troubleshooting Data Protection Manager: Resolving Common Backup Failures

Table of Contents

Data Protection Manager (DPM) is a robust enterprise backup system, but like any complex distributed application, it can encounter communication issues leading to backup failures. These failures often manifest as error messages indicating a loss of communication between the DPM server and its protected agents. Understanding the root causes and implementing effective workarounds is crucial for maintaining a reliable backup environment. This article delves into common DPM backup failure symptoms, their underlying causes, and comprehensive strategies for resolution.

Symptoms of Communication Failure¶

DPM backup and synchronization jobs can fail with a variety of error messages, all pointing towards an inability for the DPM server to effectively communicate with the protection agent on the target server. These errors can occur during various job types, including synchronization, recovery point creation, and consistency checks. Recognizing these common error patterns is the first step in diagnosing the problem.

Here are some of the frequently encountered error messages:

• Type: Recovery point Status: Failed Description: An unexpected error occurred during job execution. (ID 104 Details: Internal error code: 0x80990A51)
  This generic error often signifies an underlying communication or operational issue that DPM cannot specifically categorize. The internal error code points to a problem within the DPM agent’s processing capability or its interaction with the DPM server. It typically suggests that the agent itself encountered an unexpected state or a critical failure during the job execution phase.

• Type: Tape backup Status: Failed Description: DPM failed to communicate with the protection agent on Server_name.com because access is denied. (ID 42 Details: Access is denied (0x80070005))
  An “Access Denied” error explicitly indicates a permissions issue. This can arise if the DPM server’s service account lacks the necessary rights to interact with the protected server’s agent, or if firewall rules are blocking specific DCOM or RPC ports. It’s a critical security-related communication breakdown that prevents the DPM server from initiating or controlling the agent’s operations.

• DPM could not communicate with the Protection Agent service on ProtectedServer.Contoso.local. (ID 308 Details: The RPC server is unavailable (0x800706BA))
  This error is a classic indicator that the Remote Procedure Call (RPC) service, fundamental for DCOM communication, is either not running, blocked by a firewall, or otherwise inaccessible on the protected server. It prevents the DPM server from even establishing the initial control channel with the agent. This often points to network-level connectivity issues or service disruptions.

• The protection agent operation failed because DPM could not communicate with the Protection Agent service on protected.server.contoso.com. (ID 308 Details: The RPC server is unavailable (0x800706BA))
  Similar to the above, this variant of ID 308 reinforces that the RPC communication path is severely impaired. The DPM server attempts to execute an operation on the agent but fails because the RPC endpoint is unreachable. This could be due to network outages, incorrect DNS resolution, or an agent service crash.

• Type: Recovery point Status: Failed Description: The protection agent on Server_name.com was temporarily unable to respond because it was in an unexpected state. (ID 60 Details: Internal error code: 0x809909B0)
  Error ID 60 suggests that the DPM agent service itself is running but has entered an unstable or unresponsive state. While not a direct communication failure like RPC unavailability, it prevents the agent from processing commands from the DPM server, effectively breaking the communication flow. This might be due to resource exhaustion or an internal agent crash.

• The DPM service was unable to communicate with the protection agent on ProtectedServer.Contoso.local. (ID 52 Details: An existing connection was forcibly closed by the remote host (0x80072746))
  This error typically signifies that a TCP connection, once established, was abruptly terminated by the protected server. This can happen due to firewall rules that suddenly drop connections, network device issues, or if the agent service crashed mid-operation. It indicates a pre-existing connection being severed prematurely.

• The DPM service was unable to communicate with the protection agent on ProtectedServer.Contoso.local. (ID 52 Details: The semaphore timeout period has expired (0x80070079))
  A semaphore timeout indicates that a specific operation or resource lock on the protected server took too long to respond or release. This often points to performance bottlenecks on the protected server, such as high disk I/O, CPU contention, or severe network latency preventing timely responses. The agent is waiting for a resource that never becomes available within the allotted time.

• DPM failed to communicate with the protection agent on ProtectedServer.Contoso.local because the agent is not responding. (ID 43 Details: Internal error code: 0x8099090E)
  Error ID 43 is a general “agent not responding” error, often occurring after DPM attempts to establish contact but receives no reply within its internal timeout period. This is similar to a hard timeout and can be caused by network congestion, a crashed agent service, or the protected server being offline. The internal error code pinpoints a specific DPM communication layer timeout.

• DPM failed to communicate with ProtectedServer.Contoso.local because the computer is unreachable. (ID 41 Details: No connection could be made because the target machine actively refused it (0x8007274D))
  This is a definitive network-level rejection. “Connection refused” means that the target server (or a firewall) explicitly denied the connection attempt. This is often due to an active firewall blocking the necessary DPM agent ports (5718, 5719) or RPC dynamic ports, or the DPM agent service not running on the protected server at all.

These varied error messages, while distinct, frequently share a common underlying cause: network-related impediments that disrupt the intricate communication pathways DPM relies upon.

Underlying Cause: Network Latency and Saturation¶

The primary culprit behind these persistent communication failures in DPM environments is often excessive network latency or saturation. DPM utilizes two distinct channels for its operations: a control channel and a data channel.

The control channel is primarily managed by DCOM (Distributed Component Object Model) and RPC (Remote Procedure Call). This channel is responsible for the signaling, command execution, and status reporting between the DPM server and the protection agents. It’s crucial for initiating jobs, querying agent status, and managing the overall backup process.

The data channel, conversely, handles the actual transfer of protected data from the agent to the DPM storage pool. This channel predominantly uses TCP/IP protocols to move large volumes of data efficiently.

When the network experiences high latency or becomes saturated with data transfers—particularly during periods of heavy backup activity—DCOM traffic can be significantly delayed. DCOM, being sensitive to timely responses, may experience timeouts if its calls are not acknowledged within a predefined window. If the data path completely consumes the available network bandwidth, even for a brief period, it starves the control channel of necessary resources. This starvation leads to DCOM calls failing, which in turn causes the entire DPM job to fail. Agents may become unresponsive or report unexpected states because their communication back to the DPM server is disrupted, leading to the diverse range of error messages observed.

Common factors contributing to such network conditions include:
* Concurrent Backup Jobs: Multiple large backup jobs running simultaneously can overwhelm network links.
* Insufficient Bandwidth: The network infrastructure might not have enough capacity to handle the peak backup traffic alongside regular production traffic.
* Network Device Congestion: Routers, switches, or firewalls may become bottlenecks due to high throughput.
* Underlying Network Issues: Packet loss, excessive hops, or misconfigured network adapters can introduce latency.

Understanding this interplay between the control and data channels and the impact of network conditions is key to implementing effective workarounds.

Comprehensive Workarounds¶

To mitigate the impact of network latency and saturation on DPM communication, a multi-faceted approach involving registry modifications, DPM throttling, and network optimization is recommended. These steps aim to make DPM’s communication more resilient and prevent the control channel from being starved.

1. Registry Modifications for Enhanced Resilience¶

The first step involves adding specific registry entries on both the DPM server and all protected servers experiencing issues. These entries adjust DPM agent timeouts and TCP/IP retransmission parameters, making communication more forgiving during periods of network stress.

Steps to Apply Registry Entries:

1. Create the Registry File: Open Notepad and paste the following content precisely:

   Windows Registry Editor Version 5.00
   
   [HKEY_LOCAL_MACHINE\Software\Microsoft\Microsoft Data Protection Manager\Agent]
   "ConnectionNoActivityTimeoutForNonCCJobs"=dword:00001c20
   "ConnectionNoActivityTimeout"=dword:00001c20
   "AbortAgentOnLockTimeout"=dword:00000001
   "CommandTimeout"=dword:1b7740
   
   [HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters]
   "TcpMaxDataRetransmissions"=dword:00000010
   

2. Save the File: Save the file with a .reg extension, for example, DPMAgentTimeout.reg. Ensure “Save as type” is set to “All Files” to prevent it from being saved as DPMAgentTimeout.reg.txt.

3. Apply to Systems:

   • On the DPM Server: Double-click the DPMAgentTimeout.reg file and confirm the merge operation when prompted.
   • On Each Protected Server: Copy the DPMAgentTimeout.reg file to each affected protected server and double-click it, confirming the merge.

4. Restart Services/Servers: For these changes to take effect, you must restart the DPMRA service on both the DPM server and all affected protected servers. Alternatively, a full server restart will also apply the changes.

Explanation of Registry Keys:

• ConnectionNoActivityTimeoutForNonCCJobs and ConnectionNoActivityTimeout (dword:00001c20): These values, set to 0x1c20 (7200 decimal) seconds, significantly increase the timeout period for DPM agent connections when there’s no activity. This provides a much longer grace period before DPM considers an agent unresponsive due to temporary network stalls or delays, allowing more time for the connection to recover.
• AbortAgentOnLockTimeout (dword:00000001): Setting this to 1 (true) instructs the DPM agent to abort an operation if it encounters a lock timeout. This prevents the agent from getting indefinitely stuck waiting for a resource, allowing the job to fail gracefully and potentially retry later.
• CommandTimeout (dword:1b7740): This value, set to 0x1b7740 (1800000 decimal) milliseconds, extends the general command timeout for DPM. It allows DPM commands to wait longer for a response from the agent or the underlying system, accommodating slower environments or complex operations that might take more time.
• TcpMaxDataRetransmissions (dword:00000010): Located under the Tcpip\Parameters key, this global TCP/IP setting controls the maximum number of times TCP will retransmit a data segment before the connection is aborted. The default value is typically 5. By increasing it to 16, you make the TCP/IP stack more resilient to temporary packet loss or high network latency, allowing more retransmission attempts before a connection is deemed broken. This directly benefits the data channel and indirectly helps the control channel by making the underlying network more robust.

Important Note: Always back up your registry before making any manual modifications. While these specific changes are generally safe and recommended by Microsoft for DPM issues, understanding registry editing is crucial.

2. Implementing DPM Bandwidth Throttling¶

If the problem persists after applying the registry modifications, especially if network saturation is a known factor, enabling DPM’s built-in bandwidth throttling is highly recommended. Throttling limits the amount of network bandwidth that DPM data transfers consume, reserving crucial capacity for the DCOM control channel and other network operations.

Prerequisites for Throttling:
DPM’s throttling feature relies on Quality of Service (QoS) policies. For throttling to function correctly, the QoS Packet Scheduler component must be installed and enabled on both the DPM server and all protected servers where throttling will be applied. This is typically installed by default with Windows Server, but it’s worth verifying.

Steps to Enable DPM Throttling:

1. Open DPM Administrator Console: Navigate to the Management tab.
2. Select Agents: Click on the Agents sub-tab.
3. Configure Throttling: Select the protected server(s) you wish to throttle, right-click, and choose Configure Throttling.
4. Enable Throttling: Check the box that says Enable network bandwidth usage throttling.
5. Set Usage Limit: Initially, set the network usage limit for the data path to 85 percent during all hours. This leaves 15% of the bandwidth available for the DCOM control channel and other network traffic.
6. Adjust Schedule (Optional): You can define specific throttling schedules if your network usage patterns vary throughout the day. For example, less aggressive throttling during off-peak hours and stricter throttling during peak business hours.
7. Apply Changes: Click OK to apply the settings.

Why Throttling Helps:
By actively limiting the data transfer rate, throttling prevents the data channel from completely saturating the network. This ensures that sufficient bandwidth remains available for the DCOM/RPC control channel. If the control channel consistently has enough bandwidth to communicate, the DPM server and agents can exchange commands and status updates reliably, preventing timeouts and job failures.

If 85 percent throttling doesn’t entirely resolve the issue, consider incrementally increasing the throttling level to 90 percent or even 95 percent for particularly problematic protected servers. Monitor the job success rates and network performance after each adjustment.

3. Protection Group Optimization¶

Even with registry tweaks and throttling, an overloaded DPM server can still struggle. If the DPM server itself is overwhelmed by too many concurrent jobs or an inefficient schedule, it can lead to communication bottlenecks.

• Stagger Backups: Review your protection groups and backup schedules. Instead of having many large jobs start simultaneously, stagger them throughout the backup window. This distributes the load more evenly across the DPM server’s resources and the network.
• Stagger Synchronizations: Similar to backups, spreading out synchronization jobs can significantly reduce peak load.
• Optimize Recovery Point Creation: Adjust the frequency of recovery point creation for less critical data to reduce overhead.
• Review DPM Server Resources: Ensure the DPM server has adequate CPU, memory, and disk I/O capacity to handle its workload. Poor performance on the DPM server itself can cause delays and timeouts.

Advanced Troubleshooting and Best Practices¶

Beyond the primary workarounds, several other factors can contribute to DPM communication failures. A holistic approach involves checking network health, firewall rules, antivirus configurations, and general system health.

Network Diagnostics¶

A stable network is paramount for DPM.
* Ping and Tracert: Use ping and tracert commands from the DPM server to the protected server (and vice-versa) to check for basic connectivity, latency, and packet loss. High latency or dropped packets are direct indicators of network issues.
* Network Performance Monitoring: Utilize tools like Windows Performance Monitor (PerfMon) or third-party network monitoring solutions to track network utilization, errors, and discard rates on interfaces involved in DPM traffic. Look for spikes in bandwidth usage or network errors that correlate with job failures.
* Network Adapter Settings: Verify that network adapters on both DPM and protected servers are configured with the correct speed and duplex settings and that jumbo frames are configured consistently if used.
* DNS Resolution: Ensure proper DNS resolution between the DPM server and protected servers. Incorrect or slow DNS can lead to “unreachable” errors.

Firewall and Antivirus Considerations¶

Firewalls and antivirus software are critical for security but can inadvertently block legitimate DPM communication if not configured correctly.

• Firewall Rules: Ensure that necessary DPM ports are open on both the DPM server and all protected servers. DPM uses a range of ports for communication.

Port(s)	Protocol	Description
5718, 5719	TCP	DPM Agent Communication and Callback
135	TCP	RPC Endpoint Mapper (Initial DCOM communication)
Dynamic (49152-65535)	TCP	RPC Dynamic Ports (for DCOM object communication)
445	TCP	SMB (for file share protection, administrative shares)
3389	TCP	RDP (for remote management/troubleshooting)
1433, 1434	TCP, UDP	SQL Server (for DPM database communication)

Ensure these ports are allowed inbound/outbound as needed by DPM components. For dynamic RPC ports, ensure the range is open or a static range is configured and allowed.
* Antivirus Exclusions: Antivirus software can interfere with DPM processes and data transfers. It’s crucial to configure appropriate exclusions for DPM directories and processes on both the DPM server and all protected servers. Key exclusions typically include:
* DPM installation directory (e.g., C:\Program Files\Microsoft DPM\)
* DPM storage pool volumes (if applicable)
* DPMRA.exe, DPM.exe, msdpm.exe, dpmac.exe (and other DPM-related executables)
* SQL Server data and log files used by DPM
* System Volume Information folders on protected volumes (where change journal resides)

DPM Server and Agent Health¶

Regular checks of the DPM server and its agents can preempt many issues.
* DPM Server Resource Utilization: Monitor CPU, RAM, and disk I/O on the DPM server. High utilization can lead to delays in processing jobs and communication. Ensure the SQL Server instance hosting the DPM database is also performing optimally.
* DPMRA Service: On protected servers, verify that the DPMRA service (Microsoft System Center Data Protection Manager Agent) is running and set to Automatic startup. If it’s stopped, attempt to start it and check event logs for reasons for its stoppage.
* Reinstall Agent: If an agent consistently causes issues and troubleshooting efforts fail, consider reinstalling the protection agent on the problematic server. This can resolve corrupted agent installations or configuration issues.
* Event Logs: Review event logs on both the DPM server (Application and System logs) and the protected servers (Application, System, and DPM-specific logs if available) for any related errors or warnings. These can provide deeper insights into the specific point of failure.

System Updates and Hotfixes¶

Ensure that your DPM server, protected servers, and the underlying Windows Server operating systems are kept up-to-date with the latest service packs, cumulative updates, and specific DPM hotfixes. Microsoft frequently releases updates that address known communication and performance issues.

By systematically addressing each of these potential areas, administrators can significantly improve the stability and success rate of their DPM backup operations, ensuring robust data protection for their environment.

We hope this comprehensive guide helps you in resolving common DPM backup failures. If you have experienced these issues or have additional tips and insights, please share them in the comments section below. Your experiences contribute greatly to the community’s collective knowledge!