Fix Content Distribution Problems in Configuration Manager: A Troubleshooting Guide
Addressing Pull Distribution Point (DP) Job Processing Stalls¶
In Configuration Manager environments, distribution points (DPs) are crucial for delivering content to clients. Pull DPs, in particular, retrieve content from source DPs. However, scenarios can arise where pull DPs encounter processing stalls, preventing content from being distributed effectively. While recent updates to Configuration Manager, specifically version 1810 and later, have resolved many of these issues, environmental factors can still contribute to pull DP job processing problems.
Identifying Symptoms of Pull DP Job Stalling¶
A key indicator of a stalled pull DP is a substantial backlog of jobs that are not being processed. Administrators might observe a large number of pending jobs in the pull DP queue without any progress. This situation often becomes noticeable after deploying a new pull DP, especially when a significant volume of content is initially directed to it. When investigating, you might find thousands of Data Transfer Service (DTS) jobs within the ROOT\ccm\DataTransferService:CCM_DTS_JobEx WMI class and a considerable number of Background Intelligent Transfer Service (BITS) jobs, potentially 50 or more, stuck in a Failed state.
In such circumstances, a practical troubleshooting step involves clearing job-specific data from the pull DP’s WMI repository. Following this cleanup, a controlled redistribution of content to the pull DP can be initiated, allowing for a more focused investigation of any recurring failures. This approach helps isolate and address the root cause of the processing stall.
Resolving Stalls by Resetting Pull DP State with PowerShell¶
To streamline the process of removing job-specific items from WMI on a pull DP, a PowerShell script, Reset-PullDPState.ps1, is provided. This script is designed to reset the state of a pull DP by deleting data from various WMI classes associated with pull DP operations. It is important to execute this script with administrator privileges on the target pull DP server.
Understanding the Reset-PullDPState.ps1 Script¶
The Reset-PullDPState.ps1 script is engineered to interact with specific WMI classes critical to pull DP functionality. These classes include:
- CCM_DTS_JobEx: Stores details about DTS jobs, which are fundamental for content transfer within Configuration Manager.
- CCM_DTS_JobItemEx: Contains information on individual items within DTS jobs, representing specific content pieces.
- SMS_PullDPState: Tracks the overall state of packages on the pull DP.
- SMS_PullDPContentState: Monitors the state of individual content items on the pull DP.
- SMS_PullDPNotification (optional): Manages job notifications for the pull DP.
Beyond WMI cleanup, the script also offers the capability to check and report the count of BITS jobs present on the pull DP. BITS is the underlying transfer mechanism for content distribution, and managing its jobs can be essential for troubleshooting.
<#
.SYNOPSIS
Resets the state of the Pull DP and deletes data from various WMI classes related to Pull DP. You need to run this script as Administrator.
.DESCRIPTION
This script deletes the data from following WMI classes:
- CCM_DTS_JobEx
- CCM_DTS_JobItemEx
- SMS_PullDPState
- SMS_PullDPContentState
- SMS_PullDPNotification (optional)
The script also checks and reports the count of BITS Jobs.
.PARAMETER ComputerName
(Optional) Name of the Pull DP. You can leave this blank for local machine.
.PARAMETER DeletePullDPNotifications
(Optional) Use this switch if you want to delete the job notifications from SMS_PullDPNotification class.
.PARAMETER KeepBITSJobs
(Optional) Use this switch if you don't want the script to delete ALL BITS Jobs. If this switch is not used, ALL BITS jobs are deleted (even the ones that are not created by ConfigMgr)
.PARAMETER NotifyPullDP
(Optional) Use this switch if you want the script to execute NotifyPullDP method against SMS_DistributionPoint class. This is only useful when there aren't a lot of notifications in WMI and -DeletePullDPNotifications switch was not used.
.PARAMETER WhatIf
(Optional) Use this switch to see how many instances will be deleted.
.EXAMPLE
Reset-PullDPState -WhatIf
This command checks how many Pull PD jobs will get deleted when running the script
.EXAMPLE
Reset-PullDPState
This command resets the Pull DP related WMI classes except the Pull DP job Notification XML's
.EXAMPLE
Reset-PullDPState -DeletePullDPNotifications
This command resets the Pull DP related WMI classes along with the Pull DP job Notification XML's. If you do this, you would need to distribute/redistribute these packages to the Pull DP again.
.NOTES
07/28/2016 - Version 1.0 - Initial Version of the script
01/09/2019 - Version 2.0 - Added batch size for instance removal to prevent WMI Quota issues. Also added removal of BITS jobs (can be disabled by using -KeepBITSJobs switch) and restart of CcmExec service.
#>
[CmdletBinding()]
Param(
[Parameter(Mandatory=$false)]
[string]$ComputerName = $env:COMPUTERNAME,
[Parameter(Mandatory=$false)]
[switch]$DeletePullDPNotifications,
[Parameter(Mandatory=$false)]
[switch]$KeepBITSJobs,
[Parameter(Mandatory=$false)]
[switch]$NotifyPullDP,
[Parameter(Mandatory=$false)]
[switch]$WhatIf
)
$LogFile = Join-Path (Split-Path $SCRIPT:MyInvocation.MyCommand.Path -Parent) "Reset-PullDPState.log"
$ErrorActionPreference = "SilentlyContinue"
Function Write-Log {
Param(
[string] $text,
[switch] $NoWriteHost,
[switch] $IsErrorMessage,
[switch] $IsWarning,
[switch] $WhatIfMode
)
$timestamp = Get-Date -Format "MM-dd-yyyy HH:mm:ss"
"$timestamp $text" | Out-File -FilePath $LogFile -Append
if ($WhatIfMode) {
Write-Host $text -ForegroundColor Yellow
return
}
if (-not $NoWriteHost) {
if ($IsErrorMessage) {
Write-Host $text -ForegroundColor Red
}
elseif ($IsWarning) {
Write-Host $text -ForegroundColor Yellow
}
else {
Write-Host $text -ForegroundColor Cyan
}
}
}
Function Delete-WmiInstances {
Param(
[string] $Namespace,
[string] $ClassName,
[string] $Filter = $null,
[string] $Property1,
[string] $Property2 = "",
[string] $Property3 = "",
[int] $BatchSize = 10000
)
$success = 0
$totalfailed = 0
$counter = 0
$total = 0
Write-Host ""
Write-Log "$ClassName - Connecting to WMI Class on $ComputerName"
do {
if ($Filter -eq $null) {
$Instances = Get-WmiObject -ComputerName $ComputerName -Namespace $Namespace -Class $ClassName -ErrorVariable WmiError -ErrorAction SilentlyContinue | Select -First $BatchSize
}
else {
$Instances = Get-WmiObject -ComputerName $ComputerName -Namespace $Namespace -Class $ClassName -Filter $Filter -ErrorVariable WmiError -ErrorAction SilentlyContinue | Select -First $BatchSize
}
if ($WmiError.Count -ne 0) {
Write-Log " Failed to connect. Error: $($WmiError[0].Exception.Message)" -IsErrorMessage
$WmiError.Clear()
return
}
$currentfailed = 0
$current = ($Instances | Measure-Object).Count
if ($current -gt 0) {$script:serviceRestartRequired = $true}
if ($WhatIf) { break }
if ($current -ne $null -and $current -gt 0) {
Write-Log " Found $total total instances (Batch size $BatchSize)"
foreach($instance in $Instances) {
$instanceText = "$Property1 $($instance.$Property1)"
if ($Property2 -ne "") {
$instanceText += ", $Property2 $($instance.$Property2)"
}
if ($Property3 -ne "") {
$instanceText += ", $Property3 $($instance.$Property3)"
}
Write-Log " Deleting instance for $instanceText" -NoWriteHost
$counter += 1
$percentComplete = "{0:N2}" -f (($counter/$total) * 100)
Write-Progress -Activity "Deleting instances from $ClassName" -Status "Deleting instance #$counter/$total - $instanceText" -PercentComplete $percentComplete -CurrentOperation "$($percentComplete)% complete"
Remove-WmiObject -InputObject $instance -ErrorVariable DeleteError -ErrorAction SilentlyContinue
if ($DeleteError.Count -ne 0) {
Write-Log " Failed to delete instance. Error: $($DeleteError[0].Exception.Message)" -NoWriteHost -IsErrorMessage
$DeleteError.Clear()
$currentfailed += 1
}
else {
$success += 1
}
}
$totalfailed += $currentfailed
if ($currentfailed -eq $current) {
# Every instance in current batch failed. Break to avoid infinite while loop
break
}
}
} while (($Instances | Measure-Object).Count -ne 0)
if ($WhatIf) {
if ($total -eq $BatchSize) {
Write-Log " (What-If Mode) Found more than $BatchSize instances which will be deleted" -WhatIfMode
}
else {
Write-Log " (What-If Mode) $total instances will be deleted" -WhatIfMode
}
}
else {
if ($total -gt 0) {
# $totalfailed is likely not the accurate count here as it could include duplicate failures due to batching
Write-Log " Deleted $success instances. Failed to delete $totalfailed instances."
}
else {
Write-Log " Found 0 instances."
}
}
}
Function Check-BITSJobs {
$DisplayName = "BITS Jobs"
Write-Host ""
Write-Log "$DisplayName - Gettting jobs on $ComputerName"
Import-Module BitsTransfer
$Instances = Get-BitsTransfer -AllUsers -Verbose -ErrorVariable BitsError -ErrorAction SilentlyContinue | Where-Object {$_.DisplayName -eq 'CCMDTS Job'}
if ($BitsError.Count -ne 0) {
Write-Log " $DisplayName - Failed to get jobs. Error: $($BitsError[0].Exception.Message)" -IsErrorMessage
$BitsError.Clear()
}
else {
$total = ($Instances | Measure-Object).Count
Write-Log " $DisplayName - Found $total jobs"
if ($KeepBITSJobs) {
Write-Log " BITS Jobs will not be removed because KeepBITSJobs is true." -WhatIfMode
}
else {
if ($WhatIf) {
Write-Log " (What-If Mode) ALL BITS jobs will be removed since KeepBITSJobs is NOT specified." -WhatIfMode
}
else {
if ($total -gt 0) {
Write-Log " Removing ALL jobs since KeepBITSJobs is NOT specified."
Remove-BITSJobs
}
else {
Write-Log " There are no jobs to delete."
}
}
}
}
}
Function Remove-BITSJobs {
try {
Stop-Service BITS
Rename-Item "$($env:ALLUSERSPROFILE)\Microsoft\Network\Downloader" -NewName "Downloader.OLD.$([Guid]::NewGuid().Guid.Substring(0,8))"
Start-Service BITS
$script:serviceRestartRequired = $true
Write-Log " Removed ALL BITS Jobs successfully."
} catch {
Write-Log " Failed to delete the BITS jobs."
Write-Log " If necessary, run 'bitsadmin /reset /allusers' command under SYSTEM account (using psexec.exe) to delete the BITS Jobs."
Write-Log " Additionally, you can delete these jobs by stopping BITS service, renaming %allusersprofile%\Microsoft\Network\Downloader folder, and starting BITS service."
}
}
Function Restart-CcmExec {
$DisplayName = "SMS Agent Host"
Write-Host ""
Write-Log "$DisplayName - Checking if service restart is required."
if ($script:serviceRestartRequired) {
if ($WhatIf) {
Write-Log " (What-If Mode) Service Restart will be required." -WhatIfMode
if ($NotifyPullDP) {
Write-Log " (What-If Mode) NotifyPullDP method will be executed." -WhatIfMode
}
else {
Write-Log " (What-If Mode) NotifyPullDP method will NOT be executed because -NotifyPullDP switch was NOT used." -WhatIfMode
}
return
}
try {
Write-Host ""
Write-Log "### Restarting CCMEXEC service... ###"
Restart-Service CcmExec
Write-Log "### Success! ###"
} catch {
Write-Log "### ERROR! Restart CcmExec Manually in order to recreate BITS jobs for content transfer! ###"
}
if (-not $DeletePullDPNotifications -and $NotifyPullDP) {
# Only do this if notifications were not deleted. If they were deleted, NotifyPullDP will not do anything.
try {
Write-Host ""
Write-Log "### Invoking NotifyPullDP WMI method against the SMS_DistributionPoint class in $DPNamespace."
Invoke-WmiMethod -Namespace root\SCCMDP -Class SMS_DistributionPoint -Name NotifyPullDP | Out-Null
Write-Log "### Success! ###"
} catch {
Write-Log "### ERROR! Failed to invoke NotifyPullDP method! You can use wbemtest or WMI Explorer to invoke the method manually. ###"
}
}
else {
if (-not $NotifyPullDP) {
Write-Log "### Skipped invoking NotifyPullDP WMI method because -NotifyPullDP was NOT specified" -IsWarning
Write-Log "### You can use wbemtest or WMI Explorer to invoke the method manually, if necessary. ###"
}
if ($DeletePullDPNotifications) {
Write-Log "### Skipped invoking NotifyPullDP WMI method because -DeletePullDPNotifications was specified" -IsWarning
Write-Log "### Executing NotifyPullDP when there are no notifications does not do anything." -IsWarning
}
}
}
else {
Write-Log " Service Restart is NOT required. " -WhatIfMode
if ($NotifyPullDP) {
Write-Log " NotifyPullDP method skipped. " -WhatIfMode
}
}
}
Write-Host ""
Write-Log "### Script Started ###"
$script:serviceRestartRequired = $false
if ($WhatIf) {
Write-Host ""
Write-Log "*** Running in What-If Mode" -WhatIfMode
}
$DPNamespace = "root\SCCMDP"
$DTSNamespace = "root\CCM\DataTransferService"
Delete-WmiInstances -Namespace $DTSNamespace -ClassName "CCM_DTS_JobEx" -Filter "NotifyEndpoint like '%PullDP%'" -Property1 "ID"
Delete-WmiInstances -Namespace $DTSNamespace -ClassName "CCM_DTS_JobItemEx" -Property1 "JobID"
Delete-WmiInstances -Namespace $DPNamespace -ClassName "SMS_PullDPState" -Property1 "PackageID" -Property2 "PackageVersion" -Property3 "PackageState"
Delete-WmiInstances -Namespace $DPNamespace -ClassName "SMS_PullDPContentState" -Property1 "PackageKey" -Property2 "ContentId" -Property3 "ContentState"
if ($DeletePullDPNotifications) {
Delete-WmiInstances -Namespace $DPNamespace -ClassName "SMS_PullDPNotification" -Property1 "PackageID" -Property2 "PackageVersion"
}
else {
Write-Host ""
Write-Log "SMS_PullDPNotification - Connecting to WMI Class on $ComputerName"
$temp = Get-WmiObject -ComputerName $ComputerName -Namespace $DPNamespace -Class "SMS_PullDPNotification" -ErrorVariable WmiError -ErrorAction SilentlyContinue
if ($WmiError.Count -ne 0) {
Write-Log " SMS_PullDPNotification - Failed to connect. Error: $($WmiError[0].Exception.Message)" -IsErrorMessage
$WmiError.Clear()
}
else {
Write-Log " Found $(($temp | Measure-Object).Count) instances."
Write-Log " Skipped because DeletePullDPNotifications switch was NOT used." -IsWarning
}
}
if ($ComputerName -eq $env:COMPUTERNAME) {
Check-BITSJobs
}
else {
Write-Host ""
Write-Log "BITS Jobs"
Write-Log " Skipped because script is running against a remote computer." -IsWarning
}
Restart-CcmExec
Write-Host ""
Write-Log "### Script Ended ###"
Write-Host "### Check $LogFile for more details. ###" -ForegroundColor Cyan
#if (-not $WhatIf -and $serviceRestartRequired) {Write-Log "### Please restart the WMI service (which also restarts CcmExec). ###" -IsWarning}
Write-Host ""
This script offers several parameters to customize its behavior:
- -ComputerName: Allows you to specify the target pull DP by name. If omitted, the script operates on the local machine.
- -DeletePullDPNotifications: When used, this switch instructs the script to remove job notifications from the
SMS_PullDPNotificationclass. Be aware that using this switch necessitates redistributing or re-distributing packages to the pull DP afterward. - -KeepBITSJobs: By default, the script removes all BITS jobs. Employing this switch prevents the script from deleting BITS jobs, useful if you wish to preserve existing BITS transfers.
- -NotifyPullDP: This switch triggers the execution of the
NotifyPullDPmethod against theSMS_DistributionPointclass. This is most effective when there are few notifications in WMI and the-DeletePullDPNotificationsswitch is not used. - -WhatIf: Running the script with this switch in place allows you to preview the actions without actually making changes. It will show the number of instances that would be deleted.
Practical Usage Examples¶
To understand how to use the script, consider these examples:
Reset-PullDPState -WhatIf: This command is for assessment. It executes the script in What-If mode, providing a report of how many pull DP jobs would be deleted if the script were run without the-WhatIfswitch.Reset-PullDPState: This is the standard execution command. It resets the pull DP-related WMI classes, excluding the pull DP job notification XMLs.Reset-PullDPState -DeletePullDPNotifications: This command performs a comprehensive reset, clearing all pull DP-related WMI classes, including the job notification XMLs. Using this option will require subsequent content redistribution.
Executing the Reset-PullDPState.ps1 Script¶
To effectively use the Reset-PullDPState.ps1 script, follow these steps:
- Download the Script: Ensure you have downloaded or copied the
Reset-PullDPState.ps1script to your Configuration Manager environment. - Run as Administrator: Open PowerShell as an administrator on the pull DP server you are troubleshooting.
- Navigate to Script Location: Use the
cdcommand to navigate to the directory where you saved theReset-PullDPState.ps1script. - Execute with Parameters: Run the script with the appropriate parameters based on your needs. For instance, to perform a dry run, use
.\Reset-PullDPState.ps1 -WhatIf. For a full reset including notifications, use.\Reset-PullDPState.ps1 -DeletePullDPNotifications. - Review the Log File: After execution, examine the
Reset-PullDPState.logfile located in the same directory as the script. This log provides detailed information about the script’s actions and any potential errors.
By utilizing the Reset-PullDPState.ps1 script, administrators can efficiently address pull DP job processing stalls, clear problematic WMI data, and facilitate smoother content distribution in Configuration Manager.
Resolving File/Path Not Found Errors in Content Distribution¶
Another common challenge in Configuration Manager content distribution is encountering “file/path not found” errors. These errors typically manifest in the DistMgr (Distribution Manager) or PkgXferMgr (Package Transfer Manager) logs, indicating that content files are missing or inaccessible during the distribution process.
Identifying Common Error Codes¶
When investigating file/path not found issues, certain error codes are frequently observed in the logs. The most prevalent are:
- 0x80070002: This code signifies that the system cannot find the file specified.
- 0x80070003: This error indicates that the system cannot find the path specified.
These error codes point towards problems with content file locations within the Configuration Manager infrastructure.
Understanding the Root Cause¶
The primary cause of file/path not found errors during content distribution is often the absence of content files for a specific package within the content library on the site server. The content library serves as the central repository for all content managed by Configuration Manager. If files are missing from this library, PkgXferMgr will be unable to send them to distribution points, resulting in distribution failures.
Troubleshooting File/Path Not Found Errors¶
To diagnose and resolve file/path not found errors, several techniques can be employed:
Utilizing Content Library Explorer¶
The Content Library Explorer tool is a graphical utility that provides a view into the structure and contents of the Configuration Manager content library. This tool allows administrators to browse the library and verify if the content files for a particular package are present and accessible. While Content Library Explorer is a valuable resource, it can sometimes be time-consuming to load, especially in large environments with extensive content libraries.
Manually Tracking Content from PkgLib to FileLib¶
A more direct approach involves manually tracking the content flow from PkgLib (Package Library) to FileLib (File Library). This process involves examining the Configuration Manager database and file system to ensure content transitions correctly through these stages. By tracing the content identifiers and file locations, administrators can pinpoint where content may be missing or encountering issues.
Leveraging Process Monitor¶
Process Monitor, a Sysinternals tool, is an advanced monitoring utility that captures real-time file system, registry, and process/thread activity. Capturing a Process Monitor trace while content distribution is occurring can provide detailed insights into file access attempts. By analyzing the trace, administrators can quickly identify if the necessary files are indeed missing from the content library on the site server, as indicated by “Path Not Found” results in the trace.
Resolving File/Path Not Found Issues¶
The resolution strategy for file/path not found errors depends on the site server’s role in the package source hierarchy.
Updating Package Source on the Source Site¶
If the site server experiencing the content library issue is also the package source site (the originating location of the package content), the solution is to update the package source. This involves incrementing the Package Source Version. Incrementing this version triggers DistMgr to take a fresh snapshot of the content from the package source directory. This action effectively re-populates any missing content within the content library, resolving the file/path not found errors.
Resending Compressed Copy of Package¶
In scenarios where the site server with the content library problem is not the package source site, the resolution involves forcing the package source site to resend a compressed copy of the package to the affected site. This process re-transfers the content from the source site to the problematic site, ensuring the content library is correctly populated. This approach is particularly useful in hierarchical Configuration Manager environments where content may be distributed across multiple sites.
By applying these troubleshooting and resolution techniques, administrators can effectively address file/path not found errors, ensuring reliable and successful content distribution within their Configuration Manager infrastructure.
Conclusion¶
Content distribution problems in Configuration Manager can stem from various sources, ranging from pull DP processing stalls to file/path not found errors. By understanding the symptoms, employing the appropriate troubleshooting tools and techniques, and utilizing provided solutions like the Reset-PullDPState.ps1 script and content source updates, administrators can effectively diagnose and resolve these issues. Maintaining a healthy content distribution infrastructure is paramount for ensuring timely and successful software deployment and updates across the managed environment.
If you have encountered similar content distribution challenges or have further insights into troubleshooting these issues, please share your experiences and thoughts in the comments below. Your contributions can help the community further refine best practices for managing Configuration Manager content distribution.
Post a Comment