Troubleshooting: Azure Offline Backup Seeding Stuck on Import Job Completion

Table of Contents

Offline backup seeding in Microsoft Azure Backup (MAB) offers a practical solution for transferring large initial backup datasets to Azure. This process bypasses network limitations by allowing data to be shipped on physical storage devices directly to an Azure datacenter. However, users sometimes encounter issues where the process halts unexpectedly, particularly during the phase where the MAB agent waits for the import job initiated in the Azure portal to finalize. Understanding the workflow and potential pitfalls is crucial for successful offline seeding.

Symptoms¶

The primary symptom of this specific issue is that the Microsoft Azure Backup (MAB) agent interface ceases progress after the physical drives containing the initial backup data have been shipped and received by Azure. The MAB agent displays a persistent message indicating that it is awaiting the completion of the Azure Import Job. This message typically states:

Waiting for Azure Import Job to complete. Please check on Azure management portal for more information on job status

Despite verifying the status in the Azure portal, which might show the import job is processing or completed, the MAB agent on the source server does not update its status or proceed further. This disconnect prevents the finalization of the offline seeding process, leaving the backup data unlinked to the Recovery Services Vault in a fully usable state.

Further investigation often reveals specific error messages within the MAB agent’s logs. These logs, commonly found at C:\Program Files\Microsoft Azure Recovery Services Agent\Temp\CBEngineCurr.errlog, may contain entries suggesting a failure to locate or match the expected import job details. Log snippets frequently point to an issue retrieving the status of the specified import job, often including an error code or a message explicitly stating the import job was not found.

A typical log pattern associated with this problem might look like this:

...
FATAL GetImportJobStatus:: Import job with name 'ImportJobName' is not found
NORMAL PollAzureImportJob :: Import Job: 'ImportJobName' is in pending state: '1'
WARNING PollAzureImportJob failed with error code: 1b7afbe0
WARNING Failed: Hr: = [0x80780054] Offline Backup Progress: Failed"
.....

These log entries clearly indicate that the MAB agent is looking for an import job with a specific name but cannot find a corresponding job in Azure, or at least cannot match its internal reference to the job in the portal. This inability to synchronize status prevents the agent from recognizing the import job’s progress or completion, effectively freezing the offline seeding workflow.

Understanding the Offline Seeding Process¶

To fully grasp why this issue occurs, it helps to review the high-level steps involved in Azure offline backup seeding. The process begins with configuring a backup policy in the MAB agent and selecting the offline seeding option. During this configuration, the user specifies a target Azure Recovery Services Vault and initiates the initial backup data transfer.

Instead of sending data directly over the network, the MAB agent prepares the data locally and directs it to a staging location. Subsequently, the Azure Import/Export tool is used to prepare the physical drives with this data. This tool encrypts the data on the drives and generates the necessary manifest files. Crucially, during this phase, the user must provide details related to the Azure Import Job they intend to create in the portal.

Once the drives are prepared, they are shipped to a designated Azure datacenter address. Concurrently, or shortly after shipping, the user must create an Import Job in the Azure portal within the subscription and region associated with the Recovery Services Vault. This job informs Azure about the incoming drives, their contents, and where the data should be imported (specifically, into a staging storage account that will eventually feed the Recovery Services Vault). A critical piece of information entered during the portal job creation is the “Job Name”.

After Azure receives the drives and processes the data import, the import job status in the Azure portal updates through various stages (e.g., Received, Data Copying, Copy Completed, Completed). The MAB agent on the source server is designed to poll Azure for the status of the associated Import Job. It uses the job name provided during the initial offline seeding configuration phase to look up the job in the portal. When the agent detects that the import job has successfully completed and the data is available in the staging storage account, it proceeds to the final stage of the offline seeding process, which involves transferring the data from the staging storage account to the Recovery Services Vault and linking it to the protected items.

This polling mechanism is where the discussed issue arises. If the name the MAB agent is looking for does not exactly match the name assigned to the Import Job created in the Azure portal, the agent cannot find the job, retrieve its status, or proceed, leading to the “Waiting for Azure Import Job to complete” message being stuck indefinitely from the agent’s perspective.

Cause¶

The root cause of the MAB agent getting stuck while waiting for the Azure Import Job completion is a fundamental mismatch between the name specified for the import job within the Microsoft Azure Backup (MAB) agent configuration and the actual name assigned to the corresponding Import Job created in the Azure portal. During the setup of the offline backup policy using the MAB agent, the user is prompted to enter a name for the Azure Import Job. This name serves as the internal identifier for the MAB agent to track the state of the external import process being managed by the Azure Import/Export service.

Separately, the user navigates to the Azure portal and creates a new Import Job, providing details about the shipped drives, return address, and, critically, a “Job Name”. The Azure Import/Export service uses this name as the unique identifier for the specific import operation. For the MAB agent to successfully poll Azure for the status of this import operation, the name it uses internally must precisely match the “Job Name” used when creating the job in the Azure portal.

Any discrepancy, even a minor typo, difference in capitalization, or inclusion of extra spaces, will prevent the MAB agent from correctly identifying the associated import job in Azure. As a result, the agent’s polling mechanism fails, it cannot ascertain the completion status of the import, and therefore, it remains stuck in the waiting state. This issue highlights the importance of meticulous data entry and verification during the offline seeding configuration steps both within the MAB agent and the Azure portal.

Resolution¶

Resolving this issue requires correcting the linkage between the MAB agent’s expectation and the actual Azure Import Job. Since the MAB agent is fixated on an incorrect or non-existent job name, the most straightforward and supported method is to restart the offline seeding process with the correct parameters. This involves removing the existing, problematic backup policy and configuring a new one.

The critical step in the resolution is to remove the current backup policy using the “Stop protection and delete data” option within the MAB agent interface. Navigating to the ‘Protected Items’ section in the MAB agent allows you to see the listed protection groups or individual items. Selecting the item experiencing the issue and choosing the option to stop protection will present choices. It is imperative to select the option that not only stops future backups but also removes the associated backup data from Azure. This action effectively cleans up the state related to the previous, misconfigured offline seed job within the Recovery Services Vault and prepares the protected item for a new policy configuration.

Important Consideration: Choosing “Stop protection and delete data” means that the backup data previously associated with this policy (even if it was the data shipped via drives) will be marked for deletion from the Recovery Services Vault and the staging storage account. Ensure you understand this implication, especially if the data import process in Azure was completed successfully. The new offline seeding process will involve preparing the data and drives again or utilizing the existing data on drives if they haven’t been returned and if the data is still valid and accessible.

After the old policy state is cleared, you must schedule a new backup policy for the same data source. During the configuration of this new policy using the MAB agent, when you reach the step to configure offline seeding, you will be prompted again to specify the details related to the Azure Import Job. At this point, you must carefully enter the exact “Job Name” that was used when you created or will create the corresponding Import Job in the Azure portal for shipping the drives.

This new configuration establishes a correct link between the MAB agent and the Azure Import Job based on the perfectly matched names. Once the drives are shipped (if necessary again with newly prepared data or using the existing drives if the data is valid and matches the new backup snapshot) and the Import Job in Azure is created with the matching name, the MAB agent will be able to successfully poll the Azure Import/Export service, track the job’s progress, and proceed to the final stage of the offline seeding workflow upon job completion.

For comprehensive, step-by-step instructions on configuring offline backup seeding, including preparing drives with the Azure Import/Export tool and creating the Import Job in the Azure portal, refer to the official Microsoft documentation on the offline backup workflow. Following these steps meticulously, paying special attention to matching the Import Job name in both the MAB agent configuration and the Azure portal, is key to a successful offline seeding operation and preventing the “stuck” symptom.

Deeper Dive into the Process and Potential Pitfalls¶

While the job name mismatch is a common cause, understanding the entire offline seeding lifecycle helps troubleshoot related issues or ensure a smooth process. The offline seeding process is a multi-phase operation involving software (MAB agent, Azure Import/Export tool), hardware (physical drives), logistics (shipping), and Azure services (Storage Accounts, Import/Export service, Recovery Services Vault).

Phase 1: Data Preparation (On-Premises)
1. Backup Policy Creation in MAB Agent: You define what data to back up and select the option for offline seeding. You specify a staging location for the initial backup data. This is where the agent first writes the backup copy.
2. Offline Seeding Configuration: As part of the policy, you enable offline seeding. You are asked to enter the Azure Import Job name that you intend to create later in the portal. This is the critical point where the mismatch can occur.
3. Initial Backup and Data Staging: The MAB agent performs the first backup run as per the policy. It copies the data to the specified staging location on your server.
4. Drive Preparation using Azure Import/Export Tool: You download and run the Azure Import/Export tool (WAImportExport). You specify the staging location, the target Azure storage account (usually provided by the MAB agent setup), the drive letter(s) of the physical drives you are using, and encryption keys. The tool formats the drives, encrypts the data, and copies the staged backup data onto the drives. It also generates journal files for each drive, which contain essential metadata.

Phase 2: Shipping and Azure Import (Azure Side)
1. Creating the Import Job in Azure Portal: You log into the Azure portal and create a new Import Job. You specify the storage account, the data center destination address (provided by the portal), shipping carrier details, and upload the journal file(s) generated by the WAImportExport tool. Here, you define the “Job Name” in the Azure portal. This name must match the name entered in the MAB agent configuration.
2. Shipping the Drives: You ship the prepared drives to the specified Azure data center address.
3. Azure Data Center Processing: Azure receives the drives, validates them, and initiates the data transfer from the physical drives to the designated staging storage account in your subscription. The status of this process is reflected in the Import Job status in the Azure portal.

Phase 3: Finalization (On-Premises and Azure)
1. MAB Agent Polling: The MAB agent on your source server periodically polls the Azure Import/Export service (via the Recovery Services Vault) to check the status of the Import Job using the name configured in the MAB policy.
2. Data Transfer to Recovery Services Vault: Once the Import Job status in Azure indicates completion, the MAB agent recognizes this. It then triggers the final transfer of the data from the staging storage account into the Recovery Services Vault, linking it to the protected items configured in the MAB agent.
3. First Online Backup: After successful offline seeding, subsequent backups happen online over the network, sending only incremental changes.

Other Potential Issues (Beyond Name Mismatch):

While the name mismatch is specific to the stuck state before recognizing import completion, other problems can arise during offline seeding:

• Incorrect Staging Location Permissions/Space: The staging area must have enough free space and appropriate permissions for the MAB agent and the WAImportExport tool.
• Issues with WAImportExport Tool: Problems during drive preparation (e.g., tool crashes, drive errors, incorrect journal file generation).
• Drive Encryption Key Issues: The bitlocker encryption key used by WAImportExport must be securely stored and linked to the Import Job. Loss of this key makes the data unrecoverable.
• Shipping/Logistics Problems: Drives getting lost, damaged in transit, or delays in processing at the Azure data center.
• Import Job Creation Errors in Portal: Mistakes in uploading journal files, specifying the wrong storage account, or providing incorrect return shipping details.
• Import Job Failures in Azure: Data corruption detected during import, drive read errors, or issues within the Azure service itself. The Azure portal job status will show failure details.
• Staging Storage Account Issues: If the staging storage account is deleted, its keys are rotated, or it faces other access issues before the data is transferred to the Recovery Services Vault, the finalization step will fail.
• Network Connectivity: Although the initial data transfer is offline, the MAB agent still needs network connectivity to Azure (Recovery Services Vault and Storage Account) to configure policies, trigger jobs, and perform the final data transfer from staging to the vault.

The error logs are invaluable for diagnosing these issues. The specific error codes and messages provide clues about which phase of the process failed and why. For instance, errors related to storage account access would point away from the job name mismatch and towards network or credential problems.

Visualizing the Workflow¶

A simplified flow diagram can illustrate the interaction points and where the job name is crucial.

```mermaid
graph TD
A[Configure MAB Policy
Select Offline Seeding] → B(Enter Azure Import Job Name
e.g., ‘MyImportJob123’)
A → C(Specify Staging Location)
C → D(Perform Initial Backup to Staging)
D → E(Prepare Drives with WAImportExport Tool)
E → F(Upload Journal Files & Ship Drives)
subgraph Azure Portal
G(Create Import Job)
F → G
G → H(Enter Job Name
MUST MATCH ‘MyImportJob123’)
G → I(Specify Storage Account
Upload Journal Files)
end
I → J(Azure Receives Drives & Copies Data)
J → K(Import Job Status Updates in Portal)
B → L{MAB Agent Polling Azure?}
K → L
L → M{Job Name Match Found?}
M – No → N(MAB Stuck: “Waiting for Azure Import Job…”)
M – Yes → O{Import Job Completed?}
O – No → L
O – Yes → P(Transfer Data from Staging to Vault)
P → Q(Offline Seeding Complete)

N --> Restart[Requires Policy Removal & Reconfig]
Restart --> A

```
(Diagram: Simplified Azure Offline Seeding Workflow highlighting the Job Name match point)

This diagram shows how the ‘Azure Import Job Name’ entered in the MAB agent directly maps to the ‘Job Name’ created in the Azure portal. The loop between MAB Polling and Job Status Update (L->M->O->L or O->P) is broken if the name match fails at step M, leading to the stuck state N. The resolution involves going back to step A after clearing the old policy.

Verifying the Import Job Status in Azure Portal¶

While the MAB agent is stuck, you can independently verify the status of your Import Job in the Azure portal. Navigate to Storage accounts -> Your staging storage account -> Blob service -> Azure Import/Export. You should see your list of import (and export) jobs. Click on the relevant job name (the one you intended the MAB agent to find) to see its detailed status.

Statuses progress through stages like ‘Received’, ‘DataCopying’, ‘ParametersUpdated’, ‘CopyCompleted’, ‘Completed’, or ‘Failed’. If the status is ‘Completed’, but the MAB agent is still stuck, it strongly reinforces the likelihood of a job name mismatch as the agent cannot find this specific completed job under the name it is using. If the status is ‘Failed’, the portal will usually provide a reason, which would then become the primary issue to troubleshoot, potentially requiring a new import job after addressing the cause of failure.

Call to Action¶

Encountering issues like this during crucial processes like offline backup seeding can be frustrating. Have you experienced the MAB agent getting stuck on the import job completion? Did the job name mismatch turn out to be the cause, or did you encounter a different issue? Share your experiences and troubleshooting tips in the comments below. Your insights could help others navigate the complexities of Azure offline backup.