Securely Transfer Azure Blobs: AzCopy for Storage Account Replication with Access Restrictions

Table of Contents

Copying blobs between Azure storage accounts is a frequent requirement in various cloud scenarios, from data backup and disaster recovery to content distribution and data migration. Azure Storage provides a native and efficient way to perform these copy operations directly server-side, eliminating the need for intermediary downloads and uploads. This process can be effectively managed using the command-line utility AzCopy, a powerful tool designed for high-performance data transfer to, from, and between Azure storage accounts. AzCopy leverages the Azure Storage infrastructure, ensuring optimal throughput and minimizing bandwidth costs, especially when storage accounts are within the same Azure region.

AzCopy Commands for Copying Blobs Between Storage Accounts¶

AzCopy simplifies the process of copying blobs with straightforward commands. The authentication method you choose will dictate the specific command structure. Two primary authentication methods are commonly used: Microsoft Entra ID and Shared Access Signature (SAS) tokens.

Using Microsoft Entra ID for Authorization¶

When employing Microsoft Entra ID for authorization, your Azure account’s identity is used to verify access to both the source and destination storage accounts. This approach is generally recommended for enhanced security and centralized access management. To utilize Microsoft Entra ID, ensure your account possesses the necessary role assignments on both the source and destination storage accounts. Typically, roles like “Storage Blob Data Contributor” or “Storage Account Contributor” are required to perform copy operations.

The AzCopy command structure when using Microsoft Entra ID is as follows:

azcopy copy 'https://<source-storage-account-name>.blob.core.windows.net/<container-name>/<blob-path>' 'https://<destination-storage-account-name>.blob.core.windows.net/<container-name>/<blob-path>'

In this command:

• <source-storage-account-name>: Replace with the name of your source storage account.
• <container-name>: Specify the container within the source storage account containing the blob(s) to be copied.
• <blob-path>: Indicate the path to the specific blob or directory of blobs you intend to copy.
• <destination-storage-account-name>: Replace with the name of your destination storage account.
• The second set of <container-name> and <blob-path> specifies the destination container and optional path within the destination storage account.

When this command is executed, AzCopy will authenticate your Microsoft Entra ID credentials and initiate the server-side copy operation.

Utilizing Shared Access Signature (SAS) Tokens for Authorization¶

Alternatively, you can use Shared Access Signature (SAS) tokens to grant temporary and restricted access to your storage resources. SAS tokens are URIs that contain an authorization query string, granting specific permissions for a defined period. This method is useful when you need to delegate copy operations to users or services without granting them full access to your storage account.

To use SAS tokens, you need to generate SAS tokens for both the source and destination storage accounts, granting at least “Read” permission on the source and “Write” permission on the destination. You then append these SAS tokens to the respective URLs in your AzCopy command.

The AzCopy command structure when using SAS tokens is as follows:

azcopy copy 'https://<source-storage-account-name>.blob.core.windows.net/<container-name>/<blob-path><SAS-token>' 'https://<destination-storage-account-name>.blob.core.windows.net/<container-name>/<blob-path><SAS-token>'

In this command:

• <SAS-token>: Append the generated SAS token query string to the end of both the source and destination blob URLs.

Remember to generate SAS tokens with the principle of least privilege, granting only the necessary permissions and setting an appropriate expiry time for security best practices.

For comprehensive details and advanced AzCopy functionalities, refer to the official Azure documentation on “Copy blobs between Azure storage accounts with AzCopy v10”.

Copying Blobs Between Storage Accounts with Access Restrictions¶

In real-world scenarios, security is paramount. Organizations often implement network restrictions using storage firewalls to control access to their Azure storage accounts. These firewalls can be configured to allow access only from specific IP addresses, virtual networks, or Azure services. When both source and destination storage accounts are protected by such firewalls, copying blobs using AzCopy requires careful consideration of the network configurations.

The challenge arises because server-side copy operations between storage accounts, by default, utilize private IP addresses, which are dynamic and not directly controllable from the client initiating the AzCopy command. This can lead to “403 Forbidden” errors if the destination storage account’s firewall rules do not explicitly allow access from the source storage account’s private IP range.

However, Azure Storage intelligently handles this scenario through a mechanism that leverages the client’s network access. Let’s explore how this works in detail.

Scenario 1: Client Accessing Storage Accounts via Public Endpoints¶

In the simplest scenario, the client machine running AzCopy (e.g., your local computer or an Azure VM with a public IP) accesses both the source and destination storage accounts through their public endpoints. In this case, to ensure successful blob copying when firewalls are in place, you must add the client’s public IP address to the firewall allowlist of both the source and destination storage accounts.

Let’s break down the detailed process of how this mechanism operates:

1. Client Initiates Copy: The client executes the AzCopy command, sending a PutBlockFromURL request to the destination storage account. This request essentially instructs the destination storage to fetch data blocks from the source storage account.

2. Destination Storage Attempts to Access Source: The destination storage account, upon receiving the PutBlockFromURL request, attempts to access the source storage account to retrieve the requested data blocks using its own internal network connection (potentially private IP).

3. Firewall Block (Initial): If the source storage account’s firewall is configured to restrict access and doesn’t explicitly allow the destination storage’s private IP range (which is typically the case initially), the destination storage receives a “403 Forbidden” error when trying to access the source.

4. Client-Mediated Access: Upon encountering the “403 Forbidden” error, the destination storage account intelligently falls back to a client-mediated approach. It sends a GetBlob request to the source storage account on behalf of the client. This is crucial because this request originates from the client’s network and thus uses the client’s public IP address.

5. Firewall Allow (Client IP): Since the client’s public IP address has been added to the source storage account’s firewall allowlist, this GetBlob request from the client is successful, and the source storage account allows access.

6. Data Transfer via Client’s Path: The destination storage account now successfully retrieves the data blocks from the source storage account, effectively utilizing the client’s network path for the data transfer.

7. Commit Blocks: After receiving the data blocks, the destination storage account returns a success response to the client. The client then sends a PutBlockList request to the destination storage to commit the received blocks and finalize the blob copy operation.

This clever mechanism ensures that even with firewalls in place, as long as the client machine initiating AzCopy has public network access and its public IP is whitelisted on both storage accounts, the blob copy operation can proceed successfully.

Copying Blobs in Hub-Spoke Architecture with Private Endpoints¶

More complex network architectures, such as the hub-spoke model, introduce additional considerations for secure blob copying. In a hub-spoke architecture, resources are typically deployed across multiple spoke virtual networks (VNets) that are connected to a central hub VNet. Storage accounts might be secured with private endpoints, restricting access to only within their respective VNets.

A common issue arises when you attempt to copy blobs between storage accounts connected to private endpoints in different spoke VNets from a virtual machine (VM) located in the hub VNet. In this scenario, you may encounter a “403 This request is not authorized to perform this operation - CannotVerifyCopySource” error in AzCopy logs or Azure Storage logs.

This error occurs because, in this configuration, the destination storage account, when attempting to access the source storage account, is effectively trying to cross VNet boundaries without proper network connectivity established. Private endpoints restrict access to within their VNet, and the default server-side copy operation is blocked.

To resolve this, several workarounds can be implemented to facilitate secure blob copying in hub-spoke architectures.

Workaround 1: Create a Private Endpoint for the Destination Storage Account in the Source VNet¶

One effective solution is to create a private endpoint for the destination storage account within the source VNet. This establishes a private link connection from the source VNet to the destination storage account, enabling secure and private network communication between them.

By implementing this workaround:

• The VM in the hub VNet initiates the AzCopy command.
• The destination storage account, when attempting to access the source, can now leverage the newly created private endpoint in the source VNet.
• Network traffic between the source and destination storage accounts remains within the Azure private network, ensuring security and compliance.
• The “403 Forbidden” error is resolved, and AzCopy can successfully copy blobs.

This approach is generally recommended as it maintains a high level of security by utilizing private endpoints and keeping network traffic within the Azure backbone.

Workaround 2: Place the VM in the Source VNet and Configure VNet Peering¶

Another viable option is to relocate the VM running AzCopy to the same VNet as the source storage account. Furthermore, you need to configure virtual network peering directly between the source VNet and the destination VNet. Crucially, this peering must be direct between the two spoke VNets and not routed through the hub VNet.

By implementing this workaround:

• The VM, now residing in the source VNet, has direct private network access to the source storage account via its private endpoint (if configured).
• The VNet peering established between the source and destination VNets enables private network connectivity between the source and destination storage accounts.
• AzCopy commands executed from the VM can now successfully copy blobs, leveraging the private network paths established by VNet peering.

It’s essential to ensure that the VNet peering is correctly configured and allows traffic flow between the source and destination VNets. Avoid relying on hub VNet transit for peering in this scenario, as it might not facilitate the necessary direct communication paths for storage account access. Refer to “Virtual network peering FAQ” for detailed information on VNet peering configurations.

Workaround 3: Utilize a Temporary Staging Storage Account¶

In situations where modifying existing network configurations or creating private endpoints is not feasible due to organizational constraints or complexities, a temporary staging storage account can serve as an effective intermediary for data transfer.

The process involves the following steps:

1. Create a Staging Account: Deploy a temporary storage account within the same Azure region as both the source and destination storage accounts. This minimizes data transfer latency and costs between the staging account and the other storage accounts.

2. Copy from Source to Staging: Use AzCopy to transfer the blobs from the source storage account to the newly created temporary staging storage account. Ensure that the staging account is accessible from the client environment running AzCopy (e.g., public endpoint access or appropriate firewall rules).

3. Copy from Staging to Destination: Configure a private endpoint for the staging storage account within the same VNet as the destination storage account. This establishes a private network connection between the staging and destination accounts. Then, use AzCopy to copy the blobs from the staging storage account to the final destination storage account. This transfer will now leverage the private endpoint for secure and private network communication.

4. Cleanup (Optional): After successful data transfer, you can delete the temporary staging storage account to optimize resource utilization and costs.

This workaround provides a flexible approach when direct private network connectivity between source and destination storage accounts is challenging to establish. However, it involves an additional step and might slightly increase the overall transfer time due to the intermediate staging process.

Workaround 4: Employ a VM as an Intermediary for Download and Upload¶

As a last resort, particularly when network configuration changes are highly restricted and other workarounds are impractical, you can utilize a virtual machine (VM) as an intermediary to download data from the source storage account and then upload it to the destination storage account.

The process is as follows:

1. Provision a VM: Create an Azure VM with sufficient size, processing power, and disk capacity to accommodate the data being transferred. Ensure this VM has network access to both the source and destination storage accounts (e.g., public IP or appropriate VNet integration and firewall rules).

2. Download from Source to VM: Use AzCopy on the VM to download the blobs from the source storage account to the VM’s local storage (disk).

3. Upload from VM to Destination: Once the download is complete, use AzCopy on the same VM to upload the downloaded blobs from the VM’s local storage to the destination storage account.

4. Cleanup (Optional): After successful data transfer, you can deallocate or delete the VM to manage costs.

It is crucial to emphasize that this workaround should be considered only when other methods are not feasible. It introduces significant overhead due to the download and upload steps via the VM, which can be considerably slower and more resource-intensive compared to server-side copy operations. Network bandwidth limitations of the VM and the client-side transfer process can also impact performance. Furthermore, security considerations related to data transit through the VM and temporary storage on the VM’s disk must be carefully evaluated.

Security Considerations for Blob Copy Operations¶

When copying blobs, especially in restricted network environments, security should be a paramount concern. Consider the following best practices:

• Principle of Least Privilege: When using SAS tokens, grant only the necessary permissions (Read on source, Write on destination) and set appropriate expiry times. For Microsoft Entra ID, assign the least privileged roles required for the copy operation.

• Network Segmentation: Utilize private endpoints and VNet peering to maintain network segmentation and restrict access to storage accounts to authorized networks. Avoid relying solely on public endpoints for sensitive data transfers.

• Data Encryption: Ensure that both source and destination storage accounts are configured with appropriate encryption settings (e.g., Storage Service Encryption with Microsoft-managed keys or customer-managed keys) to protect data at rest. Data in transit is typically encrypted using HTTPS during AzCopy transfers.

• Monitoring and Logging: Enable Azure Storage logging and monitor AzCopy operations to track data transfers, identify potential security incidents, and troubleshoot issues. Analyze logs for any unauthorized access attempts or errors.

• Regular Security Audits: Conduct periodic security audits of your storage account configurations, firewall rules, and access control policies to ensure they are aligned with your organization’s security standards.

Performance Optimization for AzCopy Transfers¶

To maximize the performance of your AzCopy blob copy operations, consider these optimization tips:

• Choose the Right Azure Region: When possible, locate both source and destination storage accounts within the same Azure region to minimize latency and data transfer costs.

• Utilize Parallelism: AzCopy is designed for parallel data transfer. By default, it automatically adjusts the number of concurrent operations based on available resources. You can fine-tune parallelism settings using AzCopy parameters like --parallel-level and --block-size for advanced optimization.

• Increase Block Size (for large blobs): For very large blobs, increasing the block size used by AzCopy can improve throughput. Experiment with different block sizes using the --block-size-mb parameter.

• Optimize Network Connectivity: Ensure that the network path between the client running AzCopy and the storage accounts, and between the storage accounts themselves (in server-side copy scenarios), is optimized for bandwidth and latency. Utilize private network connections where possible for enhanced performance and security.

• Avoid Unnecessary Operations: Carefully plan your AzCopy commands to copy only the necessary blobs and containers. Use filters and include/exclude patterns to refine the scope of the copy operation and reduce unnecessary data transfer.

• Monitor AzCopy Performance: Utilize AzCopy’s built-in progress reporting and logging to monitor transfer speeds and identify any bottlenecks. Analyze performance metrics to fine-tune your AzCopy commands and network configurations.

By understanding the nuances of AzCopy, network configurations, and security best practices, you can effectively and securely transfer Azure blobs between storage accounts, even in complex and restricted environments.

We encourage you to share your experiences and questions regarding AzCopy and secure blob transfer in the comments below! Your insights can be valuable to the wider community.