AzCopy Troubleshooting: Solve Azure Storage Issues Quickly and Efficiently

Table of Contents

Using AzCopy is an efficient way to transfer data to and from Azure Storage, but like any powerful tool, it can sometimes present challenges. This article delves into common issues encountered during AzCopy operations, providing insights into their underlying causes and offering practical solutions. Understanding these problems will enable you to diagnose and resolve your Azure Storage transfer issues swiftly and effectively.

Identifying Issues Through Exit Codes¶

The first step in troubleshooting any AzCopy job is to examine its exit code, which provides immediate feedback on the operation’s outcome. This code is a crucial indicator of success or failure, guiding your subsequent diagnostic steps. Interpreting these codes correctly helps narrow down the problem area significantly.

An exit code of 0-success signifies that the AzCopy job completed successfully without any errors. This indicates all specified files and data were transferred as intended, and no further action is typically required. Conversely, an exit code of 1-error points to a failed job, necessitating a deeper investigation into the log file. The log file contains detailed error messages that are essential for pinpointing the exact cause of the failure and formulating a solution.

If you encounter an exit code of 2-panic, it indicates a more severe internal error within AzCopy. In such cases, it’s critical to check if a log file was even generated. If no log file exists, this might suggest a fundamental software issue, warranting a bug report or direct support engagement. Any other non-zero exit code, such as OOMKilled, is usually system-generated, pointing to operating system-level problems like insufficient memory. For these, consulting your operating system’s documentation for specific error code meanings is recommended.

Understanding “403” Errors¶

“403” Forbidden errors are a frequent occurrence in AzCopy logs and can be both benign or indicative of a critical problem. It’s important to distinguish between these two scenarios to avoid unnecessary troubleshooting. Often, a “403” might appear during a HEAD request when AzCopy is merely checking if a resource is publicly accessible, which can be safely ignored if the transfer ultimately succeeds.

However, when “403” errors lead to failed transfers, they signal underlying authentication, authorization, or network access issues. These persistent errors will prevent data transfer attempts from succeeding until the root cause is addressed. Resolving these typically involves reviewing security configurations, such as Shared Access Signatures (SAS) tokens, Azure Role-Based Access Control (RBAC) assignments, Access Control Lists (ACLs), or storage account firewall settings.

Authentication and Authorization Issues¶

When “403” errors prevent data transfer, authentication and authorization are often the primary culprits. These issues stem from incorrect permissions or identity configurations, preventing AzCopy from gaining the necessary access to your Azure Storage resources. Properly configuring SAS tokens, Azure RBAC roles, or ACLs is paramount for successful operations.

SAS Tokens¶

Shared Access Signature (SAS) tokens are a critical security mechanism for granting limited access to storage resources. If you are using a SAS token with AzCopy and encounter “403” errors, several factors warrant investigation. First, verify the expiration and start times of your SAS token to ensure it is still valid and within its operational window. An expired or prematurely activated token will inevitably lead to access denied errors.

Second, confirm that all necessary permissions (e.g., Read, Write, List, Delete) have been explicitly selected for the token, aligning with the type of operation AzCopy needs to perform. Forgetting a crucial permission, such as “Write” for an upload, will result in transfer failures. Lastly, always generate your SAS tokens using official Azure SDKs, the Azure portal, or tools like Azure Storage Explorer to ensure their correctness and integrity. Using Storage Explorer is an excellent way to visually confirm the permissions and expiry of your generated token.

Azure RBAC¶

Azure Role-Based Access Control (RBAC) provides fine-grained access management to Azure resources. If you are authenticating with AzCopy using the azcopy login command, which leverages your Azure AD identity, ensure that the appropriate Azure roles are assigned to your identity. For instance, to upload or download blobs, your identity typically requires the Storage Blob Data Contributor role or a custom role with equivalent permissions.

For read-only operations, the Storage Blob Data Reader role might suffice. These roles define what actions your identity can perform on the storage account’s data plane. Inadequate role assignments will lead to “403” errors, as AzCopy will be blocked from performing authorized actions. Verifying these role assignments within the Azure portal under the storage account’s “Access control (IAM)” blade is a key troubleshooting step.

ACLs¶

Access Control Lists (ACLs) are specifically used in Azure Data Lake Storage Gen2 to manage permissions at the file and directory level. If your AzCopy operation targets a Data Lake Storage Gen2 account and you’re encountering “403” errors, examine the ACLs for the specific files or directories involved. Your identity must appear in an ACL entry with the appropriate permission level (Read, Write, Execute).

For directories, “Execute” permission is crucial to traverse them, while “Read” and “Write” are necessary for interacting with files or creating new ones. ACLs work in conjunction with Azure RBAC, where RBAC roles grant broad access, and ACLs provide more granular control. Understanding the access control model, including how RBAC and ACLs are evaluated together, is essential for correctly configuring permissions in Data Lake Storage Gen2 scenarios.

Firewall and Private Endpoint Issues¶

Azure Storage firewalls provide network-level access control, restricting which networks can connect to your storage account. If the AzCopy hosting component (a physical computer, VM, or container) is running from a network location not explicitly allowed by the storage account’s firewall configuration, AzCopy operations will return an HTTP “403” error code. This is a common and straightforward cause of access issues.

Azure Private Endpoints enhance security by providing private connectivity to your storage account from a virtual network. If you intend to use private endpoints, it’s crucial that AzCopy makes its requests from a VM within the same virtual network or subnet where the Private Link is configured. Requests originating from outside this private network will not use the Private Link, potentially leading to “403” errors if public access is restricted.

Permitted Scope for Copy Operations¶

The AllowedCopyScope property of a storage account introduces an additional layer of control over cross-account copy operations, particularly for security and data governance. This setting, visible as Permitted scope for copy operations (preview) in the Azure portal, dictates the environments from which data can be copied to the destination account. By default, this property is null, permitting copies from any storage account.

Value	Description
(null) (Default)	Allows copying from any storage account, regardless of its Microsoft Entra tenant or network connectivity to the destination account.
Microsoft Entra ID	Restricts copy operations to only those accounts that belong to the same Microsoft Entra tenant as the destination storage account. This enhances security by ensuring tenant-level isolation for data movement.
PrivateLink	Permits copying exclusively from storage accounts that have private links established to the same virtual network as the destination account. This enforces network-level isolation for copy operations.

If this property is explicitly set, an AzCopy operation attempting to copy data from an unpermitted scope will fail with a “403” error. For instance, if AllowedCopyScope is set to Microsoft Entra ID, a copy from an account in a different tenant will be blocked. Similarly, PrivateLink requires both source and destination to be privately linked to the same virtual network environment where AzCopy is running. Always verify this setting in the destination storage account’s configuration if you are performing cross-account transfers.

Transfer Data From or To a Local Hosting Component¶

When performing upload or download operations between an Azure Storage account and a local hosting component, network access must be explicitly granted. The public IP address of the machine running AzCopy needs to be whitelisted within the firewall settings of the relevant storage account (either the source for downloads or the destination for uploads). This ensures that the storage account’s firewall allows inbound connections from your local environment.

To configure this, navigate to the storage account in the Azure portal, go to the “Networking” section under “Security + networking,” and add your public IP address under the “Firewalls and virtual networks” tab. Failing to include your IP address will result in “403” errors, as the storage account will block all access attempts from unrecognized network locations.

Transfer Data Between Storage Accounts¶

Transferring data directly between two Azure Storage accounts using AzCopy also requires careful consideration of network access. In this scenario, the hosting component running AzCopy acts as an orchestrator, initiating calls to both the source and destination accounts. Therefore, the public IP address of this AzCopy client must be permitted access by the firewall configurations of both the source and destination storage accounts.

Even though the data transfer often occurs service-to-service (meaning the data doesn’t physically flow through the AzCopy client machine), the client still needs to establish authorized connections to both endpoints to set up and manage the transfer. If your virtual machine lacks a public IP address or if security policies prevent exposing it, leveraging private endpoints for both storage accounts is a secure alternative. Private endpoints ensure that all traffic remains within your private network, bypassing the need for public IP whitelisting.

Use Private Link¶

Azure Private Link provides private connectivity from a virtual network to Azure PaaS services, including Azure Storage. For AzCopy requests to utilize a Private Link, the AzCopy tool must be executed from a virtual machine (VM) located within the same virtual network or even the specific subnet where the Private Link is configured. This is a critical architectural requirement for Private Link functionality.

If, for example, your Private Link configuration is within VNet1/Subnet1, but the VM running AzCopy resides in VNet1/Subnet2, AzCopy’s requests will not automatically route through the Private Link. Instead, they will attempt to use the public endpoint of the storage account. If the storage account’s public access is restricted by a firewall, these requests will likely fail with a “403” error, even if a Private Link exists in a different subnet of the same VNet. Always ensure network alignment for Private Link scenarios.

Proxy-Related Errors¶

Proxy-related errors typically manifest as TCP connection failures, such as “dial tcp: lookup proxy.x.x: no such host.” These errors indicate that AzCopy is unable to connect to the configured proxy server, often due to incorrect proxy settings or an unsupported proxy configuration. Resolving these issues involves ensuring AzCopy can correctly identify and communicate with your network’s proxy.

You must update AzCopy’s proxy settings to reflect the accurate configurations of your network. AzCopy can be configured to use a proxy through environment variables or its configuration file. Alternatively, if your environment allows and it’s suitable for your security posture, you can bypass the proxy altogether by setting the environment variable NO_PROXY="*" to allow direct connections.

AzCopy requires access to specific endpoints for both authentication and storage operations, depending on your Azure cloud environment. Ensure your proxy configuration permits traffic to these essential addresses:

Sign-in Endpoints	Azure Storage Endpoints
login.microsoftonline.com	(blob | file | dfs).core.windows.net
login.chinacloudapi.cn	(blob | file | dfs).core.chinacloudapi.cn
login.microsoftonline.de	(blob | file | dfs).core.cloudapi.de
login.microsoftonline.us	(blob | file | dfs).core.usgovcloudapi.net

For Azure Storage endpoints, the (blob | file | dfs) component represents the specific service endpoint being accessed (e.g., blob.core.windows.net for blob storage). Ensuring these endpoints are reachable through your proxy, or directly if the proxy is bypassed, is crucial for successful AzCopy operations.

x509: Certificate Signed by Unknown Authority¶

The error message “x509: certificate signed by unknown authority” is a common indication of Secure Sockets Layer (SSL) certificate validation failure. This typically occurs when AzCopy encounters an SSL certificate presented by a proxy or an intermediate network device that is not trusted by the operating system where AzCopy is running. Many corporate networks employ SSL inspection proxies, which re-sign certificates, leading to this error if the proxy’s root certificate is not installed on the client machine.

To resolve this, you need to ensure that the SSL certificate used by your proxy is trusted at the operating system level. The recommended approach is to add the proxy’s root certificate to your hosting component’s trusted root certificate store. This action allows the operating system, and by extension AzCopy, to validate the integrity of the certificate chain presented during the SSL handshake. Once the certificate is properly installed and trusted, AzCopy will be able to establish secure connections without encountering this validation error.

Unrecognized Parameters¶

Encountering an “unrecognized parameters” error message suggests a mismatch between the AzCopy version you are using and the syntax of your command. This is a particularly common issue for users transitioning from older versions of AzCopy. AzCopy v8 and earlier versions are deprecated, and AzCopy v10 represents a complete rewrite with a significantly different command syntax. Attempting to use v8 syntax with a v10 installation, or vice versa, will inevitably lead to these errors.

It is crucial to verify that you are using AzCopy v10, which is the current and actively supported version. Familiarize yourself with its syntax by utilizing the built-in help messages, accessible via the -h switch appended to any command (e.g., azcopy copy -h). This will display command-specific options and examples directly in your terminal. For more comprehensive guidance, the AzCopy command guide provides a valuable resource demonstrating popular commands and their associated flags, aiding in constructing correct commands. Regularly checking the official documentation and GitHub issues can also provide solutions for specific parameter-related queries.

Conditional Access Policy Error¶

When invoking the azcopy login command, you might encounter an error message similar to: “AADSTS50005: User tried to log in to a device from a platform (Unknown) that’s currently not supported through Conditional Access policy.” This specific error indicates that your Azure Active Directory administrator has implemented a Conditional Access Policy (CAP) designed to restrict sign-in attempts based on device platform. AzCopy’s default authentication flow, the device code flow, cannot provide the necessary device context to satisfy these types of CAPs, as it cannot guarantee the identity of the physical device being used.

To bypass this limitation, several alternatives are available. Azure Storage Explorer, for example, integrates AzCopy for data transfers but offers a different sign-in workflow that is compatible with CAPs by providing device information. This allows it to successfully authenticate and pass tokens to AzCopy. Alternatively, for automated scripts or Azure-hosted resources, AzCopy supports authentication using Managed Identities or Service Principals, which are robust and secure methods that do not rely on interactive device-based login. If these alternatives are not suitable, or if you require specific platform access, contacting your Azure AD administrator to review and adjust the Conditional Access Policy might be necessary.

Server Busy, Network Errors, or Time-Outs¶

A high volume of failed requests accompanied by “503 Server Busy” statuses indicates that the Azure Storage service is actively throttling your requests. This occurs when the service detects an overwhelming number of operations, often to protect its resources and maintain stability for all users. Similarly, persistent network errors or time-outs suggest that your local infrastructure or network connection might be struggling to handle the volume of data being pushed or pulled. Both scenarios point towards resource contention or network saturation.

The primary workaround for these issues involves adjusting AzCopy’s performance settings. If a large file consistently fails to copy due to chunk failures, consider limiting the concurrent network connections or setting an explicit throughput limit. A recommended strategy is to drastically reduce performance settings initially to confirm if the underlying problem is resolved. Once stability is achieved, gradually increase these settings to find an optimal balance between speed and reliability.

For detailed guidance on optimizing AzCopy’s performance, consult the official documentation on performance tuning. When copying data between accounts, remember that while data transfer is often service-to-service, AzCopy initiates and manages these calls from your client machine. Therefore, the quality and reliability of the network from where you run AzCopy significantly impact the overall performance and stability of inter-account transfers. A poor client-side network connection can contribute to time-outs and errors, even for server-side copy operations.

Known Constraints in AzCopy¶

While AzCopy is a versatile tool, it operates within certain known constraints that are important to acknowledge during planning and troubleshooting. Understanding these limitations can prevent unexpected behavior and guide you toward alternative solutions when necessary.

• Cross-Cloud Copy Restrictions: Copying data from government-specific clouds (e.g., Azure Government, Azure China) to commercial Azure clouds is generally not supported due to the isolated nature of these environments. However, transferring data from commercial clouds to government clouds is typically permitted.
• Synchronous Copy Operations Only: AzCopy performs synchronous copy operations. This means that when an AzCopy job reports completion, the data transfer has fully concluded, and the data is already moved to its destination. It does not support asynchronous service-side copy operations where the job might finish while data transfer continues in the background.
• SMB Permissions with Azure File Shares: When copying data to an Azure File share, forgetting to specify the --preserve-smb-permissions flag can result in permissions not being transferred. If you realize this oversight after the data transfer, and you do not wish to re-transfer the entire dataset, tools like Robocopy can be used specifically to synchronize or apply SMB permissions to the existing files.
• Azure Functions MSI Authentication: Currently, AzCopy does not natively support Managed Service Identity (MSI) authentication for Azure Functions. This means that if you are running AzCopy within an Azure Function, you may need to use alternative authentication methods like service principals or SAS tokens instead of MSI.

Engage with Us¶

We hope this comprehensive guide helps you quickly diagnose and resolve your AzCopy troubleshooting challenges. Azure Storage and AzCopy are constantly evolving, and community feedback is invaluable. If you have encountered a unique issue, discovered an innovative solution, or have further questions, please share your experiences. Your insights help improve the collective understanding and efficiency for all users. Feel free to comment below with your thoughts or any additional tips you might have!