Troubleshooting Image Pull Failures from ACR to Azure Web App

Table of Contents

Deploying containerized applications to Azure Web App directly from Azure Container Registry (ACR) is a common and efficient practice. However, users occasionally encounter issues when the Azure Web App attempts to pull the required image from ACR. These failures can stem from various configurations related to authentication, image availability, or network access. Understanding the potential causes and knowing how to diagnose them effectively is crucial for maintaining smooth deployment pipelines.

This article provides guidance on troubleshooting the most common errors encountered during the image pull process from ACR to Azure Web App. We will explore typical symptoms, delve into specific error messages, and offer detailed solutions to help you resolve these issues efficiently.

Symptoms and Initial Troubleshooting Steps¶

The first indication of an image pull failure is often observed during the deployment or scaling operation of your Azure Web App. The app instance might fail to start, reporting issues related to fetching the container image. The application logs and the deployment logs are the primary sources of information for diagnosing these problems.

Before diving into specific error messages, a good starting point is to check the overall health of your Azure Container Registry. ACR provides a health check command that can help identify potential issues with the registry service itself, network connectivity, or permissions.

To perform a basic health check on your ACR instance, you can use the Azure CLI. Execute the following command, replacing <myregistry> with the name of your Azure Container Registry:

az acr check-health --name <myregistry> --ignore-errors --yes

This command performs several checks, including connectivity to the registry, login server resolution, and repository access. The output will provide insights into the health status and report any detected issues with corresponding error codes and descriptions. If the health check reports any problems, refer to the ACR health check error reference documentation for specific solutions related to those error codes. Note that some reported errors, like those related to Helm or Notary, might not directly impact basic image pulling unless your deployment specifically relies on those features.

Once you have confirmed the basic health of the registry, the next crucial step is to identify the specific error message generated by the Azure Web App during the image pull attempt. This message provides the most direct clue to the root cause of the failure.

You can access the deployment logs and log stream through the Azure portal:

1. Sign in to the Azure portal.
2. Navigate to your Azure Web App resource.
3. In the left-hand menu, under the “Deployment” section, select Deployment Center.
4. Here, you can view past deployment attempts and access detailed logs for each one. Look for the specific deployment that failed during the image pull phase.
5. Alternatively, for real-time insights, go to the “Monitoring” section and select Log stream. The log stream provides live updates on the container startup process, including image pull attempts and any errors encountered.

The error messages you find in these logs will guide you to the specific troubleshooting steps outlined in the following sections. We will cover the most frequently encountered errors and their respective solutions.

Error 1: Unauthorized¶

One of the most common issues is an authorization failure, indicating that the Azure Web App does not have the necessary permissions or correct credentials to access the specified image in the Azure Container Registry. The error message typically looks similar to this:

Head \" https://<acr-name>.azurecr.io/v2/<repository>/manifests/<tag>\\\": unauthorized

This error explicitly states that the request to access the image manifest was denied due to lack of authorization. This can happen with either of the two primary authentication methods used by Azure Web App to pull images from ACR: Admin user credentials or Managed Identity.

Solution for Admin User-Based Authentication¶

If your Azure Web App is configured to authenticate with ACR using the admin user account credentials, the “unauthorized” error almost always points to incorrect login details provided to the web app. You need to ensure that the login server address, username, and password configured in the web app’s environment variables precisely match the admin user credentials enabled in your ACR instance.

Here’s how to verify and correct the configuration:

1. In the Azure portal, navigate to your Azure Container Registry resource.
2. In the left-hand menu, under the “Settings” section, select Access keys.
3. On the “Access keys” page, ensure that “Admin user” is enabled. Note down the Login server address, the Username, and one of the provided passwords. It’s crucial to use the exact values displayed here.
4. Now, navigate to your Azure Web App resource in the Azure portal.
5. In the left-hand menu, under the “Settings” section, select Environment variables.
6. Review the environment variables configured for your container deployment. The Web App uses specific variable names to pull images from a private registry. Commonly used variables might include DOCKER_REGISTRY_SERVER, DOCKER_REGISTRY_USERNAME, and DOCKER_REGISTRY_PASSWORD. Verify that the values assigned to these variables exactly match the Login server, Username, and password you obtained from the ACR “Access keys” page.

Any discrepancy in the login server address (even subtle typos), username, or password will result in the unauthorized error. Ensure case sensitivity is respected, especially for the username and password. If you update the environment variables, save the changes and restart the web app to apply the new configuration.

Solution for Managed Identity-Based Authentication¶

A more secure and recommended approach for authentication is using Azure Web App’s Managed Identity. A Managed Identity provides an automatically managed identity for your Azure resource, allowing it to authenticate to cloud services (like ACR) without requiring explicit credentials in code or configuration. When using Managed Identity for ACR image pulls, the identity assigned to the web app must have the necessary permissions to perform the pull operation.

The specific data action required to pull an image is Microsoft.ContainerRegistry/registries/pull/read. Azure defines built-in roles that include this permission. The most relevant roles that grant image pull access are:

• AcrPull: This role grants permissions to pull images from the registry. This is the principle of least privilege for pulling images.
• AcrPush: This role grants permissions to push and pull images.
• Reader: This role grants read access to resource management actions, which includes the ability to pull images.
• Contributor: This role grants full access to manage all resources, including pulling images.
• Owner: This role grants full access to manage all resources and delegate access.

For image pulling, the AcrPull role is the most appropriate as it grants only the necessary permissions.

When you configure an Azure Web App to use a Managed Identity (either system-assigned or user-assigned) to pull images from ACR through the Deployment Center, the Azure platform typically attempts to automatically assign the AcrPull role to the Web App’s identity on the specified ACR instance. This automatic assignment simplifies configuration significantly.

However, the “unauthorized” error can still occur with Managed Identity for a few reasons:

1. Role Assignment Blocked: An Azure Policy might be in place within your subscription or management group that prevents the creation of role assignments. If the automatic AcrPull assignment is blocked, the identity will not have the required permissions.
2. Role Assignment Deleted: The AcrPull role assignment might have been manually removed from the ACR instance after the Web App was configured.
3. Incorrect Identity Used: Ensure the Web App is configured to use the correct Managed Identity, and that identity is the one targeted by the role assignment on the ACR.

To verify the role assignment:

1. In the Azure portal, navigate to your Azure Container Registry resource.
2. In the left-hand menu, select Access control (IAM).
3. Click on the Role assignments tab.
4. Filter the list by the AcrPull role and the scope (This resource).
5. Look for an entry where the “Principal” is the Managed Identity of your Azure Web App. If the assignment is missing or incorrectly configured, you may need to manually add it. Click “Add” -> “Add role assignment”, select the AcrPull role, and select your Web App’s Managed Identity as the principal.

If using a user-assigned managed identity, ensure the identity itself exists and is assigned to the web app resource under “Identity” settings.

Managed Identity is generally preferred over Admin user authentication as it eliminates the need to manage secrets (passwords) directly, reducing the risk of exposure. If you are using Managed Identity and encounter the unauthorized error, verifying the role assignment is the primary troubleshooting step.

Error 2: Manifest tagged by is not found¶

This error message is quite specific and usually points directly to an issue with the image reference being used by the Azure Web App. The error indicates that while the registry was accessible, the specific image manifest identified by the combination of repository name and tag could not be located within that registry. The error message might look like this:

DockerApiException  : Docker API responded with status code=NotFound, response={"message":"manifest for <acr-name>.azurecr.io/<repository>:<tag> not found: manifest unknown: manifest tagged by \"<tag>\" is not found"}

This message confirms that the registry (<acr-name>.azurecr.io) was reached, and the repository (<repository>) was likely found, but the specific version or tag (<tag>) of the image within that repository does not exist.

Solution: Make sure the tag exists¶

The solution to this error is straightforward: verify that the image repository and the specific tag you are trying to pull actually exist in your Azure Container Registry and that the Web App configuration uses the correct, case-sensitive names.

You can check the available repositories and tags using either the Azure CLI or the Azure portal.

Using Azure CLI:

Run the following command, replacing <ContainerRegistryName> with your ACR name and <RepositoryName> with the name of the repository you are trying to pull from:

az acr repository show-tags -n <ContainerRegistryName> --repository <RepositoryName>

This command will list all the tags available for the specified repository in your ACR. Check if the tag mentioned in the error message is present in this list.

Using the Azure portal:

1. In the Azure portal, navigate to your Azure Container Registry resource.
2. In the left-hand menu, under the “Services” section, select Repositories.
3. This page lists all repositories within your registry. Find and select the <RepositoryName> mentioned in the error message.
4. Once inside the repository view, you will see a list of all the tags associated with that repository. Verify that the <tag> from the error message is listed here.

If the repository or tag is not found using these methods, it means:

• The image was never successfully built and pushed to this specific repository/tag in ACR.
• The repository name or tag name used in the Azure Web App configuration (in the Deployment Center or environment variables) contains a typo. Remember that repository and tag names are case-sensitive.
• The image or tag might have been accidentally deleted from the registry.
• The tag might have been overwritten during a recent push (if using mutable tags like latest, which is generally discouraged for production deployments).

Ensure that the image reference configured in your Azure Web App precisely matches an existing repository and tag in your ACR. If the image is missing, you will need to build and push it to the registry first. If there’s a typo, correct the Web App configuration and trigger a redeployment.

Error 3: Client with IP is not allowed access¶

This error indicates that the Azure Web App is being blocked by the network security settings configured on your Azure Container Registry. ACR has a built-in firewall that allows you to restrict access based on IP addresses or virtual networks. The error message typically includes the IP address of the client attempting the connection, which in this case is the outbound IP address of your Azure Web App.

DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Head \"https:// <acr-name>.azurecr.io/v2/<repository>/manifests/<tag>\": denied: client with IP '<web-app-outbound-ip>' is not allowed access. Refer https://aka.ms/acr/firewall to grant access."}

The error message clearly states that the client IP address from which the Web App is attempting to pull the image is being denied access by the ACR firewall. By default, ACR allows access from all public networks, but this can be configured to enhance security.

Solution 1: Make sure the container registry built-in firewall allows your device’s IP address (Public Access)¶

If your ACR has its public endpoint enabled and configured to restrict access to specific networks or IP addresses, you need to ensure that the outbound IP addresses used by your Azure Web App are included in the allowed list.

Azure Web Apps running on multi-tenant stamps have a set of dynamic outbound IP addresses. These are the IP addresses from which outbound network calls (like pulling images from ACR) originate. Since the specific IP can vary, you need to add the entire range of possible outbound IPs for your Web App to the ACR firewall rules.

Here’s how to find your Web App’s outbound IP addresses:

Using the Azure portal:

1. In the Azure portal, navigate to your Azure Web App resource.
2. On the Overview page, locate the “Outbound IP address” field. It initially shows a few IPs.
3. Click on Show More next to the listed IPs to display the full, potentially large list of all possible outbound IP addresses for your Web App’s stamp and region. You need to allow all of these in the ACR firewall.

Using Azure CLI:

You can also retrieve the outbound IP addresses using the Azure CLI. Refer to the Azure App Service documentation on finding outbound IPs for the specific command.

Once you have the full list of outbound IP addresses, configure your ACR firewall:

1. In the Azure portal, navigate to your Azure Container Registry resource.
2. In the left-hand menu, under the “Settings” section, select Networking.
3. On the “Networking” page, go to the Firewalls and virtual networks tab.
4. Under “Public access”, select the “Selected networks” option.
5. In the “Firewall” section, add each of the outbound IP addresses (or CIDR ranges, if applicable) obtained from your Web App to the list of allowed IP addresses.

Saving these changes will update the ACR firewall rules. Allow a few minutes for the changes to propagate. Your Web App should now be able to connect to ACR from its allowed outbound IP addresses.

Consideration: Adding the entire range of dynamic outbound IPs for a Web App can be tedious and potentially less secure if you need strict IP-based access control, as you are allowing access from any other resource on the same App Service stamp that might share those outbound IPs. A more robust solution, especially if you want to fully disable public access to ACR, is using private networking.

Solution 2: Configure Virtual Network Integration for the Web App and Private Endpoint for ACR (Private Access)¶

A more secure and scalable approach, particularly if you need to restrict ACR access to private networks only or have stringent security requirements, is to use Azure Private Link for ACR and Azure Web App VNet Integration. This allows the Web App to pull images from ACR over a private network connection, bypassing the public internet and the ACR public firewall.

This solution involves two main parts:

1. Azure Container Registry Private Endpoint: Configure a private endpoint for your ACR within a Virtual Network. This assigns a private IP address from your VNet to the ACR, making it accessible only from within that VNet or connected networks.
2. Azure Web App VNet Integration: Integrate your Azure Web App with the same Virtual Network (or a peered VNet) where the ACR private endpoint resides. This enables the Web App to send outbound traffic, including image pull requests, through the Virtual Network.

Assuming you have already configured an ACR Private Endpoint, here’s how to configure Virtual Network Integration for your Azure Web App:

1. In the Azure portal, navigate to your Azure Web App resource.
2. In the left-hand menu, under the “Settings” section, select Networking.
3. Under the “Outbound traffic configuration” section, locate Virtual Network integration.
4. Click Add virtual network integration.
5. Select the Virtual Network and a dedicated Subnet within that VNet where the integration will be configured. Ensure the VNet is the one hosting or peered with the VNet containing the ACR private endpoint. The subnet must have enough available IP addresses.
6. On the Virtual network configuration page that appears after selecting the VNet/subnet, make sure the Container image pull option is selected and enabled. This specific setting ensures that image pull requests are routed through the integrated VNet.

Once VNet integration is configured, you also need to explicitly tell the Web App to use this path for image pulls. In the Web App’s Deployment Center configuration:

1. Navigate back to the Deployment Center for your Web App.
2. Under the “Container Settings”, locate the option related to pulling images over the VNet. Ensure the Pull image over VNet option is enabled.

When pulling images over a VNet, especially if the ACR public endpoint is disabled or restricted, you might encounter an expected behavior in the Azure portal’s Deployment Center. When selecting the image and tag, the portal might display a message like “Failed to load ACR Tags - failed”. This happens because the portal UI, accessing ACR over the public internet, is blocked by the ACR firewall. This error is normal in this scenario and doesn’t prevent the Web App (which uses the private VNet path) from pulling the image. If this occurs, the drop-down options for Image and Tag won’t populate, and you will need to manually type in the exact repository name and tag.

Using VNet Integration and Private Endpoint is the recommended approach for production environments requiring enhanced security and predictable network paths for container image pulls.

Advanced Troubleshooting and Next Steps¶

If the troubleshooting steps above don’t resolve your image pull issues, there might be more complex network configurations at play or other underlying problems. Consider these advanced steps:

• Review Network Security Groups (NSGs) and Route Tables: If your VNet subnets have NSGs associated with them, review the inbound and outbound security rules. Ensure that there are no rules blocking traffic between the Web App integration subnet and the ACR private endpoint subnet (if they are different) or blocking outbound traffic from the integration subnet to the ACR’s private IP. Similarly, check if any custom route tables are directing traffic away from the intended VNet path.
• Check Firewalls or Virtual Appliances: If your VNet setup includes a network virtual appliance (like Azure Firewall or a third-party firewall appliance) controlling traffic between subnets or to private endpoints, examine its configuration and logs. The firewall might be blocking the image pull traffic.
• Utilize the Kudu Service: The Kudu service (or SCM site) for your Azure Web App is a powerful tool for debugging. You can access it by going to “Advanced Tools” -> “Go” in your Web App’s menu. From the Kudu console (Bash or CMD), you can perform network diagnostics from the perspective of your Web App instance.
  • Test DNS resolution for your ACR’s login server address (e.g., <acr-name>.azurecr.io) using commands like nslookup <acr-name>.azurecr.io. If using a private endpoint, ensure it resolves to the private IP address within your VNet.
  • Test connectivity to the ACR login server on the standard HTTPS port (443) using commands like tcpping <acr-name>.azurecr.io:443 (on Windows Kudu) or nc -vz <acr-name>.azurecr.io 443 (on Linux Kudu). This helps confirm if network paths are open.
• Examine Azure Activity Logs: Check the Azure Activity Log for your Web App and ACR resources. This log records control plane operations and can sometimes reveal policy violations, role assignment failures, or other events related to configuration changes that might impact image pulling.
• Review ACR Diagnostic Logs: Configure diagnostic settings for your ACR to send logs to a Log Analytics workspace. These logs can provide detailed information about incoming requests, including authentication attempts, and may offer more granular insight into why a pull request was denied.

By systematically reviewing authentication settings, verifying image availability, checking network configurations (firewall, VNet, NSGs, route tables), and leveraging debugging tools like Kudu, you can effectively pinpoint and resolve most image pull failures from Azure Container Registry to Azure Web App.

What has your experience been with troubleshooting ACR image pull issues in Azure Web App? Share your common challenges or unique solutions in the comments below!