Troubleshooting Terraform Azure: Resolving Policy Assignment Errors in Main.tf

Table of Contents

Infrastructure as Code (IaC) has revolutionized the way organizations manage their cloud resources. Tools like Terraform enable engineers to define, provision, and manage infrastructure using declarative configuration files. This approach brings significant benefits, including versioning, repeatability, and increased efficiency. However, working with complex cloud environments like Microsoft Azure and integrating various services can sometimes lead to unexpected issues during deployment. One common challenge encountered when managing Azure Policies with Terraform is the “Policy definition not found” error during policy assignment.

Azure Policy is a crucial service in Azure used to enforce organizational standards and assess compliance at scale. It allows you to govern your resources by defining rules that your resources must follow. These rules can enforce tagging conventions, restrict allowed resource types, or even ensure security configurations are correctly set. Azure Policy consists of several core components: Policy Definitions, Policy Set Definitions (also known as Initiatives), and Policy Assignments. A Policy Definition specifies the rule and the effect (e.g., Audit, Deny, DeployIfNotExists). A Policy Set Definition is a collection of Policy Definitions grouped together for a common goal. Finally, a Policy Assignment applies a Policy Definition or a Policy Set Definition to a specific scope, such as a subscription, resource group, or even a single resource.

When managing these Azure Policy components using Terraform, you define them as resources within your .tf configuration files. The azurerm_policy_definition, azurerm_policy_set_definition, and azurerm_policy_assignment resources in the Azure Provider for Terraform allow you to codify your governance rules. You define the properties of each component, such as the policy rule itself (often in JSON format), the display name, description, and crucially, the scope and the reference to the definition or set definition being assigned.

Symptoms: Encountering the “Policy definition not found” Error¶

A frequent issue arises when attempting to create a policy assignment using Terraform, especially when the policy definition or policy set definition it references is also being created or managed within the same or a dependent Terraform configuration. The symptom manifests during the terraform apply phase, where the operation fails with an error message similar to “Policy definition not found”.

Consider a typical Terraform configuration for creating an Azure Policy Assignment. It might look something like this:

resource "azurerm_policy_assignment" "kubernetes_assignment" {
  name                 = "enforce-kubernetes-standards"
  scope                = "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" # Replace with your scope
  policy_definition_id = azurerm_policy_set_definition.kubernetes_initiative.id # Reference to a Policy Set Definition
  description          = "Assigns the Kubernetes security initiative to the subscription."
  display_name         = "Kubernetes Security Initiative Assignment"

  # parameters = "{ ... }" # Optional: Policy parameters
}

resource "azurerm_policy_set_definition" "kubernetes_initiative" {
  name         = "kubernetes-initiative"
  policy_type  = "Custom"
  display_name = "Kubernetes Security Initiative"
  description  = "A set of policies for Kubernetes security."

  policy_definitions {
    policy_definition_id = "/providers/Microsoft.Authorization/policyDefinitions/your-first-policy-id"
    policy_definition_reference_id = "first-policy-ref"
  }

  policy_definitions {
    policy_definition_id = "/providers/Microsoft.Authorization/policyDefinitions/your-second-policy-id"
    policy_definition_reference_id = "second-policy-ref"
  }
  # ... more policy definitions
}

In this scenario, you are defining a policy set (azurerm_policy_set_definition) and then attempting to assign it (azurerm_policy_assignment). During terraform apply, Terraform determines the order in which to create these resources based on their dependencies. Ideally, Terraform should create the azurerm_policy_set_definition first, obtain its ID, and then use that ID when creating the azurerm_policy_assignment. However, the “Policy definition not found” error indicates that Terraform is attempting to create the azurerm_policy_assignment resource before the Azure control plane has fully registered and made the azurerm_policy_set_definition resource available, despite the apparent reference.

The Underlying Cause: Timing and Dependency Resolution¶

The root cause of this error lies in how Terraform resolves dependencies and the potential for timing issues or eventual consistency within the Azure API. Terraform builds a dependency graph based on the relationships between resources defined in your configuration. When one resource’s attribute references another resource (e.g., policy_definition_id = azurerm_policy_set_definition.kubernetes_initiative.id), Terraform usually understands that the referenced resource must be created before the dependent resource. This is known as an implicit dependency.

However, cloud APIs operate asynchronously and resources, once created, might take a short amount of time to be fully propagated and available across the entire platform. While Terraform waits for the API call to create the policy set definition resource to return successfully, there might still be a brief window before that resource’s ID is fully resolvable and usable by subsequent operations like creating an assignment, especially if the assignment operation hits a different backend service or replica that hasn’t received the update yet.

In some cases, particularly with certain resource types or complex configurations, Terraform’s implicit dependency detection might not be sufficient or might be perceived differently by the provider’s interaction with the API. The Azure Provider for Terraform is constantly updated to handle these nuances, but sometimes, manual intervention is required to enforce the correct order. The error “Policy definition not found” specifically points to the assignment failing because it cannot locate the definition (or set definition) it is trying to reference at the moment the assignment creation is attempted by the Azure API, even if Terraform believes the definition resource has been successfully created based on the immediate API response.

Deep Dive into Terraform Dependency Management¶

Terraform’s dependency resolution is a core function that determines the order of resource creation, update, and deletion. It primarily relies on:

1. Implicit Dependencies: These are automatically detected when one resource’s configuration refers to another resource’s attributes. For example, if a virtual machine’s network interface ID is set to azurerm_network_interface.main.id, Terraform knows that the network interface (azurerm_network_interface.main) must exist before the virtual machine. This is the preferred and most common method of managing dependencies as it keeps the configuration clean and allows Terraform to optimize the execution plan.

2. Explicit Dependencies (depends_on): This is a meta-argument that can be added to any resource block to explicitly declare that the resource depends on one or more other resources. This forces Terraform to wait for the specified resources to be created, updated, or destroyed before performing the same operation on the resource with the depends_on argument. The syntax is a list of resource addresses. For example:
   

   resource "azurerm_virtual_machine" "main" {
     # ... configuration ...
     depends_on = [
       azurerm_network_interface.main,
       azurerm_subnet.main
     ]
   }
   

   
   Explicit dependencies should be used sparingly and only when implicit dependencies are insufficient or when there are hidden dependencies not visible through resource attribute references (like the timing issue with the policy assignment). Overusing depends_on can make the configuration harder to read and maintain, and can sometimes hinder Terraform’s ability to parallelize operations, potentially slowing down deployments.

In the case of the “Policy definition not found” error during policy assignment, even though the azurerm_policy_assignment resource explicitly references the .id of the azurerm_policy_set_definition (or azurerm_policy_definition) resource, the implicit dependency might not be robust enough to account for the slight delay in the definition’s availability within the Azure control plane. This is where the explicit depends_on argument becomes necessary as a targeted solution.

The Solution: Enforcing Order with depends_on¶

To resolve the “Policy definition not found” error, the most reliable solution is to explicitly inform Terraform that the azurerm_policy_assignment resource must wait for the azurerm_policy_set_definition (or azurerm_policy_definition) resource it references to be fully available. This is achieved by adding the depends_on meta-argument to the azurerm_policy_assignment resource block.

Here is the modified Terraform configuration snippet demonstrating the solution:

resource "azurerm_policy_assignment" "kubernetes_assignment" {
  name                 = "enforce-kubernetes-standards"
  scope                = "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" # Replace with your scope
  policy_definition_id = azurerm_policy_set_definition.kubernetes_initiative.id # Reference to the Policy Set Definition
  description          = "Assigns the Kubernetes security initiative to the subscription."
  display_name         = "Kubernetes Security Initiative Assignment"

  # Explicitly declare dependency on the policy set definition
  depends_on = [
    azurerm_policy_set_definition.kubernetes_initiative
  ]

  # parameters = "{ ... }" # Optional: Policy parameters
}

resource "azurerm_policy_set_definition" "kubernetes_initiative" {
  name         = "kubernetes-initiative"
  policy_type  = "Custom"
  display_name = "Kubernetes Security Initiative"
  description  = "A set of policies for Kubernetes security."

  policy_definitions {
    policy_definition_id = "/providers/Microsoft.Authorization/policyDefinitions/your-first-policy-id"
    policy_definition_reference_id = "first-policy-ref"
  }

  policy_definitions {
    policy_definition_id = "/providers/Microsoft.Authorization/policyDefinitions/your-second-policy-id"
    policy_definition_reference_id = "second-policy-ref"
  }
  # ... more policy definitions
}

By adding depends_on = [azurerm_policy_set_definition.kubernetes_initiative] to the azurerm_policy_assignment resource, you are creating an explicit link in Terraform’s dependency graph. This instructs Terraform’s scheduler to ensure that the azurerm_policy_set_definition.kubernetes_initiative resource has been successfully created and registered in the Terraform state file before it even attempts to start the creation process for azurerm_policy_assignment.kubernetes_assignment. This additional wait time, managed by Terraform, is usually sufficient to allow the Azure API to fully propagate the existence of the policy set definition, thereby preventing the “Policy definition not found” error during the assignment operation.

If you were assigning a single policy definition instead of a set, the depends_on argument would reference the azurerm_policy_definition resource instead:

resource "azurerm_policy_assignment" "my_policy_assignment" {
  name                 = "my-specific-policy-assignment"
  scope                = "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
  policy_definition_id = azurerm_policy_definition.my_custom_policy.id
  display_name         = "My Specific Policy Assignment"

  depends_on = [
    azurerm_policy_definition.my_custom_policy
  ]
}

resource "azurerm_policy_definition" "my_custom_policy" {
  name         = "my-custom-policy"
  policy_type  = "Custom"
  mode         = "Indexed"
  display_name = "My Custom Policy"
  policy_rule  = <<POLICY_RULE
{
  "if": {
    "not": {
      "field": "tags['Environment']",
      "exists": true
    }
  },
  "then": {
    "effect": "Audit"
  }
}
POLICY_RULE
}

In both cases, the principle is the same: use depends_on to explicitly tell Terraform to wait for the resource that defines the policy or policy set before attempting to create the assignment that references it.

Implementing the Solution and Best Practices¶

Implementing the depends_on solution is straightforward, requiring only the addition of the meta-argument to your azurerm_policy_assignment resource block. Ensure the resource address listed in depends_on exactly matches the address of the azurerm_policy_definition or azurerm_policy_set_definition resource that the assignment relies on.

While depends_on is effective here, it’s still important to follow general Terraform best practices for dependency management:

• Favor Implicit Dependencies: Rely on Terraform’s automatic detection through resource attribute references whenever possible. This keeps your code cleaner and allows Terraform to build an optimal execution plan.
• Logical Code Structure: Organize your Terraform files logically. Placing policy definitions or set definitions in a file or module that is processed before assignments can sometimes help clarify dependencies, although Terraform’s graph still dictates the final order.
• Use Modules: For complex policy deployments, consider using Terraform Modules. Modules can encapsulate the creation of a policy definition or set definition and its subsequent assignment. Dependencies can be managed within or between modules, promoting reusability and manageability. When consuming a module that outputs a policy ID, the resource using that output still creates an implicit dependency.
• Validate and Plan: Always run terraform validate and terraform plan before applying. terraform plan will show you the intended order of operations, allowing you to verify that the definition is created before the assignment. While the plan might look correct even without depends_on, the explicit dependency ensures robustness against API timing.

When depends_on Might Still Not Be Enough¶

While adding depends_on resolves the most common cause related to Terraform’s execution timing, it’s worth noting that other factors could potentially lead to a “Policy definition not found” error:

1. Incorrect policy_definition_id: Double-check that the ID referenced in the policy_definition_id argument is absolutely correct. Typos or incorrect formatting will cause the assignment to fail regardless of dependencies. Ensure you are referencing the correct resource type (definition vs. set definition) and its correct identifier (the .id attribute from the Terraform resource).
2. Scope Issues: Ensure the scope where you are applying the policy assignment has the necessary permissions for the identity performing the Terraform deployment. The identity needs Microsoft.Authorization/policyAssignments/write permission at the specified scope. Also, ensure the definition or set definition exists and is accessible from the scope where you are creating the assignment.
3. Azure Propagation Delays: In rare cases, extremely rapid successive deployments or API load might still encounter brief propagation delays within Azure itself, even after Terraform has waited. These are less common but can happen. Troubleshooting might involve adding a null_resource with a provisioner "local-exec" "sleep" that depends_on the definition and is depended on by the assignment, forcing a pause (though this is generally discouraged).
4. Permissions on the Definition/Set Definition: Although less common for finding the definition, ensure the identity used by Terraform has permission to read the policy definition or set definition being referenced.

If adding depends_on does not resolve the issue, carefully review the exact error message, verify the IDs and scopes used in your configuration, and check the Azure Activity Log for more detailed insights into why the assignment API call failed.

Extending Policy Management with Terraform¶

Successfully managing policy assignments opens the door to leveraging other Azure Policy capabilities with Terraform. You can manage:

• Policy Exemptions (azurerm_policy_set_definition, azurerm_policy_exemption): Exempting specific resources or scopes from policy evaluation.
• Policy Remediations (azurerm_policy_remediation): Creating tasks to automatically bring non-compliant resources into compliance for “DeployIfNotExists” or “Modify” policies.
• Compliance Assessment: While not directly managed by Terraform resources, successful deployment of policies allows Azure to perform compliance scans, the results of which can be viewed in the Azure portal or via Azure APIs.

Managing these additional components with Terraform will also involve understanding and correctly configuring dependencies. For example, a policy remediation depends on a policy assignment existing, so you would likely need a depends_on relationship there as well.

Advanced Scenarios: Dynamic Assignments¶

When assigning policies dynamically using Terraform’s count or for_each meta-arguments (e.g., assigning a policy set to a list of subscriptions), managing dependencies becomes slightly more complex but follows the same principles.

If you are assigning the same policy set definition to multiple scopes:

resource "azurerm_policy_assignment" "scope_assignments" {
  for_each = toset(var.scopes_to_assign_policy)

  name                 = "kubernetes-assignment-${replace(each.key, "/", "-")}"
  scope                = each.key
  policy_definition_id = azurerm_policy_set_definition.kubernetes_initiative.id # Still references a single set definition
  display_name         = "Kubernetes Security Assignment on ${each.key}"

  # All assignments depend on the single policy set definition
  depends_on = [
    azurerm_policy_set_definition.kubernetes_initiative
  ]
}

resource "azurerm_policy_set_definition" "kubernetes_initiative" {
  # ... definition configuration ...
}

Here, every instance of the azurerm_policy_assignment resource managed by for_each will wait for the single azurerm_policy_set_definition resource.

If you are creating multiple policy definitions and assigning each one:

resource "azurerm_policy_definition" "my_policies" {
  for_each = var.custom_policies

  name         = "custom-policy-${each.key}"
  policy_type  = "Custom"
  mode         = "Indexed"
  display_name = each.value.display_name
  policy_rule  = each.value.rule
}

resource "azurerm_policy_assignment" "policy_assignments" {
  for_each = var.custom_policies # Assuming assignments match definitions for simplicity

  name                 = "assignment-${each.key}-${replace(var.assignment_scope, "/", "-")}"
  scope                = var.assignment_scope
  policy_definition_id = azurerm_policy_definition.my_policies[each.key].id # References a specific definition instance
  display_name         = "Assignment for ${each.value.display_name}"

  # Each assignment depends on its corresponding policy definition instance
  depends_on = [
    azurerm_policy_definition.my_policies[each.key]
  ]
}

In this more complex scenario, using depends_on = [azurerm_policy_definition.my_policies[each.key]] correctly links each assignment instance created by for_each to its corresponding policy definition instance created by for_each. This ensures that for each policy being deployed, its definition is created before its assignment is attempted. This illustrates how depends_on can be used with resource instances managed by count or for_each.

Summary¶

The “Policy definition not found” error during Azure Policy assignment with Terraform is a common issue caused by a timing mismatch or insufficient implicit dependency resolution between the creation of the policy definition/set definition and the subsequent assignment. Although Terraform attempts to manage dependencies automatically, the eventual consistency model of cloud APIs can sometimes lead to race conditions.

The effective and standard solution is to explicitly declare a dependency using the depends_on meta-argument within the azurerm_policy_assignment resource block, pointing to the azurerm_policy_definition or azurerm_policy_set_definition resource that it references. This forces Terraform to wait for the definition resource to be fully created and available before proceeding with the assignment, mitigating the timing issue.

While depends_on is a powerful tool, it should be used judiciously. In this specific policy assignment scenario, it is often necessary and the recommended approach to ensure reliable deployments. Always validate your configuration and review the Terraform plan to understand the order of operations.

---

Third-party information disclaimer

The third-party products that this article discusses are manufactured by companies that are independent of Microsoft. Microsoft makes no warranty, implied or otherwise, about the performance or reliability of these products.

---

We hope this detailed explanation helps you successfully troubleshoot and resolve policy assignment errors in your Terraform deployments. Have you encountered this issue or similar dependency challenges with Terraform and Azure Policy? Do you have alternative approaches or tips to share? Let us know in the comments below!