### Terraform Version ```shell Terraform v1.5.0 on linux_amd64 + provider registry.terraform.io/hashicorp/azurerm v3.116.0 ``` ### Terraform Configuration Files **Backend Configuration:** ```terraform terraform { required_version = ">= 1.0" required_providers { azurerm = { source = "hashicorp/azurerm" version = "~> 3.0" } } backend "azurerm" { # Backend configuration provided via --backend-config flags } } provider "azurerm" { features {} environment = "usgovernment" use_oidc = true } resource "azurerm_resource_group" "example" { name = "rg-terraform-demo" location = "usgovvirginia" } ``` ### Debug Output ``` Run terraform init \ terraform init \ -backend-config="resource_group_name=rg-tfstate" \ -backend-config="key=demo-oidc-tfstate.tfstate" \ -backend-config="storage_account_name=***" \ -backend-config="container_name=tfstate" \ -backend-config="subscription_id=***" \ -backend-config="tenant_id=***" \ -backend-config="client_id=***" \ -backend-config='use_azuread_auth=true' \ -backend-config="use_oidc=true" \ -backend-config="environment=usgovernment" shell: /usr/bin/bash -e {0} env: TF_LOG: trace 2025-07-21T15:27:14.359Z [INFO] Terraform version: 1.12.2 2025-07-21T15:27:14.359Z [DEBUG] using github.com/hashicorp/go-tfe v1.74.1 2025-07-21T15:27:14.359Z [DEBUG] using github.com/hashicorp/hcl/v2 v2.23.1-0.20250203194505-ba0759438da2 2025-07-21T15:27:14.359Z [DEBUG] using github.com/hashicorp/terraform-svchost v0.1.1 2025-07-21T15:27:14.359Z [DEBUG] using github.com/zclconf/go-cty v1.16.2 2025-07-21T15:27:14.359Z [INFO] Go runtime version: go1.24.2 2025-07-21T15:27:14.359Z [INFO] CLI args: []string{"terraform", "init", "-backend-config=resource_group_name=rg-tfstate", "-backend-config=key=demo-oidc-tfstate.tfstate", "-backend-config=storage_account_name=***", "-backend-config=container_name=tfstate", "-backend-config=subscription_id=***", "-backend-config=tenant_id=***", "-backend-config=client_id=***", "-backend-config=use_azuread_auth=true", "-backend-config=use_oidc=true", "-backend-config=environment=usgovernment"} 2025-07-21T15:27:14.359Z [TRACE] Stdout is not a terminal 2025-07-21T15:27:14.360Z [TRACE] Stderr is not a terminal 2025-07-21T15:27:14.360Z [TRACE] Stdin is not a terminal 2025-07-21T15:27:14.360Z [DEBUG] Attempting to open CLI config file: /home/runner/.terraformrc 2025-07-21T15:27:14.360Z [DEBUG] File doesn't exist, but doesn't need to. Ignoring. 2025-07-21T15:27:14.360Z [DEBUG] ignoring non-existing provider search directory terraform.d/plugins 2025-07-21T15:27:14.360Z [DEBUG] ignoring non-existing provider search directory /home/runner/.terraform.d/plugins 2025-07-21T15:27:14.360Z [DEBUG] ignoring non-existing provider search directory /home/runner/.local/share/terraform/plugins 2025-07-21T15:27:14.360Z [DEBUG] ignoring non-existing provider search directory /usr/local/share/terraform/plugins 2025-07-21T15:27:14.360Z [DEBUG] ignoring non-existing provider search directory /usr/share/terraform/plugins 2025-07-21T15:27:14.362Z [INFO] CLI command args: []string{"init", "-backend-config=resource_group_name=rg-tfstate", "-backend-config=key=demo-oidc-tfstate.tfstate", "-backend-config=storage_account_name=***", "-backend-config=container_name=tfstate", "-backend-config=subscription_id=***", "-backend-config=tenant_id=***", "-backend-config=client_id=***", "-backend-config=use_azuread_auth=true", "-backend-config=use_oidc=true", "-backend-config=environment=usgovernment"} 2025-07-21T15:27:14.363Z [TRACE] Meta.Backend: merging -backend-config=... CLI overrides into backend configuration 2025-07-21T15:27:14.363Z [TRACE] Meta.Backend: built configuration for "azurerm" backend with hash value 750355294 2025-07-21T15:27:14.363Z [TRACE] Meta.Backend: backend has not previously been initialized in this working directory 2025-07-21T15:27:14.363Z [TRACE] Meta.Backend: moving from default local state only to "azurerm" backend 2025-07-21T15:27:14.363Z [DEBUG] checking for provisioner in "." 2025-07-21T15:27:14.363Z [DEBUG] checking for provisioner in "/home/runner/work/_temp/9e59c32b-531c-4834-b15f-551006ba2464" 2025-07-21T15:27:14.363Z [TRACE] backend/local: state manager for workspace "default" will: - read initial snapshot from terraform.tfstate - write new snapshots to terraform.tfstate - create any backup at terraform.tfstate.backup 2025-07-21T15:27:14.363Z [TRACE] statemgr.Filesystem: reading initial snapshot from terraform.tfstate 2025-07-21T15:27:14.363Z [TRACE] statemgr.Filesystem: snapshot file has nil snapshot, but that's okay 2025-07-21T15:27:14.363Z [TRACE] statemgr.Filesystem: read nil snapshot 2025-07-21T15:27:14.363Z [TRACE] Meta.Backend: ignoring local "default" workspace because its state is empty 2025-07-21T15:27:14.364Z [DEBUG] Configuring built-in cloud environment by name: "usgovernment" 2025-07-21T15:27:14.365Z [DEBUG] Building the Container Client from AAD auth 2025-07-21T15:27:14.365Z [DEBUG] GET https://run-actions-1-azure-eastus.actions.githubusercontent.com/73//idtoken/e07981ec-dfec-4395-85b6-0f156e26f9ef/7e41a0e0-976f-56a4-8912-350e80c051d2?api-version=2.0&audience=api%3A%2F%2FAzureADTokenExchange Initializing the backend... 2025-07-21T15:27:14.574Z [DEBUG] POST https://login.microsoftonline.us/***/oauth2/v2.0/token ╷ │ Error: Failed to get existing workspaces: listing blobs: executing request: authorizing request: clientCredentialsToken: received HTTP status 401 with response: {"error":"invalid_client","error_description":"AADSTS700212: No matching federated identity record found for presented assertion audience 'api://AzureADTokenExchange'. Please check your federated identity credential Subject, Audience and Issuer against the presented assertion. https://learn.microsoft.com/entra/workload-id/workload-identity-federation Trace ID: 3afc84f7-a3d3-452f-8199-ea1d74f76100 Correlation ID: 5896aa6a-5ab6-402b-8a49-5bfae808ea67 Timestamp: 2025-07-21 15:27:14Z","error_codes":[700212],"timestamp":"2025-07-21 15:27:14Z","trace_id":"3afc84f7-a3d3-452f-8199-ea1d74f76100","correlation_id":"5896aa6a-5ab6-402b-8a49-5bfae808ea67","error_uri":"https://login.microsoftonline.us/error?code=700212"} │ │ ╵ Error: Process completed with exit code 1. ``` ### Expected Behavior When using `terraform init` with `--backend-config="use_oidc=true"` and `--backend-config="environment=usgovernment"`, Terraform should: 1. Detect that Azure Government cloud is being used 2. Use the **correct** OIDC audience for Azure Government: `api://AzureADTokenExchangeUSGov` 3. Successfully authenticate to the Azure Government storage backend 4. Initialize the backend without authentication errors ### Actual Behavior Terraform uses the incorrect OIDC audience (`api://AzureADTokenExchange`) when authenticating to Azure Government backends, causing authentication failure with error `AADSTS700212`. The backend authentication fails because: - Terraform sends OIDC token with audience: `api://AzureADTokenExchange` (Azure Commercial) - Azure Government expects audience: `api://AzureADTokenExchangeUSGov` - Azure Government rejects the token due to audience mismatch ### Steps to Reproduce ## Prerequisites Before reproducing this issue, you need: 1. **Azure Government Subscription**: Access to an Azure Government environment 2. **Azure Storage Account**: Create a storage account in Azure Government for the Terraform backend: ```bash # Using Azure CLI (connected to Azure Government) az cloud set --name AzureUSGovernment az login # Create resource group and storage account az group create --name "rg-tfstate" --location "USGov Virginia" az storage account create \ --name "tfstateXXXXX" \ --resource-group "rg-tfstate" \ --location "USGov Virginia" \ --sku "Standard_LRS" az storage container create \ --name "tfstate" \ --account-name "tfstateXXXXX" ``` 3. **App Registration with Federated Credentials**: Create in Azure Government with: - **Audience**: `api://AzureADTokenExchangeUSGov` - **Subject**: `repo:your-org/your-repo:ref:refs/heads/main` - **Issuer**: `https://token.actions.githubusercontent.com` 4. **Required Permissions**: Grant the App Registration: - **Contributor** role on the storage account resource group (for backend access). Reader should be sufficient as well, since no `apply` will need to be performed. - **Storage Blob Data Contributor** role on the storage account (for backend access) ## Steps to Reproduce **Important:** This issue can only be reproduced using GitHub Actions OIDC authentication. Local testing with service principals or other authentication methods will not trigger this bug. 1. **Create Terraform configuration with Azure Government backend:** ```terraform terraform { required_providers { azurerm = { source = "hashicorp/azurerm" version = "~> 3.0" } } backend "azurerm" {} } provider "azurerm" { features {} environment = "usgovernment" } resource "azurerm_resource_group" "test" { name = "rg-test" location = "usgovvirginia" } ``` 2. **Create GitHub Actions workflow to reproduce the issue:** ```yaml name: Reproduce Azure Gov OIDC Bug on: workflow_dispatch: push: branches: [ main ] permissions: id-token: write # Required for OIDC contents: read # Required to checkout code jobs: terraform: name: 'Terraform' runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v4 - name: Setup Terraform uses: hashicorp/setup-terraform@v3 with: terraform_wrapper: false - name: Terraform Init (This step will FAIL) run: | terraform init \ -backend-config="resource_group_name=rg-tfstate" \ -backend-config="storage_account_name=${{ vars.TF_BACKEND_STORAGE_ACCOUNT }}" \ -backend-config="container_name=tfstate" \ -backend-config="key=terraform.tfstate" \ -backend-config="subscription_id=${{ vars.AZURE_SUBSCRIPTION_ID }}" \ -backend-config="tenant_id=${{ vars.AZURE_TENANT_ID }}" \ -backend-config="client_id=${{ vars.AZURE_CLIENT_ID }}" \ -backend-config="use_azuread_auth=true" \ -backend-config="use_oidc=true" \ -backend-config="environment=usgovernment" env: ARM_USE_OIDC: true ARM_CLIENT_ID: ${{ vars.AZURE_CLIENT_ID }} ARM_SUBSCRIPTION_ID: ${{ vars.AZURE_SUBSCRIPTION_ID }} ARM_TENANT_ID: ${{ vars.AZURE_TENANT_ID }} ``` 3. **Configure GitHub repository variables:** - `AZURE_CLIENT_ID`: Application (client) ID from Azure Government - `AZURE_TENANT_ID`: Directory (tenant) ID from Azure Government - `AZURE_SUBSCRIPTION_ID`: Azure Government subscription ID - `TF_BACKEND_STORAGE_ACCOUNT`: Storage account name in Azure Government 4. **Run the workflow and observe the authentication failure** The workflow will fail with the audience mismatch error because Terraform sends the wrong OIDC audience to Azure Government. ### Additional Context **Environment Details:** - **CI System**: GitHub Actions (REQUIRED - this issue only occurs with GitHub OIDC) - **Authentication Method**: GitHub OIDC with Azure federated credentials - **Azure Cloud**: Azure Government (usgovernment) - **Provider**: AzureRM v3.116.0 **Important Note:** This bug can ONLY be reproduced using GitHub Actions with OIDC authentication. Local testing with service principals, Azure CLI, or other authentication methods will not trigger this issue because the problem is specifically with how Terraform handles GitHub's OIDC tokens when authenticating to Azure Government backends. **Root Cause Analysis:** The issue appears to be in Terraform Core's backend authentication logic (not the AzureRM provider) where it doesn't properly distinguish between Azure Commercial and Azure Government when determining the OIDC audience for GitHub Actions token requests. When `environment=usgovernment` is specified in backend configuration, Terraform should use `api://AzureADTokenExchangeUSGov` as the audience instead of the default `api://AzureADTokenExchange`. Using the `ARM_*` environment variables does not resolve the issue, as Terraform still defaults to the wrong audience. **Current Workaround:** The only working solution is to use Azure CLI authentication via the `azure/login@v2` action instead of native OIDC: ```yaml - name: 'OIDC Login to Azure CLI' uses: azure/login@v2 with: client-id: ${{ vars.AZURE_CLIENT_ID }} tenant-id: ${{ vars.AZURE_TENANT_ID }} subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }} environment: 'azureusgovernment' audience: 'api://AzureADTokenExchangeUSGov' - name: Terraform Init (Workaround) run: | terraform init \ -backend-config="resource_group_name=rg-tfstate" \ -backend-config="storage_account_name=${{ vars.TF_BACKEND_STORAGE_ACCOUNT }}" \ -backend-config="container_name=tfstate" \ -backend-config="key=terraform.tfstate" \ -backend-config="use_azuread_auth=true" \ -backend-config="use_cli=true" \ -backend-config="environment=usgovernment" ``` This workaround uses the Azure CLI (authenticated via OIDC with the correct audience) instead of Terraform's native OIDC backend authentication. **Impact:** - Prevents native OIDC authentication for Azure Government backends - Forces users to use CLI-based workarounds in CI/CD pipelines - Breaks infrastructure-as-code automation for Azure Government customers - Azure CLI supports only a single authentication session, preventing mixed-tenant scenarios within Azure Government i.e. Gov tenant 1 cannot access resources in Gov tenant 2 using OIDC, requiring a client secret to be used. This workaround only works for the backend + provider when both are in the same Azure Government tenant. This is a real scenario I am currently encountering and struggling to get past due to this bug. ### References - **Azure Government OIDC Documentation**: https://learn.microsoft.com/en-us/azure/azure-government/documentation-government-developer-guide#guidance-for-developers - **GitHub OIDC Documentation**: https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect - **AzureRM Provider Backend**: https://developer.hashicorp.com/terraform/language/settings/backends/azurerm ### Generative AI / LLM assisted development? Yes, generative AI tools (Copilot/Claude Sonnet 4) were used in the development of this configuration and bug report, however both the development and bug report were peer-reviewed and tested by human developers to ensure accuracy and reliability.