### Terraform Version ```shell Terraform v1.9.8 on darwin_arm64 + provider registry.terraform.io/hashicorp/aws v6.7.0 Tested across multiple versions: - AWS Provider v5.0.0 through v6.7.0 (all affected) - Terraform v1.1.9 through v1.9.8 (all affected) ``` ### Terraform Configuration Files ```terraform terraform { required_providers { aws = { source = "hashicorp/aws" version = "~> 5.0" } } } provider "aws" { region = "us-east-1" } locals { subnet_ids = [ "subnet-12345", "subnet-67890", ] } resource "aws_security_group" "test" { name = "test-resolver-endpoint" vpc_id = "vpc-12345" egress { from_port = 0 to_port = 0 protocol = "-1" cidr_blocks = ["0.0.0.0/0"] } tags = { Name = "test-resolver-endpoint" } } # This demonstrates the hash collision bug # Expected: 6 IP addresses (3 per subnet) # Actual: 2 IP addresses (1 per subnet due to hash collisions) resource "aws_route53_resolver_endpoint" "test" { direction = "OUTBOUND" security_group_ids = [aws_security_group.test.id] # Multiple auto-assigned IPs per subnet - this is the problematic case ip_address { subnet_id = local.subnet_ids[0] # First IP in subnet A } ip_address { subnet_id = local.subnet_ids[0] # Second IP in subnet A - gets deduplicated! } ip_address { subnet_id = local.subnet_ids[0] # Third IP in subnet A - gets deduplicated! } ip_address { subnet_id = local.subnet_ids[1] # First IP in subnet B } ip_address { subnet_id = local.subnet_ids[1] # Second IP in subnet B - gets deduplicated! } ip_address { subnet_id = local.subnet_ids[1] # Third IP in subnet B - gets deduplicated! } tags = { Name = "test-multiple-ips-per-subnet" } } output "ip_address_count" { value = length(aws_route53_resolver_endpoint.test.ip_address) description = "Should be 6 but will show 2 due to hash collision bug" } output "ip_addresses" { value = aws_route53_resolver_endpoint.test.ip_address } ``` ### Debug Output ``` 2025-08-05T11:53:38.760-0700 [DEBUG] ReferenceTransformer: "aws_route53_resolver_endpoint.test" references: [aws_security_group.test local.subnet_ids (expand) local.subnet_ids (expand) local.subnet_ids (expand) local.subnet_ids (expand) local.subnet_ids (expand) local.subnet_ids (expand)] 2025-08-05T11:53:38.760-0700 [DEBUG] ReferenceTransformer: "local.subnet_ids (expand)" references: [] ``` ### Expected Behavior AWS Route53 Resolver endpoints should support up to 6 IP addresses per endpoint (quotas can be increased), including multiple IP addresses per subnet with auto-assignment. When configuring multiple `ip_address` blocks for the same subnet without explicit IP addresses, AWS should auto-assign unique IP addresses within each subnet, and Terraform should track all configured IP addresses correctly. According to AWS documentation, this is a supported configuration. ```diff # aws_route53_resolver_endpoint.test will be created + resource "aws_route53_resolver_endpoint" "test" { + arn = (known after apply) + direction = "OUTBOUND" + host_vpc_id = (known after apply) + id = (known after apply) + protocols = (known after apply) + region = "us-east-1" + resolver_endpoint_type = (known after apply) + security_group_ids = (known after apply) + tags = { + "Name" = "test-multiple-ips-per-subnet" } + ip_address { + ip = (known after apply) + ip_id = (known after apply) + ipv6 = (known after apply) + subnet_id = "subnet-12345" } + ip_address { + ip = (known after apply) + ip_id = (known after apply) + ipv6 = (known after apply) + subnet_id = "subnet-12345" } + ip_address { + ip = (known after apply) + ip_id = (known after apply) + ipv6 = (known after apply) + subnet_id = "subnet-12345" } + ip_address { + ip = (known after apply) + ip_id = (known after apply) + ipv6 = (known after apply) + subnet_id = "subnet-67890" } + ip_address { + ip = (known after apply) + ip_id = (known after apply) + ipv6 = (known after apply) + subnet_id = "subnet-67890" } + ip_address { + ip = (known after apply) + ip_id = (known after apply) + ipv6 = (known after apply) + subnet_id = "subnet-67890" } } ``` ### Actual Behavior ## Affected Resource(s) * `aws_route53_resolver_endpoint` When multiple `ip_address` blocks are configured for the same subnet with auto-assigned IPs (empty `ip` field), the Terraform provider's hash function creates identical hashes, causing Terraform's Set data structure to deduplicate them. **Root Cause**: The `endpointHashIPAddress` function in `/internal/service/route53resolver/endpoint.go` generates hashes using the format `"{subnet_id}-{ip}-"`. When `ip` is empty (auto-assigned), multiple entries for the same subnet create identical hashes like `"subnet-12345-"`, causing Set deduplication. **Impact**: - Only 1 IP address is created per subnet instead of the requested multiple IPs - Forces users to specify explicit IP addresses to achieve uniqueness - Contradicts AWS best practices of letting AWS manage IP allocation - Prevents users from leveraging AWS Route53 Resolver's full high-availability capabilities No error is thrown, but terraform plan shows fewer IP addresses than configured: ```diff # aws_route53_resolver_endpoint.test will be created + resource "aws_route53_resolver_endpoint" "test" { + arn = (known after apply) + direction = "OUTBOUND" + host_vpc_id = (known after apply) + id = (known after apply) + protocols = (known after apply) + region = "us-east-1" + resolver_endpoint_type = (known after apply) + security_group_ids = (known after apply) + tags = { + "Name" = "test-multiple-ips-per-subnet" } + ip_address { + ip = (known after apply) + ip_id = (known after apply) + ipv6 = (known after apply) + subnet_id = "subnet-12345" } + ip_address { + ip = (known after apply) + ip_id = (known after apply) + ipv6 = (known after apply) + subnet_id = "subnet-67890" } } ``` ### Steps to Reproduce 1. Apply the provided configuration with multiple `ip_address` blocks for the same subnet without explicit `ip` values 2. Run `terraform plan` and observe that only 2 IP addresses are planned instead of 6 3. Run `terraform apply` and verify that only 1 IP address per subnet is created 4. Check the AWS Console to confirm only 2 IP addresses exist on the resolver endpoint 5. Check Terraform state: `terraform state show aws_route53_resolver_endpoint.test` will show only 2 IP addresses **Alternative reproduction with existing resources**: 1. Create a resolver endpoint through AWS Console with multiple IPs per subnet (this works fine) 2. Import it into Terraform using `terraform import aws_route53_resolver_endpoint.test rslvr-out-12345` 3. Run `terraform plan` - it will show a diff trying to remove the "extra" IP addresses due to the hash collision ### Additional Context ## Affected Resource(s) or Data Source(s) * `aws_route53_resolver_endpoint` ### References **AWS Service Capability vs Provider Limitation:** - AWS Route53 Resolver supports up to 6 IP addresses per endpoint ([quotas can be increased](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/DNSLimitations.html#limits-api-entities-resolver)) - AWS Route53 Resolver supports multiple IP addresses per subnet for higher query limits - AWS Route53 Resolver supports auto-assignment of IP addresses within subnets - AWS documentation encourages letting AWS manage IP allocation for operational simplicity **Business Impact:** - Prevents users from implementing multiple IP addresses per subnet for DNS resolver configurations for higher query limits - Forces manual IP management instead of leveraging AWS auto-assignment capabilities - Creates a gap between AWS service capabilities and Terraform provider support - May lead to configuration drift when users work around the limitation using AWS Console or CLI **Technical Analysis:** - Issue is specifically in the `endpointHashIPAddress` function in `/internal/service/route53resolver/endpoint.go` - Current hash format: `fmt.Fprintf(&buf, "%s-%s-", m[names.AttrSubnetID].(string), m["ip"].(string))` - When `ip` is empty (auto-assigned), identical subnets produce identical hashes: `"subnet-12345-"` - This affects both resource creation and state management (imports, refreshes, etc.) **Similar Patterns in Other Resources:** - `aws_instance` successfully handles multiple network interfaces per subnet using unique identifiers - `aws_lb_target_group_attachment` handles multiple targets per target group without hash collisions - Other AWS resources with Set-based configurations have solved similar problems **Environment Details:** - Affects all AWS regions and partitions - No workarounds exist other than manual IP specification - Issue present since the resource was first implemented ### Generative AI / LLM assisted development? n/a