### Terraform Version ```shell Terraform v1.6.4 on linux_amd64 ``` ### Terraform Configuration Files Here is backend.tf file: ```terraform terraform { backend "s3" { bucket = "terraform-bucket" dynamodb_table = "terraform-state-lock" encrypt = true key = "./terraform.tfstate" region = "eu-central-1" } ``` ### Debug Output Here is log from terraform 1.5.x (credentials removed) (result - 200 OK): ``` ----------------------------------------------------- 2023-11-15T12:11:55.779+0200 [DEBUG] [aws-sdk-go] {} 2023-11-15T12:11:55.780+0200 [DEBUG] [aws-sdk-go] DEBUG: Request s3/GetObject Details: ---[ REQUEST POST-SIGN ]----------------------------- GET /terraform.tfstate HTTP/1.1 Host: terraform-bucket.s3.eu-central-1.amazonaws.com User-Agent: APN/1.0 HashiCorp/1.0 Terraform/1.5.7 aws-sdk-go/1.44.122 (go1.20.7; linux; amd64) X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 X-Amz-Date: 20231115T101155Z Accept-Encoding: gzip ----------------------------------------------------- 2023-11-15T12:11:55.991+0200 [DEBUG] [aws-sdk-go] DEBUG: Response s3/GetObject Details: ---[ RESPONSE ]-------------------------------------- HTTP/1.1 200 OK Connection: close Content-Length: 384760 Accept-Ranges: bytes Content-Type: application/json Date: Wed, 15 Nov 2023 10:11:56 GMT Etag: "f357341b9645d1643cb8ee79022cd215" Last-Modified: Fri, 27 Oct 2023 08:46:51 GMT Server: AmazonS3 X-Amz-Id-2: Y9LAIgVKl5bjua2wExd1quq5KXdnDq5wCd8eE3GbFJlqo3bNOkVvqjoZITgeZkp9JuMHn8wRw0E= X-Amz-Request-Id: BQPC6ES63540GTHN X-Amz-Server-Side-Encryption: AES256 X-Amz-Version-Id: nhKueNmyZNat3tqAlmwAeGFTdcYYwWPN ``` Here is the log from terraform 1.6.x (404 error): ``` 2023-11-15T12:15:07.874+0200 [INFO] backend-s3: Downloading remote state: tf_backend.operation=Get tf_backend.req_id=c6b7a8aa-e870-7d70-8964-52935f98b12d tf_backend.s3.bucket=terraform-bucket tf_backend.s3.path=./terraform.tfstate 2023-11-15T12:15:07.875+0200 [DEBUG] backend-s3: HTTP Request Sent: aws.operation=HeadObject aws.region=eu-central-1 aws.sdk=aws-sdk-go-v2 aws.service=S3 tf_backend.operation=Get tf_backend.req_id=c6b7a8aa-e870-7d70-8964-52935f98b12d tf_backend.s3.bucket=terraform-bucket tf_backend.s3.path=./terraform.tfstate http.request.header.x_amz_date=20231115T101507Z http.method=HEAD net.peer.name=terraform-bucket.s3.eu-central-1.amazonaws.com http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.6.3 (+https://www.terraform.io) aws-sdk-go-v2/1.21.0 os/linux lang/go#1.21.3 md/GOOS#linux md/GOARCH#amd64 api/s3#1.38.5" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=AKIA************AKO7/20231115/eu-central-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=*****" http.request.header.x_amz_content_sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 http.url=https://terraform-bucket.s3.eu-central-1.amazonaws.com/./terraform.tfstate http.request.header.amz_sdk_invocation_id=813d4010-7c72-4ccf-a30e-59331be39944 http.request.header.accept_encoding=identity http.request.header.amz_sdk_request="attempt=1; max=5" http.request.body="" 2023-11-15T12:15:07.921+0200 [DEBUG] backend-s3: HTTP Response Received: aws.operation=HeadObject aws.region=eu-central-1 aws.sdk=aws-sdk-go-v2 aws.service=S3 tf_backend.operation=Get tf_backend.req_id=c6b7a8aa-e870-7d70-8964-52935f98b12d tf_backend.s3.bucket=terraform-bucket tf_backend.s3.path=./terraform.tfstate http.response.header.date="Wed, 15 Nov 2023 10:15:07 GMT" http.response.header.server=AmazonS3 http.response.body="" http.duration=46 http.status_code=404 http.response.header.x_amz_request_id=NX3QC1MC0N9N5TW6 http.response.header.x_amz_id_2="yUmgCqK3zpSB0bReJ5bxxCy0B88ZhARUw5Omn5VsbCUzGSUbADoqnOBurOZZ+GL0a6eDbY7gl4Q=" http.response.header.content_type=application/xml ``` ### Expected Behavior Expect plan successfully executed ### Actual Behavior terraform plan returns error: │ Error: error loading state: state data in S3 does not have the expected content. │ │ The checksum calculated for the state stored in S3 does not match the checksum │ stored in DynamoDB. │ │ Bucket: ocb-ocb-terraform-state │ Key: ./terraform.tfstate │ Calculated checksum: │ Stored checksum: bb17f4c60b305db0f0fd8210c8da8332 │ │ This may be caused by unusually long delays in S3 processing a previous state │ update. Please wait for a minute or two and try again. │ │ If this problem persists, and neither S3 nor DynamoDB are experiencing an │ outage, you may need to manually verify the remote state and remove the Digest │ value stored in the DynamoDB table │ │ │ ### Steps to Reproduce Using terraform 1.5.x apply plan: - terraform init - terraform apply Upgrade to 1.6.x and then apply: - terraform apply 1.6 version returns error: ### Additional Context _No response_ ### References Also created issue in terragrunt, because backend.tf is actually created by terragrunt: https://github.com/gruntwork-io/terragrunt/issues/2792