### Terraform Version ```shell Terraform v1.13.3 on linux_amd64 + provider registry.terraform.io/mypurecloud/genesyscloud v1.51.0 ``` ### Terraform Configuration Files ```terraform backend "s3" { key = "state-name" bucket = "terraform-states" endpoints = { s3 = "https://custom_s3.server.com" sts = "https://custom_s3.server.com" } region = "eu-de" skip_region_validation = true skip_credentials_validation = true skip_requesting_account_id = true skip_metadata_api_check = true skip_s3_checksum = true } ``` ### Debug Output ``` 2025-09-20T10:56:01.536+0200 [INFO] Terraform version: 1.13.3 2025-09-20T10:56:01.537+0200 [DEBUG] using github.com/hashicorp/go-tfe v1.74.1 2025-09-20T10:56:01.537+0200 [DEBUG] using github.com/hashicorp/hcl/v2 v2.24.0 2025-09-20T10:56:01.537+0200 [DEBUG] using github.com/hashicorp/terraform-svchost v0.1.1 2025-09-20T10:56:01.538+0200 [DEBUG] using github.com/zclconf/go-cty v1.16.3 2025-09-20T10:56:01.538+0200 [INFO] Go runtime version: go1.24.5 2025-09-20T10:56:01.538+0200 [INFO] CLI args: []string{"terraform", "state", "push", "errored.tfstate"} 2025-09-20T10:56:01.538+0200 [TRACE] Stdout is a terminal of width 164 2025-09-20T10:56:01.538+0200 [TRACE] Stderr is a terminal of width 164 2025-09-20T10:56:01.538+0200 [TRACE] Stdin is a terminal 2025-09-20T10:56:01.539+0200 [DEBUG] Attempting to open CLI config file: /home/.terraformrc 2025-09-20T10:56:01.539+0200 [DEBUG] File doesn't exist, but doesn't need to. Ignoring. 2025-09-20T10:56:01.540+0200 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins 2025-09-20T10:56:01.540+0200 [DEBUG] ignoring non-existing provider search directory /home/.terraform.d/plugins 2025-09-20T10:56:01.540+0200 [DEBUG] ignoring non-existing provider search directory /home/.local/share/terraform/plugins 2025-09-20T10:56:01.540+0200 [DEBUG] ignoring non-existing provider search directory /usr/local/share/terraform/plugins 2025-09-20T10:56:01.540+0200 [DEBUG] ignoring non-existing provider search directory /usr/share/terraform/plugins 2025-09-20T10:56:01.540+0200 [DEBUG] ignoring non-existing provider search directory /var/lib/snapd/desktop/terraform/plugins 2025-09-20T10:56:01.542+0200 [INFO] CLI command args: []string{"state", "push", "errored.tfstate"} 2025-09-20T10:56:01.676+0200 [TRACE] Meta.Backend: BackendOpts.Config not set, so using settings loaded from settings.tf:8,3-15 2025-09-20T10:56:01.676+0200 [TRACE] Meta.Backend: built configuration for "s3" backend with hash value 2607543200 2025-09-20T10:56:01.680+0200 [TRACE] Meta.Backend: working directory was previously initialized for "s3" backend 2025-09-20T10:56:01.681+0200 [TRACE] Meta.Backend: using already-initialized, unchanged "s3" backend configuration 2025-09-20T10:56:01.681+0200 [TRACE] backend-s3.aws-base: Resolving AWS configuration: tf_backend.operation=Configure tf_backend.req_id=2384ff8a-c177-f177-f02b-0c5e291d08ed tf_backend.s3.bucket=terraform-states tf_backend.s3.path=state-file-name 2025-09-20T10:56:01.681+0200 [DEBUG] backend-s3.aws-base: Resolving credentials provider: tf_backend.operation=Configure tf_backend.req_id=2384ff8a-c177-f177-f02b-0c5e291d08ed tf_backend.s3.bucket=terraform-states tf_backend.s3.path=state-file-name 2025-09-20T10:56:01.681+0200 [TRACE] backend-s3.aws-base: Building default HTTP client: tf_backend.operation=Configure tf_backend.req_id=2384ff8a-c177-f177-f02b-0c5e291d08ed tf_backend.s3.bucket=terraform-states tf_backend.s3.path=state-file-name 2025-09-20T10:56:01.682+0200 [DEBUG] backend-s3.aws-base: Loading configuration: tf_backend.operation=Configure tf_backend.req_id=2384ff8a-c177-f177-f02b-0c5e291d08ed tf_backend.s3.bucket=terraform-states tf_backend.s3.path=state-file-name 2025-09-20T10:56:01.682+0200 [DEBUG] backend-s3.aws-base: Retrieving credentials: tf_backend.operation=Configure tf_backend.req_id=2384ff8a-c177-f177-f02b-0c5e291d08ed tf_backend.s3.bucket=terraform-states tf_backend.s3.path=state-file-name 2025-09-20T10:56:01.682+0200 [INFO] backend-s3.aws-base: Retrieved credentials: tf_backend.operation=Configure tf_backend.req_id=2384ff8a-c177-f177-f02b-0c5e291d08ed tf_backend.s3.bucket=terraform-states tf_backend.s3.path=state-file-name tf_aws.credentials_source=EnvConfigCredentials 2025-09-20T10:56:01.682+0200 [TRACE] backend-s3.aws-base: Building default HTTP client: tf_backend.operation=Configure tf_backend.req_id=2384ff8a-c177-f177-f02b-0c5e291d08ed tf_backend.s3.bucket=terraform-states tf_backend.s3.path=state-file-name 2025-09-20T10:56:01.683+0200 [DEBUG] backend-s3.aws-base: Loading configuration: tf_backend.operation=Configure tf_backend.req_id=2384ff8a-c177-f177-f02b-0c5e291d08ed tf_backend.s3.bucket=terraform-states tf_backend.s3.path=state-file-name 2025-09-20T10:56:01.683+0200 [TRACE] Meta.Backend: instantiated backend of type *s3.Backend 2025-09-20T10:56:01.688+0200 [TRACE] providercache.fillMetaCache: scanning directory .terraform/providers 2025-09-20T10:56:01.740+0200 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/mypurecloud/genesyscloud v1.51.0 for linux_amd64 at .terraform/providers/registry.terraform.io/mypurecloud/genesyscloud/1.51.0/linux_amd64 2025-09-20T10:56:01.793+0200 [TRACE] providercache.fillMetaCache: including .terraform/providers/registry.terraform.io/mypurecloud/genesyscloud/1.51.0/linux_amd64 as a candidate package for registry.terraform.io/mypurecloud/genesyscloud 1.51.0 2025-09-20T10:56:02.198+0200 [DEBUG] checking for provisioner in "." 2025-09-20T10:56:02.206+0200 [DEBUG] checking for provisioner in "/usr/bin" 2025-09-20T10:56:02.208+0200 [TRACE] Meta.Backend: backend *s3.Backend does not support operations, so wrapping it in a local backend 2025-09-20T10:56:02.211+0200 [DEBUG] backend-s3: HTTP Request Sent: aws.region=eu-de aws.s3.bucket=terraform-states rpc.method=ListObjectsV2 rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Workspaces tf_backend.req_id=a6d5c99d-85d1-a37c-7f74-5f6704d8ba7a tf_backend.s3.bucket=terraform-states tf_backend.workspace-prefix=env:/ net.peer.name=terraform-states.custom_s3.server.com http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.13.3 (+https://www.terraform.io) aws-sdk-go-v2/1.36.0 ua/2.1 os/linux lang/go#1.24.5 md/GOOS#linux md/GOARCH#amd64 api/s3#1.75.2 m/C" http.request.header.amz_sdk_invocation_id=26923320-8079-416f-8eb4-b629c9dac087 http.request.header.x_amz_content_sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 http.request.body="" http.method=GET http.request.header.authorization="AWS4-HMAC-SHA256 Credential=d5094117fa03f04c4a065715fe30a4dd-cim.storage/20250920/eu-de/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=*****" http.request.header.accept_encoding=identity http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.x_amz_date=20250920T085602Z http.url="https://terraform-states.custom_s3.server.com/?list-type=2&max-keys=1000&prefix=env%3A%2F" 2025-09-20T10:56:02.555+0200 [DEBUG] backend-s3: HTTP Response Received: aws.region=eu-de aws.s3.bucket=terraform-states rpc.method=ListObjectsV2 rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Workspaces tf_backend.req_id=a6d5c99d-85d1-a37c-7f74-5f6704d8ba7a tf_backend.s3.bucket=terraform-states tf_backend.workspace-prefix=env:/ http.status_code=200 http.response_content_length=332 http.response.header.date="Sat, 20 Sep 2025 08:56:02 GMT" http.response.header.x_emc_retention_period=0 http.response.header.content_type=application/xml http.response.header.x_amz_request_id=8abebcbb:1981cebf62f:112b5c:41b7 http.response.header.x_amz_id_2=360bfb2e2c12e8198e73cb8709d10af6fbbb21b02a7aebc78f921210f8bdb7af http.response.body= | terraform-statesenv:/01000falsefalse http.duration=343 2025-09-20T10:56:02.556+0200 [INFO] backend-s3: Downloading remote state: tf_backend.operation=Get tf_backend.req_id=10271ac7-e208-aa49-72e7-a135c1918eb5 tf_backend.s3.bucket=terraform-states tf_backend.s3.path=state-file-name 2025-09-20T10:56:02.557+0200 [DEBUG] backend-s3: HTTP Request Sent: aws.region=eu-de aws.s3.bucket=terraform-states aws.s3.key=state-file-name rpc.method=HeadObject rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Get tf_backend.req_id=10271ac7-e208-aa49-72e7-a135c1918eb5 tf_backend.s3.bucket=terraform-states tf_backend.s3.path=state-file-name net.peer.name=terraform-states.custom_s3.server.com http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.13.3 (+https://www.terraform.io) aws-sdk-go-v2/1.36.0 ua/2.1 os/linux lang/go#1.24.5 md/GOOS#linux md/GOARCH#amd64 api/s3#1.75.2" http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.x_amz_content_sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 http.request.header.amz_sdk_invocation_id=4b6fc905-471f-463e-bfd1-1e2548786c48 http.request.header.authorization="AWS4-HMAC-SHA256 Credential=d5094117fa03f04c4a065715fe30a4dd-cim.storage/20250920/eu-de/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=*****" http.request.header.x_amz_date=20250920T085602Z http.request.header.accept_encoding=identity http.request.body="" http.method=HEAD http.url=https://terraform-states.custom_s3.server.com/state-file-name 2025-09-20T10:56:02.626+0200 [DEBUG] backend-s3: HTTP Response Received: aws.region=eu-de aws.s3.bucket=terraform-states aws.s3.key=state-file-name rpc.method=HeadObject rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Get tf_backend.req_id=10271ac7-e208-aa49-72e7-a135c1918eb5 tf_backend.s3.bucket=terraform-states tf_backend.s3.path=state-file-name http.duration=68 http.response_content_length=305196 http.response.header.x_emc_mtime=1748852109425 http.response.header.content_type=application/json http.response.header.date="Sat, 20 Sep 2025 08:56:02 GMT" http.response.header.x_amz_request_id=8abebc54:198183edc60:10f216:6076 http.response.header.last_modified="Mon, 02 Jun 2025 08:15:09 GMT" http.status_code=200 http.response.header.x_amz_id_2=929c3c355e96e938744782a8b828458ef9219bce593d21be1f8b21ef78d18869 http.response.header.etag="\"18f8f1a668c834f364b87c19d883f67a\"" http.response.body="" 2025-09-20T10:56:02.628+0200 [DEBUG] backend-s3: HTTP Request Sent: aws.region=eu-de aws.s3.bucket=terraform-states aws.s3.key=state-file-name rpc.method=GetObject rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Get tf_backend.req_id=10271ac7-e208-aa49-72e7-a135c1918eb5 tf_backend.s3.bucket=terraform-states tf_backend.s3.path=state-file-name http.request.header.x_amz_content_sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 http.request.header.accept_encoding=identity http.request.body="" http.method=GET http.url=https://terraform-states.custom_s3.server.com/state-file-name?x-id=GetObject http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.13.3 (+https://www.terraform.io) aws-sdk-go-v2/1.36.0 ua/2.1 os/linux lang/go#1.24.5 md/GOOS#linux md/GOARCH#amd64 api/s3#1.75.2 ft/s3-transfer m/G,b" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=d5094117fa03f04c4a065715fe30a4dd-cim.storage/20250920/eu-de/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;range;x-amz-checksum-mode;x-amz-content-sha256;x-amz-date, Signature=*****" http.request.header.x_amz_date=20250920T085602Z http.request.header.x_amz_checksum_mode=ENABLED http.request.header.amz_sdk_invocation_id=5c781bb6-77e9-4157-b5de-0d2b4e505e05 http.request.header.range=bytes=0-5242879 net.peer.name=terraform-states.custom_s3.server.com http.request.header.amz_sdk_request="attempt=1; max=5" 2025-09-20T10:56:02.726+0200 [DEBUG] backend-s3: HTTP Response Received: aws.region=eu-de aws.s3.bucket=terraform-states aws.s3.key=state-file-name rpc.method=GetObject rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Get tf_backend.req_id=10271ac7-e208-aa49-72e7-a135c1918eb5 tf_backend.s3.bucket=terraform-states tf_backend.s3.path=state-file-name http.duration=96 http.status_code=206 http.response_content_length=305196 http.response.header.x_emc_mtime=1748852109425 http.response.header.etag="\"18f8f1a668c834f364b87c19d883f67a\"" http.response.header.last_modified="Mon, 02 Jun 2025 08:15:09 GMT" http.response.header.content_type=application/json http.response.header.x_amz_request_id=8abebc4c:198183027fb:10f978:7134 http.response.header.accept_ranges=bytes http.response.header.date="Sat, 20 Sep 2025 08:56:02 GMT" http.response.header.x_amz_id_2=929c3c355e96e938744782a8b828458ef9219bce593d21be1f8b21ef78d18869 http.response.header.content_range="bytes 0-305195/305196" http.response.body="[Redacted: 298.0 KB (305,196 bytes), Type: application/json]" 2025-09-20T10:56:02.726+0200 [WARN] backend-s3: Response has no supported checksum. Not validating response payload.: aws.region=eu-de aws.s3.bucket=terraform-states aws.s3.key=state-file-name rpc.method=GetObject rpc.service=S3 rpc.system=aws-api tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Get tf_backend.req_id=10271ac7-e208-aa49-72e7-a135c1918eb5 tf_backend.s3.bucket=terraform-states tf_backend.s3.path=state-file-name 2025-09-20T10:56:02.923+0200 [DEBUG] states/remote: state read serial is: 23; serial is: 24 2025-09-20T10:56:02.924+0200 [DEBUG] states/remote: state read lineage is: bfe049f9-3de7-ae4d-29f3-db3e3608aca8; lineage is: bfe049f9-3de7-ae4d-29f3-db3e3608aca8 2025-09-20T10:56:02.934+0200 [INFO] backend-s3: Uploading remote state: tf_backend.operation=Put tf_backend.req_id=34c945f2-1169-224a-b98a-7b8c5937dad1 tf_backend.s3.bucket=terraform-states tf_backend.s3.path=state-file-name 2025-09-20T10:56:02.935+0200 [DEBUG] backend-s3: HTTP Request Sent: aws.region=eu-de aws.s3.bucket=terraform-states aws.s3.key=state-file-name rpc.method=PutObject rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Put tf_backend.req_id=34c945f2-1169-224a-b98a-7b8c5937dad1 tf_backend.s3.bucket=terraform-states tf_backend.s3.path=state-file-name http.method=PUT http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.13.3 (+https://www.terraform.io) aws-sdk-go-v2/1.36.0 ua/2.1 os/linux lang/go#1.24.5 md/GOOS#linux md/GOARCH#amd64 api/s3#1.75.2 ft/s3-transfer m/G,Z" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=d5094117fa03f04c4a065715fe30a4dd-cim.storage/20250920/eu-de/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-trailer, Signature=*****" http.request.header.accept_encoding=identity http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.x_amz_date=20250920T085602Z http.url=https://terraform-states.custom_s3.server.com/state-file-name?x-id=PutObject net.peer.name=terraform-states.custom_s3.server.com http.request.header.content_type=application/json http.request.header.x_amz_trailer=x-amz-checksum-crc32 http.request.header.x_amz_decoded_content_length=345409 http.request.header.x_amz_content_sha256=STREAMING-UNSIGNED-PAYLOAD-TRAILER http.request_content_length=345454 http.request.header.amz_sdk_invocation_id=ccd6a247-7d27-42c4-9f54-9a7146b2d53b http.request.header.content_encoding=aws-chunked http.request.body="[Redacted: 337.4 KB (345,454 bytes), Type: application/json]" 2025-09-20T10:56:03.209+0200 [DEBUG] backend-s3: HTTP Response Received: aws.region=eu-de aws.s3.bucket=terraform-states aws.s3.key=state-file-name rpc.method=PutObject rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Put tf_backend.req_id=34c945f2-1169-224a-b98a-7b8c5937dad1 tf_backend.s3.bucket=terraform-states tf_backend.s3.path=state-file-name http.status_code=400 http.response.header.content_type=application/xml http.response.header.date="Sat, 20 Sep 2025 08:56:02 GMT" http.response.header.x_amz_request_id=8abebc22:19817f1aca1:1134d0:d4f http.response.header.x_amz_id_2=929c3c355e96e938744782a8b828458ef9219bce593d21be1f8b21ef78d18869 http.response.body= | XAmzContentSHA256MismatchThe Content-SHA256 you specified did not match what we receivedterraform-states/state-file-name8abebc22:19817f1aca1:1134d0:d4f http.duration=274 2025-09-20T10:56:03.210+0200 [DEBUG] backend-s3: request failed with unretryable error https response error StatusCode: 400, RequestID: 8abebc22:19817f1aca1:1134d0:d4f, HostID: 929c3c355e96e938744782a8b828458ef9219bce593d21be1f8b21ef78d18869, api error XAmzContentSHA256Mismatch: The Content-SHA256 you specified did not match what we received: aws.region=eu-de aws.s3.bucket=terraform-states aws.s3.key=state-file-name rpc.method=PutObject rpc.service=S3 rpc.system=aws-api tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Put tf_backend.req_id=34c945f2-1169-224a-b98a-7b8c5937dad1 tf_backend.s3.bucket=terraform-states tf_backend.s3.path=state-file-name ``` ### Expected Behavior The state should be persisted in the S3 backend with custom endpoint ### Actual Behavior Failed to persist state: failed to upload state: operation error S3: PutObject, https response error StatusCode: 400, RequestID: 8abebc0c:1981842885f:10d434:1d1a, HostID: 929c3c355e96e938744782a8b828458ef9219bce593d21be1f8b21ef78d18869, api error XAmzContentSHA256Mismatch: The Content-SHA256 you specified did not match what we received ### Steps to Reproduce 1. terraform apply ### Additional Context I have not used the project for a while - I think the last time I was able to successfuly persist state in the S3, was in sometime in June 2025. This week I tried to execute the project one more time and the first thing which I have noticed that it for some reason told me that backend has been changed and I need to run "terraform init -reconfigure", although I did not make any changes into the backend. After some investigation I have found that in august 2025 I have accidently upgraded from 1.9.8 to 1.12.2 and indeed when I have rolled back to 1.9.8 S3 backend works again. I tried to upgrade to the latest version of terraform which is 1.13.3, but it did not help ### References _No response_ ### Generative AI / LLM assisted development? _No response_