### Terraform Version ```shell Terraform v1.10.4 on darwin_arm64 + provider registry.terraform.io/cloudflare/cloudflare v4.51.0 + provider registry.terraform.io/hashicorp/kubernetes v2.35.1 + provider registry.terraform.io/hashicorp/random v3.6.3 ``` ### Terraform Configuration Files ```terraform terraform { backend "s3" { bucket = "" key = "/terraform.tfstate" region = "default" profile = "Terraform" skip_region_validation = true skip_credentials_validation = true skip_metadata_api_check = true skip_requesting_account_id = true use_path_style = true skip_s3_checksum = true endpoints = { # s3 = "https://.arkobasu.space" <-- This doesn't work s3 = "http://192.168.5.81:80" <-- This works } } required_providers { random = { source = "hashicorp/random" } cloudflare = { source = "cloudflare/cloudflare" version = "~> 4" } kubernetes = { source = "hashicorp/kubernetes" version = "2.35.1" } } } provider "cloudflare" { api_token = var.cf_api_token } provider "kubernetes" { config_path = "~/.kube/config" } ``` ### Debug Output ```bash 2025-02-03T00:37:25.557-0600 [DEBUG] backend-s3: HTTP Request Sent: aws.region=default aws.s3.bucket= rpc.method=ListObjectsV2 rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Workspaces tf_backend.req_id=05ad803a-067d-c682-41aa-fab081edf53a tf_backend.s3.bucket= tf_backend.workspace-prefix=env:/ http.request.header.authorization="AWS4-HMAC-SHA256 Credential=/20250203/default/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=*****" http.request.header.x_amz_content_sha256= http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.x_amz_date=20250203T063725Z http.url="https://.arkobasu.space/?list-type=2&max-keys=1000&prefix=env%3A%2F" http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.10.4 (+https://www.terraform.io) m/C aws-sdk-go-v2/1.31.0 os/macos lang/go#1.23.3 md/GOOS#darwin md/GOARCH#arm64 api/s3#1.63.0" http.request.header.amz_sdk_invocation_id=8c5bcb89-da72-4cab-8227-74a9974c963d http.request.header.accept_encoding=identity http.request.body="" http.method=GET net.peer.name=.arkobasu.space 2025-02-03T00:37:25.736-0600 [DEBUG] backend-s3: HTTP Response Received: aws.region=default aws.s3.bucket= rpc.method=ListObjectsV2 rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Workspaces tf_backend.req_id=05ad803a-067d-c682-41aa-fab081edf53a tf_backend.s3.bucket= tf_backend.workspace-prefix=env:/ http.response.header.cf_cache_status=DYNAMIC http.response.header.x_amz_request_id=tx000004599d10cd4712119-0067a06425-1192799-default http.response.header.server=cloudflare http.duration=178 http.status_code=403 http.response.header.nel="{\"success_fraction\":0,\"report_to\":\"cf-nel\",\"max_age\":604800}" http.response.header.alt_svc="h3=\":443\"; ma=86400" http.response_content_length=219 http.response.header.cf_ray=90c0698bee2d124e-ORD http.response.header.report_to="{\"endpoints\":[{\"url\":\"https:\/\/a.nel.cloudflare.com\/report\/v4?s=bkYoWKv7FpD9Lr8Vj8bmE8KPocsIWMjJegF5%2FeUrO8Tn5GsnSAba%2B4w9sGeZcVC4n85JSWq0EHbRCydcGWdct9N7bGDqLQ8qnQkjmhEEFDWCiCnWyYi7cPmgyOe%2Bz0BqJIxpVKiEBhSPsbWj1k%2FRtw%3D%3D\"}],\"group\":\"cf-nel\",\"max_age\":604800}" http.response.header.date="Mon, 03 Feb 2025 06:37:25 GMT" http.response.body= | SignatureDoesNotMatchtx000004599d10cd4712119-0067a06425-1192799-default1192799-default-default http.response.header.content_type=application/xml http.response.header.accept_ranges=bytes http.response.header.x_envoy_upstream_service_time=34 http.response.header.server_timing="cfL4;desc=\"?proto=TCP&rtt=25278&min_rtt=17126&rtt_var=8980&sent=8&recv=12&lost=0&retrans=0&sent_bytes=4263&recv_bytes=2255&delivery_rate=169645&cwnd=254&unsent_bytes=0&cid=6d94aa0b9071c6de&ts=116&x=0\"" 2025-02-03T00:37:25.736-0600 [DEBUG] backend-s3: request failed with unretryable error https response error StatusCode: 403, RequestID: tx000004599d10cd4712119-0067a06425-1192799-default, HostID: 1192799-default-default, api error SignatureDoesNotMatch: UnknownError: aws.region=default aws.s3.bucket= rpc.method=ListObjectsV2 rpc.service=S3 rpc.system=aws-api tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Workspaces tf_backend.req_id=05ad803a-067d-c682-41aa-fab081edf53a tf_backend.s3.bucket= tf_backend.workspace-prefix=env:/ ╷ │ Error: Failed to get existing workspaces: Unable to list objects in S3 bucket "" with prefix "env:/": operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: tx000004599d10cd4712119-0067a06425-1192799-default, HostID: 1192799-default-default, api error SignatureDoesNotMatch: UnknownError │ │ ``` ### Expected Behavior Should be able to use Ceph S3 Bucket for Backend State management ### Actual Behavior I am able to use AWS CLI and other AWS SDKs - like boto3 and dart's [aws_signature_v4 ](https://pub.dev/packages/aws_signature_v4) without any issues. But when I use Terraform backend it throws the error. I don't have the issue when I am using the IP address based endpoint. ### Steps to Reproduce 1. `terraform init -migrate-state` ### Additional Context It's been working great. I have 2 RGW gateways exposed to internet using [Cloudflare Tunnels](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/). I am able to use AWS CLI and SDKs (both dart and python) to interact with it using a Cloudflare DNS.. the setup is simple. I have a cloudflare tunnel running on a Kubernetes cluster (separate from the nodes actually running the RGW Gateways) -> Offloads to a Envoy Proxy -> that load balances between my RGW Gateway instances. ```bash cat ~/.aws/config [profile Terraform] endpoint_url = https://.arkobasu.space region = default output = json alpha@Arkos-MacBook-Pro workspace ``` I can confirm that Accounts and IAM API are also functional. So this profile for example is created under a new Account, by the root user of the account -> that then created this user. I have been stuck on this for a bit. I have tried everything. The Signature validation does work when I am using something like Presign for objects and accessing them over the browser. I would appreciate it very much if you could give me some direction. ### References _No response_ ### Generative AI / LLM assisted development? _No response_