### Terraform Version ```shell Terraform v1.12.0 on linux_amd64 ``` ### Terraform Configuration Files ```terraform /** # .env file # https://github.com/hashicorp/terraform/issues/36704 AWS_REQUEST_CHECKSUM_CALCULATION="when_required" AWS_RESPONSE_CHECKSUM_VALIDATION="when_required" # Used by Terraform when authenticating with the backend's service AWS_ACCESS_KEY_ID="" AWS_SECRET_ACCESS_KEY="" # Used by Terraform's `b2` provider (not needed) B2_APPLICATION_KEY_ID="" B2_APPLICATION_KEY="" */ terraform { backend "s3" { key = "terraform.tfstate" region = "us-east-003" # Not used by Backblaze B2, but a required attribute for Terraform. use_lockfile = true # Skip all validations because this backend is not 100% compatible with Backblaze B2. skip_credentials_validation = true skip_metadata_api_check = true skip_region_validation = true skip_requesting_account_id = true skip_s3_checksum = true } required_providers { b2 = { source = "Backblaze/b2" version = "~> 0.8" } } } provider "b2" {} resource "b2_bucket" "example" { bucket_name = "example" bucket_type = "allPrivate" } ``` ### Debug Output ``` 2025-05-20T20:37:09.422-0300 [TRACE] backend/local: requesting state lock for workspace "default" 2025-05-20T20:37:09.422-0300 [INFO] backend-s3: Attempting to lock remote state (S3 Native only)...: tf_backend.lock.id=f409289d-71e7-456a-ca57-d1aea984c354 tf_backend.lock.info="" tf_backend.lock.operation=OperationTypePlan tf_backend.lock.path=/terraform.tfstate tf_backend.lock.version=1.12.0 tf_backend.lock.who= tf_backend.operation=Lock tf_backend.req_id=fea83614-5e35-9b20-40ae-140315bc9f68 tf_backend.s3.bucket= tf_backend.s3.path=terraform.tfstate 2025-05-20T20:37:09.422-0300 [DEBUG] backend-s3: Uploading lock file: tf_backend.lock.id=f409289d-71e7-456a-ca57-d1aea984c354 tf_backend.lock.info="" tf_backend.lock.operation=OperationTypePlan tf_backend.lock.path=/terraform.tfstate tf_backend.lock.version=1.12.0 tf_backend.lock.who= tf_backend.operation=Lock tf_backend.req_id=fea83614-5e35-9b20-40ae-140315bc9f68 tf_backend.s3.bucket= tf_backend.s3.path=terraform.tfstate 2025-05-20T20:37:09.423-0300 [DEBUG] backend-s3: HTTP Request Sent: aws.region=us-east-003 aws.s3.bucket= aws.s3.key=terraform.tfstate.tflock rpc.method=PutObject rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.lock.id=f409289d-71e7-456a-ca57-d1aea984c354 tf_backend.lock.info="" tf_backend.lock.operation=OperationTypePlan tf_backend.lock.path=/terraform.tfstate tf_backend.lock.version=1.12.0 tf_backend.lock.who= tf_backend.operation=Lock tf_backend.req_id=fea83614-5e35-9b20-40ae-140315bc9f68 tf_backend.s3.bucket= tf_backend.s3.path=terraform.tfstate http.request_content_length=239 http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.content_type=application/json http.request.header.if_none_match="*" http.request.header.amz_sdk_invocation_id=c9f3ff3d-4b3a-4c02-b5c0-b65ac639b1cb http.request.header.accept_encoding=identity net.peer.name=s3.us-west-004.backblazeb2.com http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.12.0 (+https://www.terraform.io) aws-sdk-go-v2/1.36.0 ua/2.1 os/linux lang/go#1.24.2 md/GOOS#linux md/GOARCH#amd64 api/s3#1.75.2 ft/s3-transfer m/G,a" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=/20250520/us-east-003/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;if-none-match;x-amz-content-sha256;x-amz-date, Signature=*****" http.request.header.x_amz_date=20250520T233709Z http.request.header.x_amz_content_sha256=UNSIGNED-PAYLOAD http.request.body="[Redacted: 239 bytes, Type: application/json]" http.method=PUT http.url=https://s3.us-west-004.backblazeb2.com//terraform.tfstate.tflock?x-id=PutObject 2025-05-20T20:37:09.641-0300 [DEBUG] backend-s3: HTTP Response Received: aws.region=us-east-003 aws.s3.bucket= aws.s3.key=terraform.tfstate.tflock rpc.method=PutObject rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.lock.id=f409289d-71e7-456a-ca57-d1aea984c354 tf_backend.lock.info="" tf_backend.lock.operation=OperationTypePlan tf_backend.lock.path=/terraform.tfstate tf_backend.lock.version=1.12.0 tf_backend.lock.who= tf_backend.operation=Lock tf_backend.req_id=fea83614-5e35-9b20-40ae-140315bc9f68 tf_backend.s3.bucket= tf_backend.s3.path=terraform.tfstate http.response_content_length=196 http.response.header.content_type=application/xml http.response.header.cache_control="max-age=0, no-cache, no-store" http.response.header.connection=keep-alive http.response.body= | | | NotImplemented | A header you provided implies functionality that is not implemented | http.status_code=501 http.response.header.x_amz_id_2=aMhFlKDXROZEzRmMfMm8zJzQtMbY5SDT7 http.response.header.strict_transport_security=max-age=63072000 http.response.header.x_amz_request_id=e408a33b38c2df62 http.response.header.server=nginx http.response.header.date="Tue, 20 May 2025 23:37:09 GMT" http.duration=218 2025-05-20T20:37:09.641-0300 [DEBUG] backend-s3: request failed with unretryable error https response error StatusCode: 501, RequestID: e408a33b38c2df62, HostID: aMhFlKDXROZEzRmMfMm8zJzQtMbY5SDT7, api error NotImplemented: A header you provided implies functionality that is not implemented: aws.region=us-east-003 aws.s3.bucket= aws.s3.key=terraform.tfstate.tflock rpc.method=PutObject rpc.service=S3 rpc.system=aws-api tf_aws.sdk=aws-sdk-go-v2 tf_backend.lock.id=f409289d-71e7-456a-ca57-d1aea984c354 tf_backend.lock.info="" tf_backend.lock.operation=OperationTypePlan tf_backend.lock.path=/terraform.tfstate tf_backend.lock.version=1.12.0 tf_backend.lock.who= tf_backend.operation=Lock tf_backend.req_id=fea83614-5e35-9b20-40ae-140315bc9f68 tf_backend.s3.bucket= tf_backend.s3.path=terraform.tfstate 2025-05-20T20:37:09.642-0300 [DEBUG] backend-s3: HTTP Request Sent: aws.region=us-east-003 aws.s3.bucket= aws.s3.key=terraform.tfstate.tflock rpc.method=GetObject rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.lock.id=f409289d-71e7-456a-ca57-d1aea984c354 tf_backend.lock.info="" tf_backend.lock.operation=OperationTypePlan tf_backend.lock.path=/terraform.tfstate tf_backend.lock.version=1.12.0 tf_backend.lock.who= tf_backend.operation=Lock tf_backend.req_id=fea83614-5e35-9b20-40ae-140315bc9f68 tf_backend.s3.bucket= tf_backend.s3.path=terraform.tfstate net.peer.name=s3.us-west-004.backblazeb2.com http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.12.0 (+https://www.terraform.io) aws-sdk-go-v2/1.36.0 ua/2.1 os/linux lang/go#1.24.2 md/GOOS#linux md/GOARCH#amd64 api/s3#1.75.2 m/c" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=/20250520/us-east-003/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=*****" http.request.header.accept_encoding=identity http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.x_amz_date=20250520T233709Z http.request.body="" http.method=GET http.url=https://s3.us-west-004.backblazeb2.com//terraform.tfstate.tflock?x-id=GetObject http.request.header.x_amz_content_sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 http.request.header.amz_sdk_invocation_id=61ab0579-d65a-4687-be7c-8e487a187dd3 Acquiring state lock. This may take a few moments... 2025-05-20T20:37:09.846-0300 [DEBUG] backend-s3: HTTP Response Received: aws.region=us-east-003 aws.s3.bucket= aws.s3.key=terraform.tfstate.tflock rpc.method=GetObject rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.lock.id=f409289d-71e7-456a-ca57-d1aea984c354 tf_backend.lock.info="" tf_backend.lock.operation=OperationTypePlan tf_backend.lock.path=/terraform.tfstate tf_backend.lock.version=1.12.0 tf_backend.lock.who= tf_backend.operation=Lock tf_backend.req_id=fea83614-5e35-9b20-40ae-140315bc9f68 tf_backend.s3.bucket= tf_backend.s3.path=terraform.tfstate http.duration=204 http.response.header.date="Tue, 20 May 2025 23:37:09 GMT" http.response.header.content_type=application/xml http.response.header.cache_control="max-age=0, no-cache, no-store" http.response.header.x_amz_id_2="aMttlwTViOVMz/2PpMiQz+jQ8MX05rTTQ" http.response.body="[Redacted: 137 bytes, Type: application/xml]" http.status_code=404 http.response_content_length=137 http.response.header.strict_transport_security=max-age=63072000 http.response.header.connection=keep-alive http.response.header.x_amz_request_id=15ae8997ca9c7168 http.response.header.server=nginx 2025-05-20T20:37:09.846-0300 [DEBUG] backend-s3: request failed with unretryable error https response error StatusCode: 404, RequestID: 15ae8997ca9c7168, HostID: aMttlwTViOVMz/2PpMiQz+jQ8MX05rTTQ, NoSuchKey: : aws.region=us-east-003 aws.s3.bucket= aws.s3.key=terraform.tfstate.tflock rpc.method=GetObject rpc.service=S3 rpc.system=aws-api tf_aws.sdk=aws-sdk-go-v2 tf_backend.lock.id=f409289d-71e7-456a-ca57-d1aea984c354 tf_backend.lock.info="" tf_backend.lock.operation=OperationTypePlan tf_backend.lock.path=/terraform.tfstate tf_backend.lock.version=1.12.0 tf_backend.lock.who= tf_backend.operation=Lock tf_backend.req_id=fea83614-5e35-9b20-40ae-140315bc9f68 tf_backend.s3.bucket= tf_backend.s3.path=terraform.tfstate ╷ │ Error: Error acquiring the state lock │ │ Error message: operation error S3: PutObject, https response error StatusCode: 501, RequestID: e408a33b38c2df62, HostID: │ aMhFlKDXROZEzRmMfMm8zJzQtMbY5SDT7, api error NotImplemented: A header you provided implies functionality that is not implemented │ unable to retrieve file from S3 bucket '' with key 'terraform.tfstate.tflock': operation error S3: │ GetObject, https response error StatusCode: 404, RequestID: 15ae8997ca9c7168, HostID: aMttlwTViOVMz/2PpMiQz+jQ8MX05rTTQ, NoSuchKey: │ │ Terraform acquires a state lock to protect the state from being written │ by multiple users at the same time. Please resolve the issue above and try │ again. For most commands, you can disable locking with the "-lock=false" │ flag, but this is not recommended. ``` ### Expected Behavior Terraform should be able to upload a state lockfile to a Backblaze B2 bucket using its S3-compatible API ### Actual Behavior Terraform is unable to acquire the state lock because the remote server does not recognize all headers in the PutObject request uploading the lockfile to the bucket ### Steps to Reproduce https://gist.github.com/pdmtt/7d48b95bd8c0592671f659e21d8a965e Simplified version: ```bash set -e BUCKET_NAME="terraformBackend" TERRAFORM_BACKEND_CONFIG_FILE="terraform-backend-config.tfvars" TERRAFORM_BACKEND_CREDENTIALS_FILE="terraform-backend-credentials.env" # Install `b2` utility in a virtual environment. python3 -m venv .venv source .venv/bin/activate pip3 install --upgrade --quiet b2 # Interactive authentication flow. Use an application key that can create buckets. b2 account authorize # Create the bucket that will be used to store Terraform's files. b2 bucket create --default-server-side-encryption SSE-B2 "$BUCKET_NAME" allPrivate # https://github.com/hashicorp/terraform/issues/36704 for varName in AWS_REQUEST_CHECKSUM_CALCULATION AWS_RESPONSE_CHECKSUM_VALIDATION; do echo "$varName=\"when_required\"" >> "$TERRAFORM_CREDENTIALS_FILE" done # Complete Terraform's partial configuration. # More info at https://developer.hashicorp.com/terraform/language/backend#partial-configuration. echo "endpoints = { s3 = $(b2 account get | jq .s3endpoint) }" >> "$TERRAFORM_BACKEND_CONFIG_FILE" echo "bucket = \"${BUCKET_NAME}\"" >> "$TERRAFORM_BACKEND_CONFIG_FILE" # Create a more restrictive application key to allow Terraform's access to the bucket. mapfile -d ' ' -t arrKey <<< "$(b2 key create --bucket "$BUCKET_NAME" agenticCollabTerraformKey listBuckets,listAllBucketNames,readFiles,writeFiles | tr -d '\n')" # Variables that will be used by Terraform to authenticate with its backend storage service. echo "$AWS_ACCESS_KEY_ID=\"${arrKey[0]}\"" >> "$TERRAFORM_CREDENTIALS_FILE" echo "$AWS_SECRET_ACCESS_KEY=\"${arrKey[1]%$'\n'}\"" >> "$TERRAFORM_CREDENTIALS_FILE" # According to Terraform's documentation, this is the recommended way to supply credentials: # > Warning: We recommend using environment variables to supply credentials and other sensitive # > data. If you use -backend-config or hardcode these values directly in your configuration, # > Terraform will include these values in both the .terraform subdirectory and in plan files. # https://developer.hashicorp.com/terraform/language/backend/s3#credentials-and-shared-configuration set -a source "$TERRAFORM_BACKEND_CREDENTIALS_FILE" terraform init -backend-config="$TERRAFORM_BACKEND_CONFIG_FILE" ``` ### Additional Context I'm opting into using the bucket to store the lock information instead of DynamoDB: `use_lockfile = true`. ### References I found an [issue in `minio`](https://github.com/minio/minio/issues/18636) similar to this (server not implementing a S3 behaviour). It has been suggested that the client should ignore the error, but I believe this is not the case here. Instead, it seems to me that it should be possible to exclude some headers from the PutObject request. ### Generative AI / LLM assisted development? _No response_