Symptoms

After upgrade to ECS 3.6.2.3, ECS 3.7.0.0 or ECS 3.7.0.1, S3 applications may show error: (HTTP 403) The request signature we calculated does not match the signature we provided.This only affects applications using signature version 4. s3cmd --host 10.246.151.145:9020 ls s3://restic ERROR: S3 error: 403 (SignatureDoesNotMatch): The request signature we calculated does not match the signature you provided. Check your Secret Access Key and signing method. For more information, see REST Authentication and SOAP# From ECS logs, "?location" request are successful but other PUT/GET requests fail: Resp Bucket/ Size Time Object/ Node Time Request ID Prot Type MPU Client IP Status (bytes) (ms) Options 10.x.x.x 04-20 10:10:47 0af69791:1802f264522:3d4a:8cb s3 GET - 10.x.x.x 200 330 4 restic/?location 10.x.x.x 04-20 10:10:47 0af69791:1802f264522:3c18:b99 s3 GET - 10.x.x.x 403 330 2 restic/?delimiter=%2F Search for the error 403: svc_log -f 0af69791:1802f264522:3c18:b99 -sr dataheadsvc svc_log v1.0.26 (svc_tools v2.3.0) Started 2022-04-20 10:15:07 Running on nodes: Time range: 2022-04-19 10:15:07 - 2022-04-20 10:15:07 Filter string(s): '0af69791:1802f264522:3c18:b99' Show nodename(s): True Search reclaim logs (if any): False 169.x.x.x 2022-04-20T10:10:47,896 [qtp2066748233-27517-0af69791:1802f264522:3c18:b99-s3-10.x.x.x] ERROR V4Signer.java (line 335) Signature mismatch CalcSignature: 692c3f2795f0d41f83202e82b6643f24cfe9e74074b0752e92b1a81d20b861db, ClientSignature: 08bfabe59c94a9b3e36d47be9f570c1f1b9dd93928d267462073ca3a84076f46, StringToSign AWS4-HMAC-SHA256 169.x.x.x 2022-04-20T10:10:47,896 [qtp2066748233-27517-0af69791:1802f264522:3c18:b99-s3-10.x.x.x] ERROR V4Signer.java (line 335) Signature mismatch CalcSignature: 692c3f2795f0d41f83202e82b6643f24cfe9e74074b0752e92b1a81d20b861db, ClientSignature: 08bfabe59c94a9b3e36d47be9f570c1f1b9dd93928d267462073ca3a84076f46, StringToSign AWS4-HMAC-SHA256

Cause

Signature version 4 incorporates the bucket region into the authentication.In ECS 3.6.2.3, ECS 3.7.0.0 and ECS 3.7.0.1, changes were made to the bucket-location API. The response from the API is currently " ", causing the signature mismatch.Invalid request: Authorization: AWS4-HMAC-SHA256 Credential=mathias/20220419/ /s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED** A valid request is formed including the region: Authorization: AWS4-HMAC-SHA256 Credential=mathias/20220419/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**

Resolution

The fix for this issue is ECS 3.7.0.2 and above. There are two options for a workaround:1. The first option is to not use signature version 4 and use signature version 2 instead, if applicable. 2. The second option is to configure a default location.Check the documentation for your application how to properly set the region. The default Region is "us-east-1"Examples: minio mc: https://docs.min.io/docs/python-client-api-reference.html restic: set variable AWS_DEFAULT_REGION to the region or -o s3.region=""

Support Cases