Symptoms

HTTP 400/404 errors are seen when trying to read and write to ECS via S3.When list operations are running in a multi VDC environment, requests may fail with ERROR_KEYPOOL_NOT_FOUND message.

Cause

This situation is common for environments with global load balancers (balancing load between two or more VDCs), where objects are accessible on one VDC, but not accessible on the other, using the same URL, but it may be seen in any environment if a user does not belong to the target bucket's namespace.

Resolution

Query the user scope settings on the ECS: # svc_rest_cmd config/object/properties.json Example: admin@g1-ecs3:~> svc_rest_cmd config/object/properties.json svc_rest_cmd 1.0.10 { "properties": { "user_scope": "GLOBAL" }, "empty": false, "allProperties": { "user_scope": "GLOBAL" } } Use svc_request to search for the failed HTTP request IDs: # svc_request errorsummary Example: admin@g1-ecs3:~> svc_request errorsummary svc_request v1.0.11 (svc_tools v1.5.4) Started 2019-07-02 16:26:14 Time range: 2019-07-02 14:26:14 - 2019-07-02 16:26:14 Running against node(s): Resp Time Retry Node Time Request ID Type Status (ms) # Exception URL 169.254.1.2 19-07-02 14:38:41 0a51548d:16b6bc5bf6e:9b15:1 GET 404 17 0 ObjectControllerException https://testjkns3.xxxobj6.ms.com/bkt1/object1 Get details of the failing request in dataheadsvc logs. dataheadsvc logs are found here /opt/emc/caspian/fabric/agent/services/object/main/log or you can use svc_request to check the logs:# svc_request -r -start "" showlogs admin@ecs:/opt/emc/caspian/fabric/agent/services/fabric/data> svc_request -r 0a51548d:16b6bc5bf6e:9b15:1 -start 14:00 showlogs svc_request v1.0.10 (svc_tools v1.5.3) Started 2019-07-01 19:55:02 Time range: 2019-07-01 14:00:00 - 2019-07-01 19:55:01 Running against node(s): Request ID: 0a51548d:16b6bc5bf6e:9b15:1 ################################################## Reqid: 0a51548d:16b6bc5bf6e:9b15:1 Reply timestamp: 2019-07-01T14:56:41,806 Protocol: s3 Request Type: GET Size: 177 Bytes (0.000 MB) Response Time: 17 ms URL: https://testjkns3.xxxobj6.ms.com/bkt1/object1 ShortURL: https://testjkns3.xxxobj6.ms.com/bkt1/object1 Bucket: Object: /testjkns3.xxxobj6.ms.com/bkt1/object1 Options: Status: 404 Client IP: 10.xxx.xx.17 Thread ID: qtp1776502451-104390 Request (or retry) completed: No Exception: ObjectAccessException Retry #: 0 (of 0 total) Multi-part upload: - log messages: 169.254.1.7 [qtp1776502451-104390-0a51548d:16b6bc5bf6e:9b15:1-s3-10.196.105.17] ERROR ResourceClient.java (line 331) request id 0a51548d:16b6bc5bf6e:9b15:2, command type REQUEST_BUCKET_GET failed with error code ERROR_KEYPOOL_NOT_FOUND and message 'bucket xxx1.bkt1 not found' 169.254.1.7 2019-07-01T14:56:41,801 INFO KeyPoolInfoAccess.java (line 283) Cannot get keypool info from DT server since kpId xxx1.bkt1 is not found there 169.254.1.7 2019-07-01T14:56:41,801 ERROR S3Exception.java (line 942) got object access exception. RequestId 0a51548d:16b6bc5bf6e:9b15:1 com.emc.storageos.objcontrol.object.exception.ObjectAccessException: NoSuchBucket 169.254.1.7 [qtp1776502451-104390] INFO RequestLog.java (line 83) 10.196.105.17 GET https://testjkns3.xxxobj6.ms.com/bkt1/object1 HTTP/1.1 404 17 - 177 Check what is configured for base URL on both sites: # svc_rest_cmd object/baseurl Example: Output from xxxobj7 VDC, we see only a default base url svc_rest_cmd 1.0.10 urn:ObjectBaseUrl:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx DefaultBaseUrl VDC xxxobj6 has customer baseurl configured which is not configured on xxxobj7 svc_rest_cmd 1.0.10 urn:ObjectBaseUrl:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx DefaultBaseUrl urn:ObjectBaseUrl:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx xxxobj6 The load balancer was sending requests to the xxxobj7 VDC with baseurl name xxxobj6, which was not configured on that VDC. So xxxobj7 VDC was unable to find the bucket. The base URL is a VDC specific configuration. It is not replicated between VDCs. The solution is add both base URLs to both VDCs since the load balancer in this case is sending cross traffic to both VDCs. The "Use with Namespace" setting is particularly important here, because this is the setting that allows the ECS to parse the namespace from the URL provided by the client. Once these settings are correct, the client can access objects/buckets from both VDCs.

Support Cases