Symptoms

In the GeoDrive Console during file uploads, some files might fail to upload to ECS preventing the files from being stubbed. Returning the following error below. They are two methods to check this condition. GeoDrive Console ECS backend using svc_log (ECS log search utility) This message is logged in the GeoDrive Console (UI - User Interface) GeoDrive Console user interface: Error writing to file in ECS server (VDC): e:\cifs-ecs data\j_drive\test\300-014-769_03[1].pdf 2149122451 - Forbidden (403). ECS: 2070 - The request signature we calculated does not match the signature you provided. Check your Secret Access Key and signing method. For more information, see REST Authentication and SOAP Authentication for details. Http: 403 - Forbidden: The server understood the request, but refuses to fulfill it. S3 Code: SignatureDoesNotMatch S3 RequestID: 0a3f1196:1688930d135:579c0:2d The command below is performed on the ECS back-end by logging into one of the ECS data nodes. By searching on the phrase "Signature mismatch" in the data head service logs (brief explanation of parameters used). Where -f is the string, message to search ('Signature mismatch').Where -sf (show log file name in output when match is found).Where -sr (specifies service log or logs to search" in this case we are checking dataheadsvc logs).Where -n (nodes to search in this, case we targeted node 1 in rack 1 (r1n1) or option can be -n all (to search all nodes).Where -start (specifies time frame or range to search. In this example, we searched logs going back 2 days which can be adjusted as needed. To match ones searching criteria).Where -C10 (Display 10 lines before and After matched filter string in -f "Signature mismatch.") svc_log output: # svc_log -f 'Signature mismatch' -sf -sr dataheadsvc -n r1n1 -start '2 days ago' -C10 dataheadsvc.log.20190403-020724.gz 2019-04-02T21:38:15,071 [qtp1418678112-602089] INFO RequestLog.java (line 83) 10.x.x.193 PUT //10.x.x.x:9020/moose_s3_bucket/test/300-014-769_03%5B1%5D.pdf HTTP/1.1 403 1 - 330 dataheadsvc.log.20190403-020724.gz 2019-04-02T21:38:15,087 [qtp1418678112-602073-0a3f1196:1688930d135:57a7f:55-s3-10.x.x.193] ERROR S3V2Signer.java (line 318) Signature mismatch CalcSignature: 2Da0D1GFb48eCfMzLSm3K7492bc=, ClientSignature: Ot0o6rsuYll2kpQnw6cufM5G+08= dataheadsvc.log.20190403-020724.gz 2019-04-02T21:38:15,088 [qtp1418678112-602073-0a3f1196:1688930d135:57a7f:55-s3-10.x.x.193] ERROR HMACAuthenticationHandler.java (line 195) authenticate failed. RequestId 0a3f1196:1688930d135:57a7f:55. Error com.emc.storageos.data.api.service.impl.resource.s3.S3Exception at com.emc.storageos.data.api.service.impl.resource.s3.authentication.HMACAuthenticationHandler.authenticate(HMACAuthenticationHandler.java:159) at com.emc.storageos.data.api.service.impl.resource.s3.S3AuthenticationFilter.doFilter(S3AuthenticationFilter.java:104) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) at com.emc.storageos.data.api.service.impl.resource.s3.S3BaseUrlFilter.doFilter(S3BaseUrlFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) at com.emc.storageos.data.api.service.impl.resource.utils.TransactionContextFilter.doFilter(TransactionContextFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) at com.emc.storageos.data.api.service.impl.resource.s3.S3RequestFilter.doFilter(S3RequestFilter.java:67) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) at com.emc.storageos.data.object.impl.ResponseCodeRecordingFilter.doFilter(ResponseCodeRecordingFilter.java:84) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:541) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1584) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1228) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:481) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1130) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:118) at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:564) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:112) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:122) at org.eclipse.jetty.util.thread.strategy.ExecutingExecutionStrategy.invoke(ExecutingExecutionStrategy.java:58) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:201) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:133) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:748) svc_request output: Time range: 2019-04-02 19:22:06 - 2019-04-02 21:22:06 Running against node(s): Request ID: 0a3f1196:1688930d135:579d5:43 ################################################## Reqid: 0a3f1196:1688930d135:579d5:43 Reply timestamp: 2019-04-02T21:17:20,642 Protocol: s3 Request Type: PUT Size: - Bytes (0.000 MB) Response Time: 9 ms URL: //10.x.x.x:9020/moose_s3_bucket/test/300-014-769_03%5B1%5D.pdf ShortURL: moose_s3_bucket/test/300-014-769_03%5B1%5D.pdf Bucket: bucket Object: test/300-014-769_03%5B1%5D.pdf Options: Status: 403 Client IP: 10.x.x.193 Thread ID: qtp1418678112-601861 Request (or retry) completed: No Exception: Retry #: 0 (of 0 total) Multi-part upload: No

Cause

The following characters were not encoded appropriately for HTTP requests. Which leads to Http: 403 - Forbidden, S3 Code: SignatureDoesNotMatch (authentication failed.)During file uploads: # pound [] square brackets

Resolution

Permanent Fix: The recommendation is to upgrade to the most current version of Geodrive. Current version is: GeoDrive 2.0.2 for Windows as of 8/05/2019 Always check the https://support.emc.com/downloads/17166_Dell-EMC-GeoDrive for the most up-to-date version Workaround: If unable to upgrade immediately. Files with square brackets "[]" or pound "#" sign can be remained by deleting these characters. This allows the files to upload to ECS successfully until a time to upgrade is identified.

Support Cases