Symptoms
A restore operation from AppSync GUI will eventually hang and fail due to a timeout error.The AppSync events will show:
,Restore Copy,fc520a7d-b08d-41ab-9878-70608fdf8ea6,ORCL_000093,,AppSync failed to restore one or all of the following storage filesystems: /my/path/data99,
In server logs will see it's timing out:
INFO [EJB default - 8] [com.emc.archway.xio_commands.RestoreXIOSnapshotCommandBean] [AppSync_Server] [] d5f58e3c-43bd-4e2a-998f-4908473c3af0->>> finished restore/refresh volumes on XIO Array
INFO [EJB default - 8] [com.emc.archway.xio_commands.RestoreXIOSnapshotCommandBean] [AppSync_Server] [] d5f58e3c-43bd-4e2a-998f-4908473c3af0->>> finished restore/refresh volumes on XIO Array
WARN [EJB default - 8] [com.arjuna.ats.arjuna] [AppSync_Server] [] ARJUNA012077: Abort called on already aborted atomic action 0:ffff0a25bf41:-396b7557:61391564:19b548
ERROR [EJB default - 8] [org.jboss.as.ejb3.invocation] [AppSync_Server] [] WFLYEJB0034: EJB Invocation failed on component RestoreXIOSnapshotCommandBean for method public boolean com.emc.archway.xio_commands.RestoreXIOSnapshotCommandBean.restorePhasePIT(com.emc.archway.context.ExecutionContext,java.util.Set,boolean): javax.ejb.EJBTransactionRolledbackException: javax.transaction.RollbackException: WFLYEJB0447: Transaction 'Local transaction (delegate=TransactionImple , owner=Local transaction context for provider JBoss JTA transaction provider)' was already rolled back at org.jboss.as.ejb3@17.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:114)
Cause
A number of snapshot were involved in the restore operation which caused delay/timeout. Possible cause not determined however this could be a performance/slowness issue on resources like array and host.
Resolution
Adjusting the transaction timeout value will allow AppSync not to timeout during restore operation.
Go to C:\EMC\AppSync\jboss\standalone\configuration\standalone.xml.Change the value from default of 300 to new value 2400 seconds - Stop all the Services (EMC AppSync Datastore and EMC AppSync Security service)Start the services(EMC AppSync Datastore Service, EMC AppSync Security Service, EMC AppSync server service)
