Symptoms

Fresh Install of Solutions Enabler and Unisphere for PowerMax v10 on Linux SUSE 15 SP3 server. Solutions Enabler v10 is working properly but Unisphere for PowerMax v10 has issues. Confirmed from Product Management that SUSE 15 SP3 is compatible with Unisphere v10.Installation of Both Solutions Enabler v10 and Unisphere for PowerMax v10 had no error or warning messages.SMAS service does not stay up. It runs for a couple of seconds then goes down. SMAS_DB service stays OK. Therefore, Unisphere for PowerMax v10 was inaccessible.See below output for /etc/init.d/smas start server_only and /etc/init.d/smas start_db_only: From GRABS provided by customer:SMAS Log: Caused by: javax.ejb.EJBException: java.lang.RuntimeException: SymInitialize failed with code 28 (SYMAPI_C_DB_PERMISSION_DENIED) Caused by: java.lang.RuntimeException: SymInitialize failed with code 28 (SYMAPI_C_DB_PERMISSION_DENIED) Caused by: SymInitialize failed with code 28 (SYMAPI_C_DB_PERMISSION_DENIED)" 2022-09-27 16:09:27,238 ERROR [org.jboss.as.ejb3.invocation] (ServerService Thread Pool -- 35) WFLYEJB0034: Jakarta Enterprise Beans Invocation failed on component RemoteSFSService for method public abstract void com.emc.em.domain.symmetrix.remote.sfsrpc.IRemoteSFSService.setRemoteRequestRouter(com.emc.em.domain.symmetrix.remote.internal.IIncomingRemoteObjectRouter): javax.ejb.EJBException: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance 2022-09-27 16:09:27,252 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 76) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "domain-symm0.ear")]) - failure description: {"WFLYCTL0080: Failed services" => { 2022-09-27 16:09:27,255 ERROR [org.jboss.as.server] (ServerService Thread Pool -- 76) WFLYSRV0021: Deploy of deployment "domain-symm.ear" was rolled back with the following failure message: 2022-09-27 16:09:27,532 ERROR [com.emc.em.deployer.ServerManagerService] (EJB default - 1) ServerManagerService.attemptDeployingArchive(): Failed to deploy domain-symm0.ear with exit value 1 2022-09-27 16:09:27,534 ERROR [com.emc.em.deployer.ServerManagerService] (EJB default - 1) ServerManagerService.deployArchive(): failed to attempt deploying with controller status failed 2022-09-27 16:09:27,539 ERROR [com.emc.em.deployer.DeployService] (EJB default - 1) DeployService.deployArchive(): Failed to deploy domain-symm0.ear 2022-09-27 16:09:27,540 ERROR [com.emc.em.deployer.DeployService] (EJB default - 1) DeployService.deployArchives(): Failed to deploy archive domain-symm 2022-09-27 16:09:27,540 ERROR [com.emc.em.deployer.DeployService] (EJB default - 1) DeployService.deploy(): Failed to deploy: com.emc.em.deployer.DeployServiceException: Failed to deploy domain-symm0.ear 2022-09-27 16:09:27,542 ERROR [com.emc.em.deployer.sharding.ShardingService] (EJB default - 1) IDeployStatusListener.notifyDeployStatus(): notified of failed DeploymentStatus--shutting down the domain... Run the log: [Server:server-0] [31m[0m[31m16:09:27,236 ERROR [org.jboss.as.ejb3.invocation] (ServerService Thread Pool -- 35) WFLYEJB0034: Jakarta Enterprise Beans Invocation failed on component SymapiSessionService for method public abstract com.emc.em.domain.symmetrix.symapi.ISymapiSessionPool com.emc.em.domain.symmetrix.symapi.ISymapiSessionService.getSymapiSessionPool(): javax.ejb.EJBException: java.lang.RuntimeException: SymInitialize failed with code 28 (SYMAPI_C_DB_PERMISSION_DENIED)[0m [Server:server-0] [31m[0m[31m16:09:27,238 ERROR [org.jboss.as.ejb3.invocation] (ServerService Thread Pool -- 35) WFLYEJB0034: Jakarta Enterprise Beans Invocation failed on component RemoteSFSService for method public abstract void com.emc.em.domain.symmetrix.remote.sfsrpc.IRemoteSFSService.setRemoteRequestRouter(com.emc.em.domain.symmetrix.remote.internal.IIncomingRemoteObjectRouter): javax.ejb.EJBException: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance[0m [Server:server-0] [31m[0m[31m16:09:27,239 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 35) MSC000001: Failed to start service jboss.deployment.subunit."domain-symm.ear"."domain-symm-ejb.jar".component.RemoteUnisphereService.START: org.jboss.msc.service.StartException in service jboss.deployment.subunit."domain-symm.ear"."domain-symm-ejb.jar".component.RemoteUnisphereService.START: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance [Server:server-0] [31m[0m[31m16:09:27,255 ERROR [org.jboss.as.server] (ServerService Thread Pool -- 76) WFLYSRV0021: Deploy of deployment "domain-symm.ear" was rolled back with the following failure message: [0m [Server:server-0] [31m[0m[31m16:09:27,532 ERROR [com.emc.em.deployer.ServerManagerService] (EJB default - 1) ServerManagerService.attemptDeployingArchive(): Failed to deploy domain-symm0.ear with exit value 1[0m [Server:server-0] [31m[0m[31m16:09:27,534 ERROR [com.emc.em.deployer.ServerManagerService] (EJB default - 1) ServerManagerService.deployArchive(): failed to attempt deploying with controller status failed[0m [Server:server-0] [31m[0m[31m16:09:27,539 ERROR [com.emc.em.deployer.DeployService] (EJB default - 1) DeployService.deployArchive(): Failed to deploy domain-symm0.ear[0m [Server:server-0] [31m[0m[31m16:09:27,540 ERROR [com.emc.em.deployer.DeployService] (EJB default - 1) DeployService.deployArchives(): Failed to deploy archive domain-symm[0m [Server:server-0] [31m[0m[31m16:09:27,540 ERROR [com.emc.em.deployer.DeployService] (EJB default - 1) DeployService.deploy(): Failed to deploy: com.emc.em.deployer.DeployServiceException: Failed to deploy domain-symm0.ear[0m [Server:server-0] [31m[0m[31m16:09:27,542 ERROR [com.emc.em.deployer.sharding.ShardingService] (EJB default - 1) IDeployStatusListener.notifyDeployStatus(): notified of failed DeploymentStatus--shutting down the domain...[0m Deployer Log: 2022-09-27 16:08:50,388 INFO [com.emc.em.deployer.DeploymentInitiator] (ServerService Thread Pool -- 76) DeploymentInitiator.init() uses error reporting service and gets: false 2022-09-27 16:08:53,300 INFO [com.emc.em.deployer.ServerManagerService] (EJB default - 1) ServerManagerService.attemptDeployingArchive(): Error stream ROOT0.war 2022-09-27 16:09:27,497 INFO [com.emc.em.deployer.ServerManagerService] (EJB default - 1) ServerManagerService.getLine(): Caused by: java.lang.RuntimeException: SymInitialize failed with code 28 (SYMAPI_C_DB_PERMISSION_DENIED) 2022-09-27 16:09:27,497 INFO [com.emc.em.deployer.ServerManagerService] (EJB default - 1) ServerManagerService.getLine(): Caused by: SymInitialize failed with code 28 (SYMAPI_C_DB_PERMISSION_DENIED)"}}}}}}}}}}}} 2022-09-27 16:09:27,532 ERROR [com.emc.em.deployer.ServerManagerService] (EJB default - 1) ServerManagerService.attemptDeployingArchive(): Failed to deploy domain-symm0.ear with exit value 1 2022-09-27 16:09:27,534 ERROR [com.emc.em.deployer.ServerManagerService] (EJB default - 1) ServerManagerService.deployArchive(): failed to attempt deploying with controller status failed 2022-09-27 16:09:27,539 ERROR [com.emc.em.deployer.DeployService] (EJB default - 1) DeployService.deployArchive(): Failed to deploy domain-symm0.ear 2022-09-27 16:09:27,540 ERROR [com.emc.em.deployer.DeployService] (EJB default - 1) DeployService.deployArchives(): Failed to deploy archive domain-symm 2022-09-27 16:09:27,540 ERROR [com.emc.em.deployer.DeployService] (EJB default - 1) DeployService.deploy(): Failed to deploy: com.emc.em.deployer.DeployServiceException: Failed to deploy domain-symm0.ear 2022-09-27 16:09:27,542 ERROR [com.emc.em.deployer.sharding.ShardingService] (EJB default - 1) IDeployStatusListener.notifyDeployStatus(): notified of failed DeploymentStatus--shutting down the domain... Notice that SYMAPI_C_DB_PERMISSION_DENIED entry is logged several times.

Cause

Analysis: Server requirements were met. That is no issue with disk space or RAM on concerned standalone Server.Port 8443 was used by another monitoring software and hence customer was using port 9443 for Unisphere on this Standalone LINUX Server./tools/list/splunkforwarder/etc/apps/Splunk_TA_nix/bin/netstat.sh:netstat -an | grep 9443 did not show any result meaning the port is not in LISTENING as SMAS service is down.Went to /var/symapi and ran ls -ll to check database permissions and it looked fineUnder install_dir/SMAS/jboss/standalone/data/pgsql, verified if postgres user is configured as per Unisphere Install guide (see page 20) Doing so said that the user already exists. Cd /etc then vi passwd to check postgres, and it looked good as wellFurther, /emc/SMAS/jboss/standalone/data/pgsql/data then ls -ll to verify the permission of the user. Back to /etc to verify as below: Root looked ok here as well.

Resolution

Workaround: As per discussion with the customer, uninstalled both Unisphere for PowerMax v10 and SE v10 to test installing SE v9.2 and Unisphere for PowerMax v9.2.Removed SMAS folder (only the SMAS)Verified noexec is disabled in /tmp (mount | grep /tmp)Successfully installed Solutions Enabler v9.2 on the same server and Unisphere for PowerMax v9.2 as well. Both Solutions Enabler v9.2 and Unisphere for PowerMax v9.2 worked properly.Verified the arrays are listed correctly on both Solutions Enabler v9.2 and Unisphere UI v9.2Successfully upgraded Solutions Enabler from v9.2 to v10. Verified Solutions Enabler works fine after upgrade too.Further, Initiated Unisphere for PowerMax upgrade from v9.2 to target v10. Upgrade went smooth without any errors or warnings.Both Solutions Enabler v9.2 and Unisphere for PowerMax v9.2 upgrade succeeded to v10After upgrade, Solutions Enabler v10 and Unisphere for PowerMax v10 were working without any issues.

Support Cases