Symptoms

This article only applies after cancelling a VRM upgrade. VxVerify has an issue with lcm_state:#========================#=======#========#====================================================================#==============# | Hostname / Category |Status Dell_KB | Warnings or Failures, unless tests Passed ; Product S.N. | #========================#=======#========#====================================================================#==============# | VxRM | Failure 194221 | lcm_state: Upgrade LCM API call error .|In vxv.log reports:2023-05-04 23:59:28-ERROR [lcm_state] API Error: status code = 404 API requested resource was not found.No more log entry has been added in lcm-web.log at this stage, seems to stop generating new logs.Step 4 ~ 6 confirming canceled LCM status: The state is NONE in the bundle_state.jsonvrm:/home/mystic # cat /var/lib/vmware-marvin/bundle_state.json | jq { "state": "NONE", "deployed_for_public_api": false }There is no IN_PROGRESS LCM task in the database:vrm:/home/mystic # psql -U postgres vxrail vxrail=# select id,owner,state,progress,start_time,end_time from system.operation_status where owner like 'Lcm%';id | owner | state | progress | start_time | end_time ----+-------+-------+----------+------------+---------- (0 rows)No lock info in the lockbox:vrm:/var/lib/vmware-marvin # curl -X GET --unix-socket /var/lib/vxrail/nginx/socket/nginx.sock "http://127.0.0.1:5000/rest/vxm/internal/lockservice/v1/lock" {"error_code": 1001, "message": "Failed to get the lock information due to the lock does not exist."}

Cause

LCM webapp has failed.

Resolution

To resolve this issue follow the steps below: Note: The below commands are for VRM's version starting from 7.0.370. In the command examples below, replace vcadmin@localos with your management account and vxm01.vxrail.local with your VxRail Manager FQDN. Confirm that the LCM was canceled by following KB: VxRail: How to Cancel a VxRail 7.0.X/8.0.X Upgrade Manually Without Using the Cancel Buttonor as documented in the steps 4 ~ 6 in the symptoms above.All the Pod must be running. Run the command: kubectl get Pod -o widevxm01:~ # kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES api-gateway-b757f8446-n5srv 1/1 Running 7 60d 172.28.175.54 vxrail <none> <none> auth-service-785796d47b-wt4fq 1/1 Running 24 60d 172.28.175.153 vxrail <none> <none> cacheservice-6666c8f46d-cz9ph 1/1 Running 7 60d 172.28.175.217 vxrail <none> <none> cms-service-659c67c6cf-sbfzv 1/1 Running 40 60d 172.28.175.58 vxrail <none> <none> cmsvcupgrade-service-7b5cd8db66-zg67k 1/1 Running 24 60d 172.28.175.226 vxrail <none> <none> dell-ese-58f4f5464-hsjvq 1/1 Running 7 60d 172.28.175.218 vxrail <none> <none> ... ... ...Restart the marvin and runjars services to refresh lcm webapp status:vrm: # systemctl restart vmware-marvinvrm: # systemctl restart runjarsRun the below curl command.The expected output is "Request method 'GET' not supported"vxm01:~ # curl -v -k -u vcadmin@localos https://vxm01.vxrail.local/rest/vxm/v1/lcm/upgrade Enter host password for user 'vcadmin@localos': * Trying xxx.xxx.xxx:443... * TCP_NODELAY set * Connected to vxm01.vxrail.local (xxx.xxx.xxx) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): ..... ..... < * Connection #0 to host vxm01.vxrail.local left intact Request method 'GET' not supportedRun the below curl command. The expected return is "{" state ": " NONE "}"vxm01:~ # curl -v -k -u vcadmin@localos https://127.0.0.1/rest/vxm/private/lcm/status Enter host password for user 'vcadmin@localos': * Trying 127.0.0.1:443... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): ... * Connection #0 to host 127.0.0.1 left intact {"state":"NONE"}Rerun the VxVerify, the error should disappear.

Support Cases