Loading...
Loading...
During the installation of Dell NativeEdge, you may encounter a stall while the Helm chart rolls out. The latest message of the installation script execution is looking like: Operation Type: INSTALL Operation Status: IN_PROGRESS ChartKey: NEO Operation Type: INSTALL Operation Status: FAILED ChartKey: NEO native-edge-orchestrator has NOT deployed and has status: failed .... Saving logs to files Cleaning up the execution log file
In some environments, the container POD blocks external DNS requests. To verify this issue, check the logs of the pod stuck during deployment: kubectl get po -A ... nativeedge hzp-product-svc-595b7b888d-tpz88 2/3 Running 8 (54m ago) 80m ... nativeedge fusion-stage-backend-845dfff4d7-9zqww 1/2 CrashLoopBackOff 20 (4m34s ago) 79m ... kubectl logs fusion-stage-backend-845dfff4d7-9zqww -c hzp-iam-proxy -n hzp | grep error -B17 ... runtime/debug.Stack() /usr/local/go/src/runtime/debug/stack.go:26 +0x5e eos2git.cec.lab.emc.com/ISG-Edge/hzp-iam-proxy/server/commons/utils.LogRecover(...) /app/server/commons/utils/utils.go:37 eos2git.cec.lab.emc.com/ISG-Edge/hzp-iam-proxy/server/services/proxysvc.(*proxyService).pipeConnections.func1() /app/server/services/proxysvc/service.go:236 +0x174 panic({0xdbb580?, 0x177f950?}) /usr/local/go/src/runtime/panic.go:785 +0x132 io.copyBuffer({0x0, 0x0}, {0x0, 0x0}, {0xc00057e000, 0x2000, 0x2000}) /usr/local/go/src/io/io.go:429 +0x17e io.CopyBuffer({0x0?, 0x0?}, {0x0?, 0x0?}, {0xc00057e000?, 0x1?, 0x1?}) /usr/local/go/src/io/io.go:402 +0x36 eos2git.cec.lab.emc.com/ISG-Edge/hzp-iam-proxy/server/services/proxysvc.(*proxyService).pipeConnections(0xc0001b0b00, {0x10a17a8, 0xc0005a7200}, 0xc0003f4b00, 0xc0003f4b10, {0xecbefe, 0x2}, 0xc000483f20?) /app/server/services/proxysvc/service.go:254 +0x2ae created by eos2git.cec.lab.emc.com/ISG-Edge/hzp-iam-proxy/server/services/proxysvc.(*proxyService).proxyConnections in goroutine 7423 /app/server/services/proxysvc/service.go:224 +0x1db{"level":"error","error":"no such file or directory","time":"2025-05-09T13:16:16Z","message":"Error getting socket from operating system"} {"level":"error","error":"no such file or directory","time":"2025-05-09T13:16:16Z","message":"failed to get destination address from outgoing connection"} ... ## Note: if during the installation you specified your own namespace, then you will have to provide your namespace after -n argument key.kubectl log fusion-stage-backend-845dfff4d7-9zqww -c hzp-iam-proxy -n hzp | grep "Forwarding connection id" ... {"level":"info","time":"2025-05-09T13:16:16Z","message":"Forwarding connection id: 37da2adc-9f56-4e93-8875-83dd597c1aab, source: 161.200.192.67:43732, destination: , direction: Outbound, encryption: NotDetermined"} ...## Note: the destination is empty in this part: destination: , direction: Outbound, encryption: NotDetermined"} ## Also note the "error getting socket from operating system, this is another symptom of this particular issue Note, in this installation fusion-stage-backend and hzp-product-svc pods are not coming up. Check the logs of the pod stuck during deployment to verify this issue. For this kind of errors in the hzp-iam-proxy container, there is a known workaround for this type of issue.
To fix this issue, do the following steps. Restart the installation script to pick up the latest state and wait until the same installation step starts. Operation Type: INSTALL Operation Status: IN_PROGRESS ChartKey: NEO Start monitoring the pods are coming up with kubectl get po -A command. When you see that some pod is not coming up, check that the logs from the 'Cause' section appear in the logs. When you identified this issue on step 3, do these steps for three deployments: fusion-stage-backend, hzp-catalog-svc, hzp-product-svc If you see the error only within one of the pods, ensure that you add the following annotations to all three pods. kubectl edit deployment name_of_deployment -nhzp # For example: kubectl edit deployment fusion-stage-backend -nhzp # Find the spec.template.metadata.annotations section: # For example, you should see something like: .... spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: stage-backend strategy: .... type: RollingUpdate template: metadata: annotations: hzp.iam.webhook/authentication-enabled: "false" hzp.iam.webhook/authorization-enabled: "false"....# Add annotation 'hzp.iam.webhook/exclude-outbound-ports: "443"' to this section as below: spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: stage-backend strategy: ... type: RollingUpdate template: metadata: annotations: hzp.iam.webhook/authentication-enabled: "false" hzp.iam.webhook/authorization-enabled: "false" hzp.iam.webhook/exclude-outbound-ports: "443" # check the deployment is restarting. For example: kubectl get po -A | grep fusion-stage hzp fusion-stage-backend-5887976b57-4shlp 2/2 Running 0 23s hzp fusion-stage-backend-745ff99c8f-dkl7t 2/2 Terminating 4 (7d21h ago) 7d21h This unblocks external DNS lookup queries and lets your installation continue. After the pod restarts, the installation will continue. Additional notes: There is evidence that the problem happens after the installation completes. This means that this problem may affect other pods. In case you see same footprint in the logs of the pod, you must perform same steps for the deployment. Example of error appearing in the other pod below: {"level":"debug","time":"2025-06-04T12:37:33Z","message":"Accepted connection with flow id: c565d0f3-d568-42e0-a0a6-21cd0080b6ee, remote address: 161.200.27.4:41880, local address: 127.0.0.1:15001, direction: Outbound"} recovered while piping connections, flow id: %sa6178189-f353-46d2-8c07-20e67a8c72d9(0xdbb580,0x177f950) {"level":"debug","time":"2025-06-04T12:37:33Z","message":"Open connections: 17"}goroutine 513252 [running]:runtime/debug.Stack() /usr/local/go/src/runtime/debug/stack.go:26 +0x5eeos2git.cec.lab.emc.com/ISG-Edge/hzp-iam-proxy/server/commons/utils.LogRecover(...) /app/server/commons/utils/utils.go:37eos2git.cec.lab.emc.com/ISG-Edge/hzp-iam-proxy/server/services/proxysvc.(*proxyService).pipeConnections.func1() /app/server/services/proxysvc/service.go:236 +0x174 panic({0xdbb580?, 0x177f950?}) /usr/local/go/src/runtime/panic.go:785 +0x132io.copyBuffer({0x0, 0x0}, {0x0, 0x0}, {0xc00064a000, 0x2000, 0x2000}) /usr/local/go/src/io/io.go:429 +0x17eio.CopyBuffer({0x0?, 0x0?}, {0x0?, 0x0?}, {0xc00064a000?, 0x1?, 0x1?}) /usr/local/go/src/io/io.go:402 +0x36eos2git.cec.lab.emc.com/ISG-Edge/hzp-iam-proxy/server/services/proxysvc.(*proxyService).pipeConnections(0xc0001e6580, {0x10a17a8, 0xc0004811a0}, 0xc00050c930, 0xc00050c8f0, {0xecbefc, 0x2}, 0xc0001f4b70?) /app/server/services/proxysvc/service.go:254 +0x2aecreated by eos2git.cec.lab.emc.com/ISG-Edge/hzp-iam-proxy/server/services/proxysvc.(*proxyService).proxyConnections in goroutine 513234 /app/server/services/proxysvc/service.go:225 +0x271 {"level":"debug","time":"2025-06-04T12:37:33Z","message":"Open connections: 16"} {"level":"info","time":"2025-06-04T12:37:33Z","message":"Redirect Address before lookup 161.210.184.19:80"} recovered while piping connections, flow id: %sa6178189-f353-46d2-8c07-20e67a8c72d9(0xdbb580,0x177f950)goroutine 513251 [running]:runtime/debug.Stack() /usr/local/go/src/runtime/debug/stack.go:26 +0x5eeos2git.cec.lab.emc.com/ISG-Edge/hzp-iam-proxy/server/commons/utils.LogRecover(...) /app/server/commons/utils/utils.go:37eos2git.cec.lab.emc.com/ISG-Edge/hzp-iam-proxy/server/services/proxysvc.(*proxyService).pipeConnections.func1() /app/server/services/proxysvc/service.go:236 +0x174 panic({0xdbb580?, 0x177f950?}) /usr/local/go/src/runtime/panic.go:785 +0x132io.copyBuffer({0x0, 0x0}, {0x0, 0x0}, {0xc00071c000, 0x2000, 0x2000}) /usr/local/go/src/io/io.go:429 +0x17eio.CopyBuffer({0x0?, 0x0?}, {0x0?, 0x0?}, {0xc00071c000?, 0x1?, 0x1?}) /usr/local/go/src/io/io.go:402 +0x36eos2git.cec.lab.emc.com/ISG-Edge/hzp-iam-proxy/server/services/proxysvc.(*proxyService).pipeConnections(0xc0001e6580, {0x10a17a8, 0xc0004811a0}, 0xc00050c8f0, 0xc00050c930, {0xecbefe, 0x2}, 0xc0001f47e0?) /app/server/services/proxysvc/service.go:254 +0x2aecreated by eos2git.cec.lab.emc.com/ISG-Edge/hzp-iam-proxy/server/services/proxysvc.(*proxyService).proxyConnections in goroutine 513234 /app/server/services/proxysvc/service.go:224 +0x1db {"level":"error","error":"no such file or directory","time":"2025-06-04T12:37:33Z","message":"Error getting socket from operating system"}## Note the error "getting socket from operation system" again Preventive troubleshooting Instead of waiting for failure during the installation, you may start doing steps 2-4 during your first installation. When this message shown on the screen for the 10 minutes Operation Type: INSTALL Operation Status: IN_PROGRESS ChartKey: NEO Start monitoring the pods. If you see the footprint of the issue in the fusion-stage-backend-svc and product-svc pods, apply the annotations for both of the pods. Also apply the annotation to the catalog-svc deployment, which may encounter this issue later.
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.