Loading...
Loading...
Upgrade from Dell Automation Platform 1.2 to 2.0 fails or takes excessively long to complete, no matter it is a bundle deployed environment or OVA deployment environment.This issue only affects the upgrade process, so if the upgrade is complete this issue no longer applies.NATS Jetstream pod (nats-jetstream-0) enters CrashLoopBackOff, blocking initialization of dependent services.Upgrade jobs related to NATS stream operations (backup/restore, scaling replicas) take a long time or stall.Environment contains large number of NATS streams (for example, thousands).Sometimes, upgrade fails due to NATS configuration parsing error (for example, unquoted password starting with a digit) Users should use below command to confirm the number of NATS streams curl -s "http://$(kubectl get svc jetstream -n dap10 -o jsonpath='{.spec.clusterIP}'):8222/jsz" | grep -o '"streams":[^,}]*'
The drift service (drift‑svc) creates a new NATS KV stream for every check_drift execution.In environments with frequent updates (for example, every ~5 minutes), automatic drift checks result in rapid accumulation of orphaned streams. The root cause of why frequent updates are run is still under investigation Over time, this leads to thousands of streams (for example, 17k+) being present in the environment ideally it should be less than 100. During upgrade, the process: Scans back up, restores, and modifies all streams.Performs operations such as rescaling stream replicas. The large volume of streams increases processing time, causing upgrade delays or failures.Also, configuration issues (for example, unquoted password values) can cause NATS pod startup failures.
Permanent Fix (Recommended) Upgrade to a fixed version: Dell Automation Platform 2.0.0.1 or later This fix prevents creation of excessive new NATS streams during runtime. **Important:** This fix does NOT remove existing streams. Cleanup may still be required before upgrade (see below). Cleanup Required for Existing Environments The fix prevents new stream creation, but existing streams must be cleaned up: Run this command to get the streams count curl -s "http://$(kubectl get svc jetstream -n dapo -o jsonpath='{.spec.clusterIP}'):8222/jsz" | grep -o '"streams":[^,}]*' If streams counts are high, run script nats-stream-delete.sh as root user before upgrade. Download the knowledge base attachment nats-stream-delete_pkb.zip and then upload to the Dell Automation Platform VM (either OVA or Bundle).Unzip the file (nats-stream-delete_pkb.zip) and run the script with elevated permissions for example.: root@DAP_Server:/tmp# bash ./nats-stream-delete.sh Note: Name space is hardcoded in the script as "dapo". If you are using different namespace, edit the script (nats-stream-delete.sh) and change the name space accordingly. Please verify the stream counts curl -s "http://$(kubectl get svc jetstream -n dapo -o jsonpath='{.spec.clusterIP}'):8222/jsz" | grep -o '"streams":[^,}]*' Please proceed with a fresh DAP 1.2 to 2.0 upgrade.
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.