Loading...
Loading...
After ~365 days, REST API polling and/or gNMI telemetry begins to fail (REST/gNMI not responding). Workaround restores services temporarily: Docker restart mgmt-framework and docker restart telemetry. Observation/comment: REST and Telemetry cert validity on all GA releases is 1 year by default. Note: Certificate validity cannot be changed using crypto cert generate request. That command produces a CSR; validity is controlled by the issued certificate (CA/self-signed process), not by SONiC CLI. Crypto commands are run from the SONiC Management Framework CLI (sonic-cli); running them in a Linux shell may fail (for example, command not found). API - Application Programming Interface CSR - Certificate Signing Request
The REST and Telemetry TLS certificate validity on all GA releases is 1 year (365 days) by default. After expiry, REST/gNMI clients may fail TLS validation or the service may stop responding as expected. Restarting mgmt-framework and telemetry containers acts as a workaround that renews/reinitializes the certificates and temporarily restores REST/gNMI connectivity for another year. Memory growth in the mgmt-framework container (observed using show histogram memory docker) may contribute to service instability and increases the likelihood of failure near the time window. Initial Configuration: Platform: E3248P (E3248P family) Image: Enterprise SONiC 4.4.0 Lite Services impacted: REST API polling/gNMI telemetry Certificates: default/self-signed used for REST and telemetry (1-year validity by default) Command Used: Workaround: docker restart mgmt-framework docker restart telemetry Verification - Option 1 (Recommended): openssl s_client -connect <IP:443> # Check NotAfter in certificate output Example output (certificate validity): CONNECTED(00000003) Can't use SSL_get_servername depth=0 O = Example Co verify error:num=18:self-signed certificate verify return:1 --- Certificate chain 0 s:O = Example Co i:O = Example Co v:NotBefore: Feb 3 17:22:48 2025 GMT; NotAfter: Feb 3 17:22:48 2027 GMT Verification - Option 2: 1) Open browser and access the switch REST API explorer 2) Inspect certificate validity dates
IMPORTANT: Perform certificate install and security-profile configuration from SONiC MF-CLI (sonic-cli), not Linux shell. Step 1: Create a certificate with 2 years validity from outside the switch. Any server linux PC/VM openssl req -x509 -nodes -newkey rsa:4096 \ -keyout hostkey.pem \ -out hostcert.pem \ -days 730 \ -subj "/CN=sonic-switch.mydomain.com"(or) If self-signed CA is available with extended validity, generate CSR on switch: crypto cert generate request cert-file home://hostcert.pem key-file home://hostkey.pem cname sonic-switch.mydomain.com Step 2: Install host certificate: crypto cert install cert-file home://hostcert.pem key-file home://hostkey.pem # Expected: Installed host certificate as "hostcert" Step 3: Create security profile and associate certificate: conf t crypto security-profile REST-TLS crypto security-profile certificate REST-TLS hostcert Step 4: Apply security profile to REST server: ip rest security-profile REST-TLS Step 5: Apply security profile to telemetry/gNMI. ip telemetry security-profile REST-TLS After performing commands: Verify on switch: show crypto cert all show crypto security-profile REST-TLS show ip rest show ip telemetry Expected behavior: REST API and gNMI telemetry respond normally and present the newly installed certificate. Resolution: Confirm that failure coincides with certificate expiry (~365 days) using openssl s_client -connect <IP:443> or browser certificate viewer. Use container restart workaround for immediate recovery: Docker restart mgmt-framework and docker restart telemetry. For a permanent fix, install a new host certificate and bind it through crypto security-profile to REST and Telemetry services. For extended validity (example: 2 years), generate the certificate externally (OpenSSL/PKI) or use an internal self-signed CA with extended validity; SONiC CLI does not set certificate validity in CSR generation.
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
BugZero Plan
Streamline upgrades with automated vendor bug scrubs
BugZero Prevent
Wish you caught this bug sooner? Get proactive today.