Loading...
Loading...
after upgrading controllers from 20.9.4.1 to 20.12.3, seeing custom-apps missing from the vmanage and device as well..
ERROR DbRecoveryExport:44 - fail to zip src:/scratch/postgresql/recovery, dst:/tmp/sdavc\_db\_recovery\_1720653872955.zip java.nio.file.AccessDeniedException: /scratch/postgresql/recovery at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?] at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:432) ~[?:?] at java.nio.file.Files.newDirectoryStream(Files.java:472) ~[?:?] bash-5.0# tail -f -n 100 avc_client.log 2024-07-10 23:24:31 INFO Main:91 - system start time:Wed Jul 10 23:24:31 GMT 2024 2024-07-10 23:24:31 INFO Main:103 - client info: {"version":"b485ee25c28b04959f4cccbf443e872c2b96c2a4","agentId":"ed12ee3aa743bf61312cbda6778f7eb0","hostname":"d26ab4ae53f1","startTimeMs":1720653871216}, branch:b485ee25c28b04959f4cccbf443e872c2b96c2a4, build time:2023-02-22T10:30:56-0500 2024-07-10 23:24:31 INFO NbarHttpClient:87 - create http client 2024-07-10 23:24:32 INFO ConfigService:728 - got new sdwan token response 2024-07-10 23:24:32 INFO ConfigService:800 - setting dynamic config:{"sdavcDomain":"prod-us1.sdwan-sdavc-saas.com","sdavcPort":-1,"sdavcHttpPort":-1,"serverHttpsExternalPort":-1} 2024-07-10 23:24:32 INFO ConfigService:680 - successfully connected to sdavc (prod-us1.sdwan-sdavc-saas.com), got new token 2024-07-10 23:24:32 INFO Main:295 - config: {"sdavcDomain":null,"sdavcPort":443,"sdavcProtocol":"https","sdavcHttpPort":8080,"sdwanAuthUrl":"https://datamanagement-us-01.sdwan.cisco.com/validate_sdavc/","sdwanSpOrgName":"xxxxxxxxxxxxx","saasModeEnabled":true,"httpProxy":null,"httpsProxy":null,"noProxy":null,"httpExportPktQueueCapacity":10000,"exportFileQueueCapacity":1000,"exportFileMaxTtlSec":1800,"exportMaxDiskSizeBytes":524288000,"serverHttpsPort":8443,"serverHttpsExternalPort":10502,"serverHttpPort":8080,"serverUdpExportPort":50000,"serverHttpsTransparentPort":8444,"httpClientConnectTimeoutSec":30,"httpClientReadTimeoutSec":120,"httpClientWriteTimeoutSec":-1,"httpClientCallTimeoutSec":-1,"proxyModeEnabled":false,"tokenLifetimeMillis":43200000,"httpExportFileSizeThresholdBytes":10485760,"httpExportFileTimeThresholdMsec":2000,"fileCacheDisabled":false,"httpClientMaxRequests":310,"httpClientMaxRequestsPerHost":300,"dbRecoveryRetryIntervalSe c":90,"certKeyStoreFileLocation":"/scratch/certificate/sdavcGw.p12","systemLoggingIntervalSec":600,"dbRecoveryExportEnabled":true,"transparentPortEnabled":false,"importHandlerBypass":false,"exportUploadBypass":false,"exportFileCompressionLevel":1,"httpClientForceDisableCertificateVerification":false,"logExportEnabled":true,"logExportIntervalSec":86400,"logExportInitialDelaySec":600,"logExportLimitBytes":10000000} 2024-07-10 23:24:32 INFO Main:175 - starting export collectors 2024-07-10 23:24:32 INFO Main:184 - starting network listeners 2024-07-10 23:24:32 INFO NbarHttpsServer:228 - HTTPS server started on port 8443 2024-07-10 23:24:32 INFO NbarHttpServer:48 - HTTP server started on port 8080 2024-07-10 23:24:32 INFO Main:198 - init done 2024-07-10 23:24:32 INFO ConfigService:850 - token created. cycle sleep (ms):120000 2024-07-10 23:24:32 INFO ConfigService:858 - sleep for 719 cycles. max sleep time:86399521 2024-07-10 23:24:32 ERROR DbRecoveryExport:44 - fail to zip src:/scratch/postgresql/recovery, dst:/tmp/sdavc_db_recovery_1720653872955.zip java.nio.file.AccessDeniedException: /scratch/postgresql/recovery at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?] at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:432) ~[?:?] at java.nio.file.Files.newDirectoryStream(Files.java:472) ~[?:?] at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:300) ~[?:?] at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322) ~[?:?] at java.nio.file.Files.walkFileTree(Files.java:2717) ~[?:?] at java.nio.file.Files.walkFileTree(Files.java:2797) ~[?:?] at com.cisco.nbar.sdavcClient.export.DbRecoveryExport.zipFolder(DbRecoveryExport.java:35) [sdavcClient-0.0.1-SNAPSHOT.jar:?] at com.cisco.nbar.sdavcClient.export.DbRecoveryExport.run(DbRecoveryExport.java:67) [sdavcClient-0.0.1-SNAPSHOT.jar:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2024-07-10 23:24:32 ERROR DbRecoveryExport:68 - zip failed 2024-07-10 23:24:32 INFO LogExport:37 - found log file. name:/scratch/logs/avc_client/avc_client.log customer is manually making changes to custom apps and which is pushing the app list to vmanage
upgrading controllers
Workaround after the upgrade to 20.12.3 and otp/url configured in administration =>settings =>sd-avc 1. Correct the file permission of the recovery folder, on all vManages of the cluster. vmanage: cd /opt/data/containers/mounts/sdavc vmanage: chmod 755 recovery vmanage:/opt/data/containers/mounts/sdavc# ls -l total 8 drwx------ 20 vmanage vmanage 4096 Sep 15 17:24 data drwxr-xr-x 2 vmanage-admin vmanage 4096 Sep 15 17:25 recovery 2. Restart the sdavc-gw service # request nms sdavc-gw restart
None
Cisco Integration
Learn more about where this data comes from
BugZero Plan
Streamline upgrades with automated vendor bug scrubs
BugZero Prevent
Wish you caught this bug sooner? Get proactive today.