OPERATIONAL DEFECT DATABASE
...
...
Deployment fails with 2023-11-29 08:36:50 ngfw-job-scheduler-worker-1: INFO NgfwJob:201 - DeploymentImmediateJob status update: Deployment Task: 'User (admin) Triggered Deployment' Failed after 30m 9.673s - IN_PROGRESS
If the configuration has object groups in access to 20k hosts and the customer tried to modify many such rules. 2023-11-29 00:31:44 deploy-task-worker-2: INFO AccessPolicyPreDeltaResolver:91 - scanNegatedRules - Starts 2023-11-29 01:08:53 deploy-task-worker-2: INFO AccessPolicyPreDeltaResolver:145 - scanNegatedRules - Ends. numNegatedBlockRules is 13, isAnySrcNetwork is false, isAnyDestNetwork is true info=[task=[running=[RUNNING ON deploy-task-worker-2] >> This function was running beyond 30 min causing the deployment failure.
N/A
2023-11-29 08:36:44 DeploymentRunner: ERROR DeploymentController:182 - Deployment Error: {}Waited 1800000 milliseconds for com.google.common.util.concurrent.CollectionFuture$ListFuture@477030db[status=PENDING, info=[futures=[[com.google.common.util.concurrent.TrustedListenableFutureTask@55974026[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]], com.google.common.util.concurrent.TrustedListenableFutureTask@4003aee0[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]], com.google.common.util.concurrent.TrustedListenableFutureTask@1b156899[status=PENDING, info=[task=[running=[RUNNING ON deploy-task-worker-2], com.cisco.ngfw.onbox.deploy.tasks.LinaCliExportTask@7181b8a3]]], com.google.common.util.concurrent.TrustedListenableFutureTask@5da575c8[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]]]]]] 2023-11-29 08:36:44 DeploymentRunner: ERROR NormalDeploymentController:169 - Failed to complete deployment: java.util.concurrent.TimeoutException: Waited 1800000 milliseconds for com.google.common.util.concurrent.CollectionFuture$ListFuture@477030db[status=PENDING, info=[futures=[[com.google.common.util.concurrent.TrustedListenableFutureTask@55974026[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]], com.google.common.util.concurrent.TrustedListenableFutureTask@4003aee0[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]], com.google.common.util.concurrent.TrustedListenableFutureTask@1b156899[status=PENDING, info=[task=[running=[RUNNING ON deploy-task-worker-2], com.cisco.ngfw.onbox.deploy.tasks.LinaCliExportTask@7181b8a3]]], com.google.common.util.concurrent.TrustedListenableFutureTask@5da575c8[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]]]]]] 2023-11-29 08:36:44 DeploymentRunner: INFO NormalDeploymentController:175 - Posted event to revert back the event rate after deploy through NGFWEventBus 2023-11-29 08:36:44 DeploymentRunner: ERROR DeploymentRunner:230 - DeploymentException occurred while executing deployment request : 39fdf936-8e8e-11ee-9e41-f1f51281e2ca DeploymentException:java.util.concurrent.TimeoutException: Waited 1800000 milliseconds for com.google.common.util.concurrent.CollectionFuture$ListFuture@477030db[status=PENDING, info=[futures=[[com.google.common.util.concurrent.TrustedListenableFutureTask@55974026[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]], com.google.common.util.concurrent.TrustedListenableFutureTask@4003aee0[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]], com.google.common.util.concurrent.TrustedListenableFutureTask@1b156899[status=PENDING, info=[task=[running=[RUNNING ON deploy-task-worker-2], com.cisco.ngfw.onbox.deploy.tasks.LinaCliExportTask@7181b8a3]]], com.google.common.util.concurrent.TrustedListenableFutureTask@5da575c8[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]]]]]] 2023-11-29 08:36:44 DeploymentRunner: ERROR DeploymentRunner:231 - DeploymentException:java.util.concurrent.TimeoutException: Waited 1800000 milliseconds for com.google.common.util.concurrent.CollectionFuture$ListFuture@477030db[status=PENDING, info=[futures=[[com.google.common.util.concurrent.TrustedListenableFutureTask@55974026[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]], com.google.common.util.concurrent.TrustedListenableFutureTask@4003aee0[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]], com.google.common.util.concurrent.TrustedListenableFutureTask@1b156899[status=PENDING, info=[task=[running=[RUNNING ON deploy-task-worker-2], com.cisco.ngfw.onbox.deploy.tasks.LinaCliExportTask@7181b8a3]]], com.google.common.util.concurrent.TrustedListenableFutureTask@5da575c8[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]]]]]] at com.cisco.ngfw.onbox.deploy.controllers.DeploymentController.doInParallel(DeploymentController.java:183) at com.cisco.ngfw.onbox.deploy.controllers.DeploymentController.doInParallel(DeploymentController.java:146) at com.cisco.ngfw.onbox.deploy.controllers.NormalDeploymentController.executeTasks(NormalDeploymentController.java:193) at com.cisco.ngfw.onbox.deploy.controllers.NormalDeploymentController.execute(NormalDeploymentController.java:167) at com.cisco.ngfw.onbox.deploy.controllers.NormalDeploymentController$$FastClassBySpringCGLIB$$879e94a2.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) at com.cisco.ngfw.onbox.deploy.controllers.NormalDeploymentController$$EnhancerBySpringCGLIB$$2ec6309.execute() at com.cisco.ngfw.onbox.deploy.DeploymentRunner.executeDeploymentTask(DeploymentRunner.java:225) at com.cisco.ngfw.onbox.deploy.DeploymentRunner.run(DeploymentRunner.java:132) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.concurrent.TimeoutException: Waited 1800000 milliseconds for com.google.common.util.concurrent.CollectionFuture$ListFuture@477030db[status=PENDING, info=[futures=[[com.google.common.util.concurrent.TrustedListenableFutureTask@55974026[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]], com.google.common.util.concurrent.TrustedListenableFutureTask@4003aee0[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]], com.google.common.util.concurrent.TrustedListenableFutureTask@1b156899[status=PENDING, info=[task=[running=[RUNNING ON deploy-task-worker-2], com.cisco.ngfw.onbox.deploy.tasks.LinaCliExportTask@7181b8a3]]], com.google.common.util.concurrent.TrustedListenableFutureTask@5da575c8[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]]]]]] at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:468) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:90) at com.cisco.ngfw.onbox.deploy.controllers.DeploymentController.doInParallel(DeploymentController.java:171) ... 14 more 2023-11-29 08:36:44 eventbus-service-2: INFO SensorEventManager:122 - com.cisco.sensor.event.engine.SensorEventManager 2023-11-29 08:36:44 eventbus-service-2: INFO SensorEventManager:125 - com.cisco.sensor.event.engine.SensorEventManager 2023-11-29 08:36:44 DeploymentRunner: INFO NormalDeploymentController:249 - Performing clean up of deployment with success = false 2023-11-29 08:36:44 DeploymentRunner: INFO BaseDeploymentRepositoryImpl:434 - Updating deploymentStatus:126371 DEPLOY_FAILED 2023-11-29 08:36:44 DeploymentRunner: INFO BaseDeploymentRepositoryImpl:454 - Updated DeploymentStatus:126371 39fdf936-8e8e-11ee-9e41-f1f51281e2ca DEPLOY_FAILED 2023-11-29 08:36:44 DeploymentRunner: INFO FileUtil:44 - Max number of files is 0, number of files found is 0 2023-11-29 08:36:45 ajp-nio-8009-exec-9: INFO LogService:171 - 2023-11-29 08:36:45 172.1.96.10 - plaza-ftd01.bpjsketenagakerjaan.go.id 443 GET /api/fdm/latest/jobs/deployments/3a085978-8e8e-11ee-9e41-a3dfcb7d6393 - 200 - 0 78 https://172.1.10.10 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 https://172.1.10.10/ 2023-11-29 08:36:46 ajp-nio-8009-exec-2: INFO LogService:171 - 2023-11-29 08:36:46 172.1.96.10 - plaza-ftd01.bpjsketenagakerjaan.go.id 443 GET /api/fdm/latest/devices/default/operational/ha/status/default - 200 - 0 68 https://172.1.10.10 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 https://172.1.10.10/ 2023-11-29 08:36:47 DeploymentRunner: INFO BaseDeploymentRepositoryImpl:248 - Marking objects for deployment aborted. 2023-11-29 08:36:50 ajp-nio-8009-exec-3: INFO LogService:171 - 2023-11-29 08:36:50 172.1.96.10 - plaza-ftd01.bpjsketenagakerjaan.go.id 443 GET /api/fdm/latest/jobs/deployments/3a085978-8e8e-11ee-9e41-a3dfcb7d6393 - 200 - 0 6 https://172.1.10.10 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 https://172.1.10.10/ 2023-11-29 08:36:50 DeploymentRunner: INFO DeploymentRunner:151 - Posting onto eventbus to indicate the current status of the deployment 2023-11-29 08:36:50 DeploymentRunner: INFO DeploymentRunner:159 - Posting event: DEPLOYMENT_FAIL onto event bus 2023-11-29 08:36:50 DeploymentRunner: INFO DeploymentRunner:325 - Sending notification of end of deployment:ngfw.deploy.fail 2023-11-29 08:36:50 eventbus-service-4: ERROR UpdateCachesHandler:65 - com.cisco.sensor.eventing.cacheupdate.UpdateCachesHandler 2023-11-29 08:36:50 ngfw-job-scheduler-worker-1: INFO NgfwJob:201 - DeploymentImmediateJob status update: Deployment Task: 'User (admin) Triggered Deployment' Failed after 30m 9.673s - IN_PROGRESS 2023-11-29 08:36:50 DeploymentRunner: INFO DeploymentRunner:169 - Notifying CloudEnrollmentProcessor about the event DEPLOYMENT_FAIL 2023-11-29 08:36:50 DeploymentRunner: INFO DeploymentRunner:169 - Notifying SlaMonitorOperationalStateCacheHandler$$EnhancerBySpringCGLIB$$538f9098 about the event DEPLOYMENT_FAIL 2023-11-29 08:36:50 DeploymentRunner: INFO DeploymentRunner:169 - Notifying DiskFileObjectProcessor about the event DEPLOYMENT_FAIL
Cisco Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
