Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
Deployment fails with 2023-11-29 08:36:50 ngfw-job-scheduler-worker-1: INFO NgfwJob:201 - DeploymentImmediateJob status update: Deployment Task: 'User (admin) Triggered Deployment' Failed after 30m 9.673s - IN_PROGRESS
If the configuration has object groups in access to 20k hosts and the customer tried to modify many such rules. 2023-11-29 00:31:44 deploy-task-worker-2: INFO AccessPolicyPreDeltaResolver:91 - scanNegatedRules - Starts 2023-11-29 01:08:53 deploy-task-worker-2: INFO AccessPolicyPreDeltaResolver:145 - scanNegatedRules - Ends. numNegatedBlockRules is 13, isAnySrcNetwork is false, isAnyDestNetwork is true info=[task=[running=[RUNNING ON deploy-task-worker-2] >> This function was running beyond 30 min causing the deployment failure.
N/A
2023-11-29 08:36:44 DeploymentRunner: ERROR DeploymentController:182 - Deployment Error: {}Waited 1800000 milliseconds for com.google.common.util.concurrent.CollectionFuture$ListFuture@477030db[status=PENDING, info=[futures=[[com.google.common.util.concurrent.TrustedListenableFutureTask@55974026[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]], com.google.common.util.concurrent.TrustedListenableFutureTask@4003aee0[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]], com.google.common.util.concurrent.TrustedListenableFutureTask@1b156899[status=PENDING, info=[task=[running=[RUNNING ON deploy-task-worker-2], com.cisco.ngfw.onbox.deploy.tasks.LinaCliExportTask@7181b8a3]]], com.google.common.util.concurrent.TrustedListenableFutureTask@5da575c8[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]]]]]] 2023-11-29 08:36:44 DeploymentRunner: ERROR NormalDeploymentController:169 - Failed to complete deployment: java.util.concurrent.TimeoutException: Waited 1800000 milliseconds for com.google.common.util.concurrent.CollectionFuture$ListFuture@477030db[status=PENDING, info=[futures=[[com.google.common.util.concurrent.TrustedListenableFutureTask@55974026[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]], com.google.common.util.concurrent.TrustedListenableFutureTask@4003aee0[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]], com.google.common.util.concurrent.TrustedListenableFutureTask@1b156899[status=PENDING, info=[task=[running=[RUNNING ON deploy-task-worker-2], com.cisco.ngfw.onbox.deploy.tasks.LinaCliExportTask@7181b8a3]]], com.google.common.util.concurrent.TrustedListenableFutureTask@5da575c8[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]]]]]] 2023-11-29 08:36:44 DeploymentRunner: INFO NormalDeploymentController:175 - Posted event to revert back the event rate after deploy through NGFWEventBus 2023-11-29 08:36:44 DeploymentRunner: ERROR DeploymentRunner:230 - DeploymentException occurred while executing deployment request : 39fdf936-8e8e-11ee-9e41-f1f51281e2ca DeploymentException:java.util.concurrent.TimeoutException: Waited 1800000 milliseconds for com.google.common.util.concurrent.CollectionFuture$ListFuture@477030db[status=PENDING, info=[futures=[[com.google.common.util.concurrent.TrustedListenableFutureTask@55974026[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]], com.google.common.util.concurrent.TrustedListenableFutureTask@4003aee0[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]], com.google.common.util.concurrent.TrustedListenableFutureTask@1b156899[status=PENDING, info=[task=[running=[RUNNING ON deploy-task-worker-2], com.cisco.ngfw.onbox.deploy.tasks.LinaCliExportTask@7181b8a3]]], com.google.common.util.concurrent.TrustedListenableFutureTask@5da575c8[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]]]]]] 2023-11-29 08:36:44 DeploymentRunner: ERROR DeploymentRunner:231 - DeploymentException:java.util.concurrent.TimeoutException: Waited 1800000 milliseconds for com.google.common.util.concurrent.CollectionFuture$ListFuture@477030db[status=PENDING, info=[futures=[[com.google.common.util.concurrent.TrustedListenableFutureTask@55974026[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]], com.google.common.util.concurrent.TrustedListenableFutureTask@4003aee0[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]], com.google.common.util.concurrent.TrustedListenableFutureTask@1b156899[status=PENDING, info=[task=[running=[RUNNING ON deploy-task-worker-2], com.cisco.ngfw.onbox.deploy.tasks.LinaCliExportTask@7181b8a3]]], com.google.common.util.concurrent.TrustedListenableFutureTask@5da575c8[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]]]]]] at com.cisco.ngfw.onbox.deploy.controllers.DeploymentController.doInParallel(DeploymentController.java:183) at com.cisco.ngfw.onbox.deploy.controllers.DeploymentController.doInParallel(DeploymentController.java:146) at com.cisco.ngfw.onbox.deploy.controllers.NormalDeploymentController.executeTasks(NormalDeploymentController.java:193) at com.cisco.ngfw.onbox.deploy.controllers.NormalDeploymentController.execute(NormalDeploymentController.java:167) at com.cisco.ngfw.onbox.deploy.controllers.NormalDeploymentController$$FastClassBySpringCGLIB$$879e94a2.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) at com.cisco.ngfw.onbox.deploy.controllers.NormalDeploymentController$$EnhancerBySpringCGLIB$$2ec6309.execute() at com.cisco.ngfw.onbox.deploy.DeploymentRunner.executeDeploymentTask(DeploymentRunner.java:225) at com.cisco.ngfw.onbox.deploy.DeploymentRunner.run(DeploymentRunner.java:132) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.concurrent.TimeoutException: Waited 1800000 milliseconds for com.google.common.util.concurrent.CollectionFuture$ListFuture@477030db[status=PENDING, info=[futures=[[com.google.common.util.concurrent.TrustedListenableFutureTask@55974026[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]], com.google.common.util.concurrent.TrustedListenableFutureTask@4003aee0[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]], com.google.common.util.concurrent.TrustedListenableFutureTask@1b156899[status=PENDING, info=[task=[running=[RUNNING ON deploy-task-worker-2], com.cisco.ngfw.onbox.deploy.tasks.LinaCliExportTask@7181b8a3]]], com.google.common.util.concurrent.TrustedListenableFutureTask@5da575c8[status=SUCCESS, result=[DeploymentStatus(126371)/39fdf936-8e8e-11ee-9e41-f1f51281e2ca (DEPLOYING)]]]]]] at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:468) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:90) at com.cisco.ngfw.onbox.deploy.controllers.DeploymentController.doInParallel(DeploymentController.java:171) ... 14 more 2023-11-29 08:36:44 eventbus-service-2: INFO SensorEventManager:122 - com.cisco.sensor.event.engine.SensorEventManager 2023-11-29 08:36:44 eventbus-service-2: INFO SensorEventManager:125 - com.cisco.sensor.event.engine.SensorEventManager 2023-11-29 08:36:44 DeploymentRunner: INFO NormalDeploymentController:249 - Performing clean up of deployment with success = false 2023-11-29 08:36:44 DeploymentRunner: INFO BaseDeploymentRepositoryImpl:434 - Updating deploymentStatus:126371 DEPLOY_FAILED 2023-11-29 08:36:44 DeploymentRunner: INFO BaseDeploymentRepositoryImpl:454 - Updated DeploymentStatus:126371 39fdf936-8e8e-11ee-9e41-f1f51281e2ca DEPLOY_FAILED 2023-11-29 08:36:44 DeploymentRunner: INFO FileUtil:44 - Max number of files is 0, number of files found is 0 2023-11-29 08:36:45 ajp-nio-8009-exec-9: INFO LogService:171 - 2023-11-29 08:36:45 172.1.96.10 - plaza-ftd01.bpjsketenagakerjaan.go.id 443 GET /api/fdm/latest/jobs/deployments/3a085978-8e8e-11ee-9e41-a3dfcb7d6393 - 200 - 0 78 https://172.1.10.10 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 https://172.1.10.10/ 2023-11-29 08:36:46 ajp-nio-8009-exec-2: INFO LogService:171 - 2023-11-29 08:36:46 172.1.96.10 - plaza-ftd01.bpjsketenagakerjaan.go.id 443 GET /api/fdm/latest/devices/default/operational/ha/status/default - 200 - 0 68 https://172.1.10.10 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 https://172.1.10.10/ 2023-11-29 08:36:47 DeploymentRunner: INFO BaseDeploymentRepositoryImpl:248 - Marking objects for deployment aborted. 2023-11-29 08:36:50 ajp-nio-8009-exec-3: INFO LogService:171 - 2023-11-29 08:36:50 172.1.96.10 - plaza-ftd01.bpjsketenagakerjaan.go.id 443 GET /api/fdm/latest/jobs/deployments/3a085978-8e8e-11ee-9e41-a3dfcb7d6393 - 200 - 0 6 https://172.1.10.10 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 https://172.1.10.10/ 2023-11-29 08:36:50 DeploymentRunner: INFO DeploymentRunner:151 - Posting onto eventbus to indicate the current status of the deployment 2023-11-29 08:36:50 DeploymentRunner: INFO DeploymentRunner:159 - Posting event: DEPLOYMENT_FAIL onto event bus 2023-11-29 08:36:50 DeploymentRunner: INFO DeploymentRunner:325 - Sending notification of end of deployment:ngfw.deploy.fail 2023-11-29 08:36:50 eventbus-service-4: ERROR UpdateCachesHandler:65 - com.cisco.sensor.eventing.cacheupdate.UpdateCachesHandler 2023-11-29 08:36:50 ngfw-job-scheduler-worker-1: INFO NgfwJob:201 - DeploymentImmediateJob status update: Deployment Task: 'User (admin) Triggered Deployment' Failed after 30m 9.673s - IN_PROGRESS 2023-11-29 08:36:50 DeploymentRunner: INFO DeploymentRunner:169 - Notifying CloudEnrollmentProcessor about the event DEPLOYMENT_FAIL 2023-11-29 08:36:50 DeploymentRunner: INFO DeploymentRunner:169 - Notifying SlaMonitorOperationalStateCacheHandler$$EnhancerBySpringCGLIB$$538f9098 about the event DEPLOYMENT_FAIL 2023-11-29 08:36:50 DeploymentRunner: INFO DeploymentRunner:169 - Notifying DiskFileObjectProcessor about the event DEPLOYMENT_FAIL