Symptom

Unconfiguring VRF leak related configs on the leaf nodes causes urib and ufdm proc to crash. The crash occurs unconfiguring the VRFs, evpn and BGP on the leaf nodes(with spine in pass thru): The topo involves 2 nodes as Leaf(tor1,tor3), tor2 -> Border Leaf and node03 is pass thru spine. The configs resident on each of the devices just before this is saved under: CONFIGS:

Conditions

Need to be shutting down a vrf and need leaked route in this vrf.

Workaround

Further Problem Description

The crash occurs unconfiguring the VRFs, evpn and BGP on the leaf nodes(with spine in pass thru): The topo involves 2 nodes as Leaf(tor1,tor3), tor2 -> Border Leaf and node03 is pass thru spine. The configs resident on each of the devices just before this is saved under: CONFIGS: sjc-ads-5816:/ws/balajiv-sjc/VRF_LEAK > ls node01_vrf_leak_p3.cfg node03_vrf_leak_p3.cfg node02_vrf_leak_p3.cfg spine_vrf_leak_profile2.cfg sjc-ads-5816:/ws/balajiv-sjc/VRF_LEAK > TRIGGERS: The sequence of triggers was when un configuring the protocols from the leaf nodes: Leaf 1: sanity72-tor1# conf Enter configuration commands, one per line. End with CNTL/Z. sanity72-tor1(config)# no evpn sanity72-tor1(config)# no router bgp 65000 Leaf 2: sanity72-tor3(config)# no vrf context VRFA sanity72-tor3(config)# no vrf context VRFB sanity72-tor3(config)# no vrf context VRFC sanity72-tor3(config)# no router bgp 65000 Leaf 3: sanity72-tor2(config)# no evpn sanity72-tor2(config)# no vrf context VRFA sanity72-tor2(config)# no vrf context VRFB sanity72-tor2(config)# no vrf context VRFC sanity72-tor2(config)# no router bgp 65000 CORES: I have the core files but am seeing an issue with the UCD stacktrace utility tool. Will work on getting that, meanwhile here is the location of the zipped cores. /ws/balajiv-sjc/VRF_LEAK >ls 1557374069_0x101_urib_log.1000.tar.gz node01_vrf_leak_p3.cfg 1557374258_0x101_ufdm_log.23583.tar.gz node02_vrf_leak_p3.cfg 1557374259_0x101_ufdm_log.23594.tar.gz node03_vrf_leak_p3.cfg cores spine_vrf_leak_profile2.cfg stacktrace: INFO: Given Core Name is '/tmp/ucd-ws.p94m4kfo/1557374069_0x101_urib_log.1000.tar.gz' INFO: Relocated Core Name is '/ramfs/ucd_tmp/ucd.g80iEVHdNA3Af9yNlF/1557374069_0x101_urib_log.1000.tar.gz' INFO: Core Type is 'NXOS' INFO: Found Build Type is 'final' INFO: Found Build Root is '/auto/n5kbld12/release//irvine/nexus/REL_9_3_0_227' INFO: Found Hardware Model is 'core_n9000' INFO: Found Release Version is '9.3(0.227)' INFO: VBE Version is '5.2.2' INFO: Application Name is '/auto/n5kbld12/release//irvine/nexus/REL_9_3_0_227/build/ins/x86e/final/urib/sup/urib' INFO: CPU Architecture is 'Intel' INFO: BLD_BRANCH: across Intel, PowerPC, MIPS & ARM CPU architectures Core type is :: NXOS /ramfs/ucd_tmp/ucd.g80iEVHdNA3Af9yNlF/.gdbinit:3: Error in sourced command file: bld type is :: /auto/n5kbld12/release//irvine/nexus/REL_9_3_0_227 /build: No such file or directory. [New LWP 1190] warning: .dynamic section for "/ramfs/ucd_tmp/ucd.g80iEVHdNA3Af9yNlF/usr/lib/libssl.so.1.0.0" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/auto/n5kbld12/release/irvine/nexus/REL_9_3_0_227/build/gdblinks/final_core_n9000/isan/lib/libdme_stub.so" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/ramfs/ucd_tmp/ucd.g80iEVHdNA3Af9yNlF/lib/libgcc_s.so.1" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/ramfs/ucd_tmp/ucd.g80iEVHdNA3Af9yNlF/usr/lib/libglib-2.0.so.0" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/ramfs/ucd_tmp/ucd.g80iEVHdNA3Af9yNlF/lib/libcrack.so.2" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/ramfs/ucd_tmp/ucd.g80iEVHdNA3Af9yNlF/usr/lib/libgpg-error.so.0" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/ramfs/ucd_tmp/ucd.g80iEVHdNA3Af9yNlF/lib/libcap.so.2" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/ramfs/ucd_tmp/ucd.g80iEVHdNA3Af9yNlF/lib/libncurses.so.5" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/ramfs/ucd_tmp/ucd.g80iEVHdNA3Af9yNlF/lib/libtinfo.so.5" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/ramfs/ucd_tmp/ucd.g80iEVHdNA3Af9yNlF/usr/lib/libpython2.7.so.1.0" is not at the expected address (wrong library or version mismatch?) [New LWP 1155] [New LWP 1207] [New LWP 1194] [New LWP 1000] [New LWP 1196] [New LWP 3420] [New LWP 1178] [New LWP 1177] [New LWP 1179] [New LWP 1195] [New LWP 1150] Core was generated by `/isan/bin/routing-sw/urib'. Program terminated with signal 6, Aborted. #0 0xf77bdbf0 in __kernel_vsyscall () #0 0xf77bdbf0 in __kernel_vsyscall () #1 0xf5fc0cd6 in ssignal () from /ramfs/ucd_tmp/ucd.g80iEVHdNA3Af9yNlF/lib/libc.so.6 #2 0xde399284 in ?? () #3 #4 0xf6cb42dc in pt_get_next_node () from /auto/n5kbld12/release/irvine/nexus/REL_9_3_0_227/build/gdblinks/final_core_n9000/isan/lib/librsw.so #5 0x10138117 in urib_qt_flush_all () #6 0x1007f129 in urib_process_delete_tib_work () at ../routing-sw/routing/urib/urib.c:2973 #7 0x1009e8ee in urib_main () at ../routing-sw/routing/urib/urib.c:11520 #8 0xf6ccbab7 in procket_pthread_rtn () from /auto/n5kbld12/release/irvine/nexus/REL_9_3_0_227/build/gdblinks/final_core_n9000/isan/lib/librsw.so #9 0xf698d2be in start_thread () from /ramfs/ucd_tmp/ucd.g80iEVHdNA3Af9yNlF/lib/libpthread.so.0 #10 0xf607de8e in fanotify_mark () at ../sysdeps/unix/syscall-template.S:84 #11 0x00000000 in ?? () ufdm: Core type is :: NXOS /ramfs/ucd_tmp/ucd.0SvZaGCfevYbb4Myxx/.gdbinit:3: Error in sourced command file: bld type is :: /auto/n5kbld12/release//irvine/nexus/REL_9_3_0_227 [New LWP 23583] /build: No such file or directory. warning: .dynamic section for "/ramfs/ucd_tmp/ucd.0SvZaGCfevYbb4Myxx/usr/lib/libssl.so.1.0.0" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/ramfs/ucd_tmp/ucd.0SvZaGCfevYbb4Myxx/lib/libgcc_s.so.1" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/ramfs/ucd_tmp/ucd.0SvZaGCfevYbb4Myxx/usr/lib/libglib-2.0.so.0" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/auto/n5kbld12/release/irvine/nexus/REL_9_3_0_227/build/gdblinks/final_core_n9000/isan/lib/libdme_stub.so" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/ramfs/ucd_tmp/ucd.0SvZaGCfevYbb4Myxx/lib/libcrack.so.2" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/ramfs/ucd_tmp/ucd.0SvZaGCfevYbb4Myxx/usr/lib/libpython2.7.so.1.0" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/ramfs/ucd_tmp/ucd.0SvZaGCfevYbb4Myxx/usr/lib/libgpg-error.so.0" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/ramfs/ucd_tmp/ucd.0SvZaGCfevYbb4Myxx/lib/libcap.so.2" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/ramfs/ucd_tmp/ucd.0SvZaGCfevYbb4Myxx/lib/libncurses.so.5" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/ramfs/ucd_tmp/ucd.0SvZaGCfevYbb4Myxx/lib/libtinfo.so.5" is not at the expected address (wrong library or version mismatch?) Core was generated by `/isan/bin/ufdm'. Program terminated with signal 11, Segmentation fault. #0 0xf6b5b7c0 in urib_ufdm_send_post_routes () from /auto/n5kbld12/release/irvine/nexus/REL_9_3_0_227/build/gdblinks/final_core_n9000/isan/lib/liburib_ufdm.so #0 0xf6b5b7c0 in urib_ufdm_send_post_routes () from /auto/n5kbld12/release/irvine/nexus/REL_9_3_0_227/build/gdblinks/final_core_n9000/isan/lib/liburib_ufdm.so #1 0x100dfa01 in ufdm_fib_download_routes () #2 0x100e9709 in ufdm_fib_download_state () #3 0x100e9c37 in ufdm_fib_download_slot () #4 0x100eb814 in ufdm_fib_report () #5 0x10115e58 in ufdm_fu_fib_report () #6 0x100ffd85 in ufdm_mts_msg_handler () #7 0x10101ccc in ufdm_demux () #8 0xf694423d in fu_fsm_engine_process_app_ev () from /auto/n5kbld12/release/irvine/nexus/REL_9_3_0_227/build/gdblinks/final_core_n9000/isan/lib/libfsrv.so #9 0xf6945c6a in fu_fsm_engine () from /auto/n5kbld12/release/irvine/nexus/REL_9_3_0_227/build/gdblinks/final_core_n9000/isan/lib/libfsrv.so #10 0x10111fee in main ()