Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
EP learning for a Tunnel interface (going to remote TEP) fails and EP entries are removed shortly after being added : EPM EP MAC key: BD VNID 14680080 MAC: xxxx.xxxx.xxx epmc_delete_ep:4069: Deleting EP: epmc_debug_dump_epmc_ep:475: Log collect EP event : EP entry: : nMAC xxxx.xxxx.xxxx; numIPs 0 : VLAN 371; epg VNID 14680080; BD VNID 14680080; VRF VNID 2686977 : IF idx 0; Tunnel IF idx 0x18010003 : sclass 49155; ref_cnt 4 : flags MAC|sclass|timer| epmc_debug_dump_ep_key_err:310 [E]: EPM EP MAC key: BD VNID 14680080 MAC: xxxx.xxxx.xxx epmc_ep_force_del:4006 [E]: Sending force delete to EPM on ep failure : EPM EP MAC key: BD VNID 14680080 MAC: xxxx.xxxx.xxx epmc_delete_ep:4069: Deleting EP: epmc_debug_dump_epmc_ep:475: Log collect EP event : EP entry: : nMAC xxxx.xxxx.xxx; numIPs 0 : VLAN 371; epg VNID 14680080; BD VNID 14680080; VRF VNID 2686977 : IF idx 0; Tunnel IF idx 0x18010003 : sclass 49155; ref_cnt 4 : flags MAC|sclass|timer| Tunnel to remote TEP is in failed state (when checked in vsh prompt): Tunnel3 is down (Hardware prog failed) MTU 9000 bytes, BW 9 Kbit Transport protocol is in VRF "overlay-1" Tunnel protocol/transport ivxlan Tunnel source x.x.x.x, destination x.x.x.x Tx 0 packets output, 1 minute output rate 0 packets/sec Rx 0 packets input, 1 minute input rate 0 packets/sec
unknown
Clean reload of Leaf switch.
Tunnel programming fails. ELTMC seems to be stuck at retrying tunnel bring-up : [2019 Aug 4 02:51:31.822932092:1398377:eltm_del_port_rt_primary_chunk:6896:t] key: 3, key_size: 4 [2019 Aug 4 02:51:31.822979085:1398378:eltm_objstore_delete:824:t] delChunk: 0 [2019 Aug 4 02:51:31.822981271:1398379:eltm_intf_delete:8482:t] LOG_COLLECT_ELTM_INTF; optype = DELETE; opdetail = ELTM_INTF_DELETE; ifname = Tunnel3; ifindex = 0x18010003; ret_code = 0; ret_desc = SUCCESS; [2019 Aug 4 02:51:31.823085731:1398380:eltm_send_and_add_out_resp:415:t] Inserted out_resp:req_rr_token 0x187e1664 req payload 0xde1e5e9c eltm_to_eltmc_rr_token 0x187e1668,req saddr 393,req daddr 0 [2019 Aug 4 02:51:31.823088503:1398381:eltm_send_and_add_out_resp:421:t] Increment ref_cnt for payload 0xde1e5e9c ref_cnt 1 [2019 Aug 4 02:51:31.824974073:1398382:eltm_handle_l3vm_if_context_change_ntfn:9267:E] Unable to find intf 0x18010003 [2019 Aug 4 02:51:31.827764550:1398383:eltm_match_out_resp:495:t] Match out response from ELTMC:req_rr_token 0x187e1664 req payload 0xde1e5e9c eltmc_resp_rr_token 0x187e1668 req saddr 393,req daddr 0,resp saddr 995,resp daddr 192 [2019 Aug 4 02:51:31.827772953:1398384:eltm_match_out_resp:586:t] Decrement ref_cnt for payload 0xde1e5e9c ref_cnt 0 [2019 Aug 4 02:51:31.827773913:1398385:eltm_match_out_resp:599:t] Dropping payload 0xde1e5e9c [2019 Aug 4 02:51:31.884235654:1398386:eltm_handle_tunnel_crt_vrfy_req:3217:t] Handling IF Create Request for Intf = Tunnel3, idx= 402718723, ept=33 [2019 Aug 4 02:51:31.884245004:1398387:eltm_intf_create:8115:t] LOG_COLLECT_ELTM_INTF; optype = CREATE; opdetail = ELTM_INTF_CREATE; ifname = Tunnel3; ifindex = 0x18010003; iod = 149; If we check MTS queues , we may see Tunnel MTS stuck: node-2003 (vsh)# show system internal mts buffers summary node sapno recv_q pers_q npers_q log_q sup 26411 0 0 1 0 sup 26403 0 0 1 0 sup 26385 0 0 1 0 sup 26364 0 0 1 0 sup 26227 0 0 1 0 sup 26168 0 0 1 0 sup 393 0 78976 243270 0 sup 196 0 0 1 0 sup 856 0 0 1 0