Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
Multicast traffic coming from the VXLAN fabric from east-west received from the VPC peer-link is encapsulated back into the fabric Link usage on the fabric links between Leaf--Spine--Leaf increased
Using N36180YC-R and VPC running 10.4.1 ( also tested in 10.4.2) Using VXLAN fabric with TRM ( TRM is not involved on the path)
Following config must be added: hardware access-list tcam region mac-ifacl 0 Documentation BUG was opened to include this in the config guide for VXLAN on N3600
Having a VXLAN fabric using fretta devices with 10.4.1 ( also tested on 10.4.2) # show mod Mod Ports Module-Type Model Status --- ----- ------------------------------------------------ --------------------- -------- 1 54 48x25G + 6x100G Ethernet Module N3K-C36180YC-R active * Mod Sw Hw Slot --- ----------------------- ------ ---- 1 10.4(1) 1.0 NA topology PC: MAC:0050.5684.3384 | IPv6: fe80::ce26:e7b4:59b8:de4f---L2SW---LeafVPC3&4------SPINE---------LeafVPC1&2-------L2SW NVE info: VPC on left side: Leaf-3 c77.6d4d.a147 Leaf-4 70df.2feb.919b secondary IP:10.10.2.103 Leaf-3# show nve int nve1 det Interface: nve1, State: Up, encapsulation: VXLAN VPC Capability: VPC-VIP-Only [notified] Local Router MAC: 4c77.6d4d.a147 Host Learning Mode: Control-Plane Source-Interface: loopback1 (primary: 10.10.1.103, secondary: 10.10.2.103) Source Interface State: Up Virtual RMAC Advertisement: Yes NVE Flags: Interface Handle: 0x49000001 Source Interface hold-down-time: 180 Source Interface hold-up-time: 30 Remaining hold-down time: 0 seconds Virtual Router MAC: 0200.0ac5.0267 Interface state: nve-intf-add-complete Fabric convergence time: 135 seconds Fabric convergence time left: 0 seconds Leaf-4(config-router)# show nve int nve1 det Interface: nve1, State: Up, encapsulation: VXLAN VPC Capability: VPC-VIP-Only [notified] Local Router MAC: 70df.2feb.919b Host Learning Mode: Control-Plane Source-Interface: loopback1 (primary: 10.10.1.104, secondary: 10.10.2.103) Source Interface State: Up Virtual RMAC Advertisement: Yes NVE Flags: Interface Handle: 0x49000001 Source Interface hold-down-time: 180 Source Interface hold-up-time: 30 Remaining hold-down time: 0 seconds Virtual Router MAC: 0200.0ac5.0267 Interface state: nve-intf-add-complete Fabric convergence time: 135 seconds Fabric convergence time left: 0 seconds VPC on right side: Leaf-1 7070.8bd0.519b Leaf-2 7070.8bd2.0503 secondary IP:10.10.2.101 Leaf-1# show nve int nve1 det Interface: nve1, State: Up, encapsulation: VXLAN VPC Capability: VPC-VIP-Only [notified] Local Router MAC: 7070.8bd0.519b Host Learning Mode: Control-Plane Source-Interface: loopback1 (primary: 10.10.1.101, secondary: 10.10.2.101) Source Interface State: Up Virtual RMAC Advertisement: Yes NVE Flags: Interface Handle: 0x49000001 Source Interface hold-down-time: 180 Source Interface hold-up-time: 30 Remaining hold-down time: 0 seconds Virtual Router MAC: 0200.0ac5.0265 Interface state: nve-intf-add-complete Fabric convergence time: 135 seconds Fabric convergence time left: 0 seconds Leaf-2# show nve int nve1 det Interface: nve1, State: Up, encapsulation: VXLAN VPC Capability: VPC-VIP-Only [notified] Local Router MAC: 7070.8bd2.0503 Host Learning Mode: Control-Plane Source-Interface: loopback1 (primary: 10.10.1.102, secondary: 10.10.2.101) Source Interface State: Up Virtual RMAC Advertisement: Yes NVE Flags: Interface Handle: 0x49000001 Source Interface hold-down-time: 180 Source Interface hold-up-time: 30 Remaining hold-down time: 0 seconds Virtual Router MAC: 0200.0ac5.0265 Interface state: nve-intf-add-complete Fabric convergence time: 135 seconds Fabric convergence time left: 0 seconds Traffic sent from PC on the left using MAC:0050.5684.3384 destined to ipv6 Mcast fe80::ce26:e7b4:59b8:de4f in vlan 50 is looped back into the fabric from Leaf on the right hand side Leaf1&2 to the SPINE. Using mDNS for testing since customer is experiencing a loop on the network PC 0050.5684.3384 generates the MDNS traffic and it is hashed to Leaf3 on its VPC leg in vlan 50 mapped to the VNI 10050 vlan 50 vn-segment 10050 member vni 10050 suppress-arp mcast-group 239.1.2.3 Leaf-3 encap the traffic on VXLAN using VNI 10050 with destination IP 239.1.2.3 and it sends to the SPINE Spine replicates the traffic to both VPC devices on the right hand side Leaf1&2 Leaf-2 is the VPC primary and the prf-source forwarder, Leaf-1 but Leaf-2 should NOT be forwarding the traffic back into the fabric Leaf-2(config-router)# show ip pim internal vpc rpf-source PIM vPC RPF-Source Cache for Context "default" - Chassis Role Primary Source: 10.10.2.101 Pref/Metric: 0/0 Ref count: 0 In MRIB: yes Source role: primary Forwarding state: Win-force (forwarding) MRIB Forwarding state: forwarding Source: 10.10.2.103 Pref/Metric: 115/8005 Ref count: 1 In MRIB: yes Source role: primary Forwarding state: Tie (forwarding) MRIB Forwarding state: forwarding <<<< Leaf-2 is the forwarder Source: 10.10.255.1 Pref/Metric: 115/8001 Ref count: 0 In MRIB: yes Source role: primary Forwarding state: Tie (forwarding) MRIB Forwarding state: forwarding Leaf-1 should not be sending the traffic back into the fabric Leaf-1# show vpc role vPC Role status ---------------------------------------------------- vPC role : secondary Leaf-1# show ip pim internal vpc rpf-source PIM vPC RPF-Source Cache for Context "default" - Chassis Role Secondary Source: 10.10.2.101 Pref/Metric: 0/0 Ref count: 0 In MRIB: yes Source role: secondary Forwarding state: Win-force (forwarding) MRIB Forwarding state: forwarding Source: 10.10.2.103 <<<< source belongs to Leaf3&4 Pref/Metric: 115/8005 Ref count: 1 In MRIB: yes Source role: secondary Forwarding state: Tie (not forwarding) MRIB Forwarding state: not forwarding <<< should not be forwarding Source: 10.10.255.1 Pref/Metric: 115/8001 Ref count: 0 In MRIB: yes Source role: secondary Forwarding state: Tie (not forwarding) MRIB Forwarding state: not forwarding Taking an ERSPAN capture on the SPINE RX leg facing 3600-1 we can see that it is sending the traffic back into the fabric Generic Routing Encapsulation (ERSPAN) Flags and Version: 0x1000 0... .... .... .... = Checksum Bit: No .0.. .... .... .... = Routing Bit: No ..0. .... .... .... = Key Bit: No ...1 .... .... .... = Sequence Number Bit: Yes .... 0... .... .... = Strict Source Route Bit: No .... .000 .... .... = Recursion control: 0 .... .... 0000 0... = Flags (Reserved): 0 .... .... .... .000 = Version: GRE (0) Protocol Type: ERSPAN (0x88be) Sequence Number: 215468 Encapsulated Remote Switch Packet ANalysis Type II 0001 .... .... .... = Version: Type II (1) .... 0000 0000 0000 = Vlan: 0 000. .... .... .... = COS: 0 ...0 0... .... .... = Encap: Originally without VLAN tag (0) .... .0.. .... .... = Truncated: Not truncated (0) .... ..00 0000 0011 = SpanID: 3 0000 0000 0000 .... .... .... .... .... = Reserved: 0 .... .... .... 0000 0000 0001 0000 1000 = Index: 264 Ethernet II, Src: 70:70:8b:d0:51:9b, Dst: 01:00:5e:7f:00:32 <<<<<<<<<<<<<using the LocalRMAC from Leaf-1 Destination: 01:00:5e:7f:00:32 Address: 01:00:5e:7f:00:32 .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast) Source: 70:70:8b:d0:51:9b Address: 70:70:8b:d0:51:9b .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Type: IPv4 (0x0800) Internet Protocol Version 4, Src: 10.10.2.101, Dst: 239.1.2.3 <<<<<<<<<<<<< Using the Secondary IP from the VPC 0100 .... = Version: 4 .... 0101 = Header Length: 20 bytes (5) Differentiated Services Field: 0x07 (DSCP: LE, ECN: CE) 0000 01.. = Differentiated Services Codepoint: Lower Effort (1) .... ..11 = Explicit Congestion Notification: Congestion Experienced (3) Total Length: 247 Identification: 0x0000 (0) Flags: 0x0000 0... .... .... .... = Reserved bit: Not set .0.. .... .... .... = Don't fragment: Not set ..0. .... .... .... = More fragments: Not set Fragment offset: 0 Time to live: 255 Protocol: UDP (17) Header checksum: 0xbd93 [validation disabled] [Header checksum status: Unverified] Source: 10.10.2.101 Destination: 239.1.2.3 <<<<<<<<<<<<<<< muticast group for L2VNI User Datagram Protocol, Src Port: 20514, Dst Port: 4789 Source Port: 20514 Destination Port: 4789 Length: 227 [Checksum: [missing]] [Checksum Status: Not present] [Stream index: 5] [Timestamps] [Time since first frame: 0.000000000 seconds] [Time since previous frame: 0.000000000 seconds] Virtual eXtensible Local Area Network Flags: 0x0800, VXLAN Network ID (VNI) 0... .... .... .... = GBP Extension: Not defined .... .... .0.. .... = Don't Learn: False .... 1... .... .... = VXLAN Network ID (VNI): True .... .... .... 0... = Policy Applied: False .000 .000 0.00 .000 = Reserved(R): 0x0000 Group Policy ID: 0 VXLAN Network Identifier (VNI): 10050 <<<<<<<<<<<<<<<<<<<L2VNI for vlan50 Reserved: 0 Ethernet II, Src: 00:50:56:84:33:84, Dst: 33:33:00:00:00:fb <<<<< MAC for source PC connected on left side Destination: 33:33:00:00:00:fb Address: 33:33:00:00:00:fb .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default) .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast) Source: 00:50:56:84:33:84 Address: 00:50:56:84:33:84 .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Type: IPv6 (0x86dd) Internet Protocol Version 6, Src: fe80::ce26:e7b4:59b8:de4f, Dst: ff02::fb <<<<<< source IPv6 addr for PC 0110 .... = Version: 6 .... 0000 0000 .... .... .... .... .... = Traffic Class: 0x00 (DSCP: CS0, ECN: Not-ECT) .... 0000 00.. .... .... .... .... .... = Differentiated Services Codepoint: Default (0) .... .... ..00 .... .... .... .... .... = Explicit Congestion Notification: Not ECN-Capable Transport (0) .... .... .... 0110 1100 1011 0110 0000 = Flow Label: 0x6cb60 Payload Length: 157 Next Header: UDP (17) Hop Limit: 255 Source: fe80::ce26:e7b4:59b8:de4f Destination: ff02::fb User Datagram Protocol, Src Port: 5353, Dst Port: 5353 Source Port: 5353 Destination Port: 5353 Length: 157 Checksum: 0x9258 [unverified] [Checksum Status: Unverified] [Stream index: 6] [Timestamps] [Time since first frame: 0.000000000 seconds] [Time since previous frame: 0.000000000 seconds] Multicast Domain Name System (response) Transaction ID: 0x0000 As seen above the packet that came over the VPC from SPINE---Leaf2----vpc----Leaf1 is encapsulated back into the fabric using the VIP and Local RMAC from Leaf1