Symptoms

Making normal zoneset activations in a vsan that crosses FCIP links with enhanced mode zoning enabled .Symptom 1. zoneset activation fails to complete zone distribution as follows on both fcip peers in the same vsan crossing the fcip link. They point to each others domain id under show zone status::switch A1 show zone statusVSAN: 1 default-zone: deny distribute: full Interop: default mode: enhanced merge-control: allow session: cli [Admin] hard-zoning: enabled broadcast: unsupported smart-zoning: disabled rscn-format: fabric-address activation overwrite control: disabledDefault zone: qos: none broadcast: unsupported ronly: unsupportedFull Zoning Database : DB size: 28916 bytes Zonesets:1 Zones:142 Aliases: 81 Attribute-groups: 1Active Zoning Database : DB size: 12644 bytes Name: zoneset_vsan1 Zonesets:1 Zones:142Current Total Zone DB Usage: 41560 / 2097152 bytes (1 % used)Pending (Session) DB size: Full DB Copy size: 28996 bytes Active DB Copy size: 12644 bytesSFC size: 41640 / 2097152 bytes (1 % used)Status: Commit failed [Error: SFC retry exceeded Dom 21]: at 12:14:59 EST Jan 22 2019show fcdomain domain-listVSAN 1Number of domains: 2Domain ID WWN--------- ----------------------- 0x50(80) 20:0b:00:de:fb:0d:12:f1 [Local] [Principal] 0x15(21) 20:0b:00:de:fb:56:7d:b1switch A2:VSAN: 1 default-zone: deny distribute: full Interop: default mode: enhanced merge-control: allow session: remote [dom: 80][ip: 10.1.24.1] hard-zoning: enabled broadcast: unsupported smart-zoning: disabled rscn-format: fabric-address activation overwrite control: disabledDefault zone: qos: none broadcast: unsupported ronly: unsupportedFull Zoning Database : DB size: 28916 bytes Zonesets:1 Zones:142 Aliases: 81 Attribute-groups: 1Active Zoning Database : DB size: 12644 bytes Name: active_vsan_11_zoneset Zonesets:1 Zones:142Current Total Zone DB Usage: 41560 / 2097152 bytes (1 % used)Pending (Session) DB size: Full DB Copy size: 0 bytes Active DB Copy size: 0 bytesSFC size: 0 / 2097152 bytes (0 % used)Status: Commit failed [Error: SFC retry exceeded Dom 80]: at 10:39:21 EST Jan 17 2019VSAN 1Number of domains: 2Domain ID WWN--------- ----------------------- 0x50(80) 20:0b:00:de:fb:0d:12:f1 [Principal] 0x15(21) 20:0b:00:de:fb:56:7d:b1 [Local] Symptom 2. The output of show zone internal change event-history vsan 1 show RETRY_TIMEOUT for SFC frames sent during the zoneset distribution following zone commit:Change Protocol Event Log For VSAN: 11 >>>>FSM has 50 logged transitions37) Transition at Wed Jan 23 14:02:35 2019 Prev State: [ACA Complete] Trig event: [SEND_RCA] Next State: [RCA Sent]38) Transition at Wed Jan 23 14:02:35 2019 Prev State: [RCA Sent] Trig event: [RCVD_ACC] (Dom:21) Next State: [RCA Sent]39) Transition at Wed Jan 23 14:02:35 2019 Prev State: [RCA Sent] Trig event: [RCVD_ALL_ACC] Next State: [Idle]40) Transition at Wed Jan 23 14:03:16 2019 Prev State: [Idle] Trig event: [SEND_ACA] Next State: [ACA Sent]41) Transition at Wed Jan 23 14:03:16 2019 Prev State: [ACA Sent] Trig event: [RCVD_ACC] (Dom:21) Next State: [ACA Sent]42) Transition at Wed Jan 23 14:03:16 2019 Prev State: [ACA Sent] Trig event: [RCVD_ALL_ACC] Next State: [ACA Complete]43) Transition at Wed Jan 23 14:07:58 2019 Prev State: [ACA Complete] Trig event: [SEND_SFC] (Distribute DB) Next State: [Tz Req Sent]44) Transition at Wed Jan 23 14:07:58 2019 Prev State: [Tz Req Sent] Trig event: [RCVD_ALL_ACC] Next State: [SFC Sent]45) Transition at Wed Jan 23 14:08:58 2019 Prev State: [SFC Sent] Trig event: [RETRY_TIMEOUT] Next State: [SFC Sent]46) Transition at Wed Jan 23 14:09:58 2019 Prev State: [SFC Sent] Trig event: [RETRY_TIMEOUT] Next State: [SFC Sent]47) Transition at Wed Jan 23 14:10:58 2019 Prev State: [SFC Sent] Trig event: [RETRY_TIMEOUT] Next State: [SFC Sent]48) Transition at Wed Jan 23 14:11:58 2019 Prev State: [SFC Sent] Trig event: [RETRY_TIMEOUT] Next State: [SFC Sent]49) Transition at Wed Jan 23 14:12:58 2019 Prev State: [SFC Sent] Trig event: [RETRY_TIMEOUT] Next State: [SFC Sent]50) Transition at Wed Jan 23 14:12:58 2019 Prev State: [SFC Sent] Trig event: [RETRY_EXCEED] Next State: [ACA Complete] Curr state: [ACA Complete]Symptom 3. show zone internal change log events show RETRY_TIMEOUT and RETRY_EXCEED messages at the event time:show zone internal change log events1) Event:E_DEBUG, length:94, at 1621 usecs after Wed Jan 23 14:12:59 2019 [1515146821] vsan 11: Change abort cap remote domain=21 cap1=0x00000001 cap2 = 0x00000000 vsan_id 112) Event:E_DEBUG, length:93, at 1549 usecs after Wed Jan 23 14:12:59 2019 [1515146821] vsan 11: Change abort cap local domain=80 cap1=0x00000001 cap2 = 0x00000000 vsan_id 113) Event:E_DEBUG, length:49, at 1478 usecs after Wed Jan 23 14:12:59 2019 [1515146821] vsan 11: Checking change abort cap vsan=114) Event:E_DEBUG, length:108, at 1118 usecs after Wed Jan 23 14:12:59 2019 [1515146821] vsan 11: (C)Rcvd event (RETRY_EXCEED) on vsan 11, current state = SFC Sent, next state = ACA Complete5) Event:E_DEBUG, length:105, at 909 usecs after Wed Jan 23 14:12:59 2019 [1515146821] vsan 11: (C)Rcvd event (RETRY_TIMEOUT) on vsan 11, current state = SFC Sent, next state = SFC Sent6) Event:E_DEBUG, length:105, at 990951 usecs after Wed Jan 23 14:11:58 2019 [1515146821] vsan 11: (C)Rcvd event (RETRY_TIMEOUT) on vsan 11, current state = SFC Sent, next state = SFC Sent7) Event:E_DEBUG, length:105, at 980952 usecs after Wed Jan 23 14:10:58 2019 [1515146821] vsan 11: (C)Rcvd event (RETRY_TIMEOUT) on vsan 11, current state = SFC Sent, next state = SFC Sent8) Event:E_DEBUG, length:105, at 970889 usecs after Wed Jan 23 14:09:58 2019 [1515146821] vsan 11: (C)Rcvd event (RETRY_TIMEOUT) on vsan 11, current state = SFC Sent, next state = SFC Sent9) Event:E_DEBUG, length:105, at 970948 usecs after Wed Jan 23 14:08:58 2019 [1515146821] vsan 11: (C)Rcvd event (RETRY_TIMEOUT) on vsan 11, current state = SFC Sent, next state = SFC Sent

Cause

Cisco bug id CSCvo11849 - MDS 9250i Cannot distribute zonesets over fcip linkSymptom:MDS 9250i with fcip zoning change commits do not complete.Output of the zoning change event history shows SFC Retry time out.Conditions:To be determinedWorkaround:To recover, reload the IPS module using the following command.ips reset module 1 port 1This is disruptive to the fcip link.Known Affected Releases:6.2(21)

Resolution

Workaround:To recover, reload the IPS module ( this is the ASIC for IPStorage ports configured with FCIP) using the following command.ips reset module 1 port 1Note: This is disruptive to the fcip links. Fix: Bug is presently under investigation. Permanent fix is yet to be determined.

Support Cases