OPERATIONAL DEFECT DATABASE
...
...
Wireless Access Points fail to connect to the Wireless LAN Controller. Symptom 1 (where the AP's certificate has expired): At the time of the join failure, the WLC's msglog may show messages similar to the following: Jul 10 16:13:52.443 spam_lrad.c:6164 LWAPP-3-PAYLOAD_ERR: Join request does not contain valid certificate in certificate payload - AP 00:11:22:33:44:55 Symptom 2 (where the WLC's manufacturing installed certificate has expired): Once the WLC's MIC expires, the currently joined AP CAPWAP sessions will remain established. However, once an AP needs to reestablish the CAPWAP connection, it will fail. The AP logger will show messages similar to the following: *Oct 29 18:01:56.107: %PKI-3-CERTIFICATE_INVALID_EXPIRED: Certificate chain validation has failed. The certificate (SN: 7E3446C40000000CBD95) has expired. Validity period ended on 14:38:08 UTC Oct 26 2021Peer certificate verification failed 001A *Oct 29 18:01:56.107: DTLS_CLIENT_ERROR: ../capwap/base_capwap/capwap/base_capwap_wtp_dtls.c:496 Certificate verified failed! *Oct 29 18:01:56.107: %DTLS-5-SEND_ALERT: Send FATAL : Bad certificate Alert to 192.168.10.10:5246 *Oct 29 18:01:56.107: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 192.168.10.10:5246 On the WLC side, you will only see a message like this: *osapiBsnTimer: Oct 29 11:05:04.571: #DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:2962 Failed to complete DTLS handshake with peer 192.168.202.8
For MIC expiration: this symptom will occur after 10 years of the device manufacturing date. For SSC expiration: this symptom will occur after Jan. 1, 2020 The oldest APs (1120, 1130, 1230, 1310 series) with MICs were manufactured in July 2005, so those APs will be unable to join AireOS controllers starting in July 2015. This problem also affects WLCs approximately 10 years after manufacturing date. To determine when the AP's MIC was created, run this command on the WLC to find the SN: (Cisco Controller) >show ap inventory all Inventory for lap1130-sw3-9 NAME: "Cisco AP" , DESCR: "Cisco Wireless Access Point" PID: AIR-LAP1131AG-E-K9, VID: V01, SN: FCZ1128Q0PE NAME: "Dot11Radio0" , DESCR: "802.11G Radio" PID: UNKNOWN, VID: , SN: GAM112706LC NAME: "Dot11Radio1" , DESCR: "802.11A Radio" PID: UNKNOWN, VID: , SN: ALP112706LC The AP chassis SN is in the first section of the output, for example: PID: AIR-LAP1131AG-E-K9, VID: V01, SN: FCZ1128Q0PE The serial number format is: "LLLYYWWSSSS"; where "YY" is the year of manufacture and "WW" is the week of manufacture. The date code can be found in the 4 middle digits of the serial number. Manufacturing Year Codes: 01 = 1997 06 = 2002 11 = 2007 16 = 2012 02 = 1998 07 = 2003 12 = 2008 17 = 2013 03 = 1999 08 = 2004 13 = 2009 18 = 2014 04 = 2000 09 = 2005 14 = 2010 05 = 2001 10 = 2006 15 = 2011 Manufacturing Week Codes: 1-5 : January 15-18 : April 28-31 : July 41-44 : October 6-9 : February 19-22 : May 32-35 : August 45-48 : November 10-14 : March 23-27 : June 36-40 : September 49-52 : December Example: SN FCZ1128Q0PE has year code 11, meaning it was manufactured in 2007. The week code is 12, meaning it was manufactured in March. The SN can also be found using Prime Infrastructure Reporting to find SNs for all of the APs.
Workaround 1: Generate and use Locally Significant Certificates for authentication between the affected WLC(s) and APs. Workaround 2: Configure the WLC/APs to ignore MIC expiration. Code with fix is available on CCO for 7.0, 7.4, 8.x For 7.6, you may contact TAC for escalation code, although it is recommended to move to 8.0 for future support Recovery for APs in a failed scenario: NOTE: this workaround should be used only in order to allow APs with expired certificates to join the WLC for long enough to upgrade the software. If the certificates have expired, disable NTP, then change the WLC clock time to a recent earlier time when the certificates were still valid. If you set the clock back too far, newer APs may not be able to join. Once the software has been upgraded, and the affected APs have joined, the WLC clock should be reset to the valid time. Solution: Cisco has released AireOS 7.0.252.0, and will release rebuilds for 7.4 in April 2015 and 8.0 in June 2015. These rebuilds will implement a new CLI command to disable on the WLC the lifetime validity checks for MICs and SSCs. By default, the command will be disabled, i.e. APs with expired MICs and SSCs will not be able to join. After upgrading to the new rebuild, use the new command to disable the lifetime validity check, allowing APs with MICs or SSCs older than 10 years to join.
This bug CSCuq19142 does not fix the problem for certain 4400 series WLCs manufactured in 2005, whose MICs expire as early as March, 2015. For such WLCs, track the bug CSCuu02970. The command implemented via this bug CSCuq19142, to disable expiration checking, is "config ap lifetime-check {mic|ssc} enable". Because the effect of this command is logically opposite to the plain sense of this command, later AireOS releases implement a different command to disable expiration checking (config ap cert-expiry-ignore {mic|ssc} enable) For more information see: Field Notice 63942 - http://www.cisco.com/c/en/us/support/docs/field-notices/639/fn63942.html https://community.cisco.com/t5/wireless-mobility-documents/lightweight-ap-fail-to-create-capwap-lwapp-connection-due-to/ta-p/3155111 Important note: For 7.6 escalation code, this command is not sync to the peer in a High Availability pair, it may be needed to be configured in each of the controllers for HA while they are active The command syncs correctly in 7.4, and 8.x available CCO codes Important note: AP-COS APs (i.e. 802.11ac Wave 2 / 802.11ax APs such as 1800/2800/3800/9100 series) cannot ignore certificate expiration prior to 8.5. See CSCvb93909. Important note: IOS APs (i.e. 802.11n / 802.11ac Wave 1 APs), which were manufactured with SHA-2 certificates, cannot ignore WLC certificate expiration prior to 8.5.160.0. See CSCvs22835.
Cisco Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
