Symptom

Device will report traceback and/or crash due malloc. Exception to Fastpath Thread: Frame pointer 0xFF9A22A4B0, PC = 0xAAB70DEC38 -Traceback= 1#4e77356e181cb88f5db27cbef08e5fd4 :AAB3B65000+3579C38 :AAB3B65000+1428B98 :AAB3B65000+142C378 iosd_ngwc_unix:FFED32F000+28754 iosd_ngwc_unix:FFED32F000+24004 prelib:FFEF89E000+2CDC pthread:FFED0D7000+C0AC c:FFED148000+10E92C IOS Thread backtrace: UNIX-EXT-SIGNAL: User defined signal 2(17), Process = Sched -Traceback= 1#4e77356e181cb88f5db27cbef08e5fd4 pthread:FFED0D7000+9264 Auxiliary Thread backtrace: -Traceback= 1#4e77356e181cb88f5db27cbef08e5fd4 pthread:FFED0D7000+11174 iosd_ngwc_unix:FFED32F000+36784 prelib:FFEF89E000+2CDC pthread:FFED0D7000+C0AC c:FFED148000+10E92C $0 : 0x00000000, AT : 0x811DA3B0, v0 : 0x00000028, v1 : 0x00000008 a0 : 0xFF99CD2098, a1 : 0x00000020, a2 : 0xFFA9650168, a3 : 0xFFA02B9FA8 t0 : 0xFF9A22A500, t1 : 0x00000000, t2 : 0x000A0043, t3 : 0x00000000 t4 : 0x00000000, t5 : 0x80080018, t6 : 0x81192648, t7 : 0x00000000 s0 : 0xFF9FCDD020, s1 : 0x00000028, s2 : 0x00000000, s3 : 0xFF99CD2098 s4 : 0x00000000, s5 : 0xFF99CD2098, s6 : 0xFF9FCDD020, s7 : 0xAABCD93E60 t8 : 0x00000000, t9 : 0xAAB70D8AD8, k0 : 0x0571C99C, k1 : 0x00000000 gp : 0xAABD9218B8, sp : 0xFF9A22A4B0, s8 : 0xFF9A22A4B0, ra : 0xAAB70DEC24 EPC : 0xAAB70DEC38, SREG : 0x00000000 Cause : 0xFFFFFFFF Error EPC : 0x00000000, BADVADDR = 0x00000000 *Aug 31 10:18:57.229: %SYS-3-INVMEMINT: Invalid memory action (malloc) at interrupt level -Traceback= 1#4e77356e181cb88f5db27cbef08e5fd4 :AAB3B65000+3F4175C :AAB3B65000+3F407E4 :AAB3B65000+3F40B94 :AAB3B65000+5FC7438 :AAB3B65000+68E8790 :AAB3B65000+3EE6A20 :AAB3B65000+3EE7044 :AAB3B65000+3EE78D0 :AAB3B65000+3EE7E10 :AAB3B65000+145DC6C :AAB3B65000+1453B44 *Aug 31 10:18:57.231: %SYS-2-MALLOCFAIL: Memory allocation of 64 bytes failed from 0xAAB7A4BA18, alignment 0 Pool: Processor Free: 599790368 Cause: Interrupt level allocation Alternate Pool: None Free: 0 Cause: Interrupt level allocation -Process= "", ipl= 2 -Traceback= 1#4e77356e181cb88f5db27cbef08e5fd4 :AAB3B65000+3F4175C :AAB3B65000+3F407E4 :AAB3B65000+3F40B94 :AAB3B65000+5FBD5E4 :AAB3B65000+5FC7474 :AAB3B65000+68E8790 :AAB3B65000+3EE6A20 :AAB3B65000+3EE7044 :AAB3B65000+3EE78D0 :AAB3B65000+3EE7E10 :AAB3B65000+145DC6C *Aug 31 10:18:57.238: %SYS-3-INTPRINT: Illegal printing attempt from interrupt level. -Process= "", ipl= 2 -Traceback= 1#4e77356e181cb88f5db27cbef08e5fd4 :AAB3B65000+3F4175C :AAB3B65000+3F407E4 :AAB3B65000+3F40B94 :AAB3B65000+5EE0F20 :AAB3B65000+5EDEE6C :AAB3B65000+3EE6A38 :AAB3B65000+3EE7044 :AAB3B65000+3EE78D0 :AAB3B65000+3EE7E10 :AAB3B65000+145DC6C :AAB3B65000+1453B44 IOSD RP: 0 messages not written to btrace log fileAug 31 10:20:26.961 R0/0: %PMAN-3-PROCHOLDDOWN: The process linux_iosd-image has been helddown (rc 139) Aug 31 10:20:27.364 R0/0: %PMAN-0-PROCFAILCRIT: A critical process linux_iosd_image has failed (rc 139) Aug 31 10:20:27.720 R0/0: %PMAN-3-RELOAD_SYSTEM: Reloading: Peer switch is not standby ready. System will be reloaded

Conditions

Wireshark on highly utilised control-plane is enabled. For example: monitor capture CAP1 control-plane match any buffer size 1 monitor capture CAP1 start In the lab stream of around 1Gbps of ICMP is sent to SVI on the switch. Right after enabling the wireshark tracebacks and/or crash due malloc is occurring.

Workaround

Configure packet capture to only capture X packets per second. Example: monitor capture mycap limit pps 100

Further Problem Description