...
This article explains about the list of outputs that need to be collected in the cases where memory leaking is suspected.
Memory leak in most cases is a very slow process and extensive log analysis is required on such cases. We may require multiple samples of the outputs for our analysis and to understand which part of the memory is affected. Memory monitoring tool A memory monitoring tool was incorporated into the S4810, MXL and Z9000 9.1(0.0P6+) releases. This tool captures detailed data allocation usage valuable in troubleshooting potential memory leaks. The memory monitoring tool can be accessed through a console or telnet session. 1: Enable Memory Monitoring Tool To enable the memory monitoring tool. Usage e.g.: remote-exec cp f10mc -i 0x87 Example: Command: Z9000#remote-exec cp f10mc -i 0x87 System Response: f10mc -i 0x87 f10mc: Initialized for malloc, pool, pool_cache, with filter for F10 allocations SStk-0 # 2: Collecting Data 2a: Collecting Periodic General Memory Data Collect the data periodically (4 to 6 hour intervals) each day for several days. To collect the output, execute the following command. The following command displays general memory details of the system. The cycle number in the output will increment by one each time the output is collected. Usage e.g.: remote-exec cp f10mc -r 0 Example: Command: Z9000#remote-exec cp f10mc –r 0 System Response: f10mc -r 0 { ---------------------- #### This cycle: 46176 bytes held by 56 allocations #### M: 1888(16); +2329280(18371), -2329280(18371) P: 0(0); +0(0), -0(0) PC: 44288(40); +3053056(11226), -3053056(11226) #### Since MCInit: 708902 bytes held by 2112 allocations #### M: 52006(1338); +110122134(928732), -110070128(927394) P: 0(0); +0(0), -0(0) PC: 656896(774); +148369408(548411), -147712512(547637) } SStk-0 # The following output may be requested in addition to the memory monitoring tool output. Related Commands: remote-exec cp top -b -o res all remote-exec cp netstat –m remote-exec cp vmstat –s show process memory show process memory management show memory show clock 2b: Collecting Detailed Memory Data After the collection period for the "remote-exec cp f10mc –r 0" command has completed, the output of the following command will be requested. The following command will provide precise memory details of the system. Usage e.g.: remote-exec cp f10mc -r 1 Example: Command: Z9000#remote-exec cp f10mc -r 1 System Response: f10mc -r 1 { PC:mbpl:0xC71F5200:256:3 PC:mclpl:0xD7C24800:2048:3 PC:mbpl:0xC71F3500:256:3 {output truncated} PC:mbpl:0xCAFC7F00:256:2 PC:mclpl:0xD7C4D000:2048:2 ---------------------- #### This cycle: 286144 bytes held by 266 allocations #### M: 3520(16); +21362(417), -20514(415) P: 0(0); +0(0), -0(0) PC: 282624(250); +1129472(2200), -1126912(2197) #### Since MCInit: 599392 bytes held by 542 allocations #### M: 5216(20); +67838(1335), -62622(1315) P: 0(0); +0(0), -0(0) PC: 594176(522); +3513088(6660), -2918912(6138) } SStk-0 # Engineering will evaluate the collected data and provide details on how to collect further information that will assist in identifying source for memory leak. Output requested will be for suspect memory allocations: Usage e.g.: remote-exec cp f10mc –c 0x Example: Command: Z9000# remote-exec cp f10mc –c 0xCE0DDD00 System Response: f10mc -c 0xCE0DDD00 PC:mbpl:0xCE0DDD00:256:2> 0xC0100EAD 0xC033DA54 0xC0137ADE 0xC0136DD0 0xC0142682 0xC013FA17 0xC0148C38 0xC037C808 0xC037BE3F 0xC09B805A call trace: 0xC0100EAD in Xsoftintr 0xC033DA54 in softint_dispatch 0xC0137ADE in ipintr 0xC0136DD0 in ip_input 0xC0142682 in tcp_input 0xC013FA17 in syn_cache_get 0xC0148C38 in tcp_template 0xC037C808 in m_gethdr 0xC037BE3F in m_get 0xC09B805A in f10MC_pool_cache_get_paddr SStk-0 # 3:Disable Memory Monitoring To disable the memory monitoring tool after the data review is complete.. Usage e.g.: remote-exec cp f10mc -i 0x0 Example: Command: Z9000#remote-exec cp f10mc -i 0x0 System Response: f10mc -i 0x0 f10mc: deinitialized SStk-0 #