Symptom
ASR1001-X was crashed giving a cpp-ucode core file, whose internal decode revealed it crashed due to an interrupt called: SOR_CSR32_SOR_ERR_LEAF_INT__INT_SOR_OPF_GRANT_PTCL_UVF with Polaris 16.6.07 image
Conditions
Based upon the logs and Microcode core file, DMVPN Session moved its session QOS Hierarchy while there is traffic in on the Queue. Exact trigger for transformation of QOS hierarchy is unknown
Workaround
Requesting the customer to move 16.11 or above. This does not guarantee the fix, but if the issue happens again, we would have better infra/serviceability to understand the sequence of events.
Further Problem Description
SOR channel shows that some packets are in flight. There are no credits mismatch between SOR and OPF but in that time, some transformations in hierarchy is going on. Hardware Batch ops cannot be traced in 16.6.07, hence root cause of crash is not known.
