I experienced a strange issue after making some configuration changes to a Juniper MX104 device with a MS-MIC-16G installed. After removing some NAT rules the services interface that the NAT service was attached to started doing a large amount of traffic. The same change was made on a second router and it had the same results. The devices were active/standby, and I confirmed that no traffic was going in/out of the standby router, the traffic was all internal to the MS-MIC-16G. The CPU usage of the card was also showing at a constant 96% when it usually is around 15%.
The services interface on the router was installed into FPC 1 PIC 0 and named
ms-1/0/0. I thought I could request the chassis take the pic offline but after trying that it does not seem to support the command:
root@router> request chassis pic offline fpc-slot 1 pic-slot 0 FPC slot 1 does not support this command
I didn’t want to reboot the entire router to fix this issue as it was only localized just to the MS-MIC-16G. Juniper supplies a guide on how to get a shell on the MS-MPC and MS-MIC devices which was useful, I followed the guide and got the shell and had a poke around. I couldn’t find anything useful in the logs so I entered the debug CLI (
mspdbg-cli -ps). From there the command
testcrash can be sent which will force a crash on the device causing it to reboot. After a few minutes I verified the device came back online and everything was working as expected.
The output from my session looked like this:
root@router> show tnp addresses Name TNPaddr MAC address IF MTU E H R master 0x1 02:00:00:00:00:04 em0 1500 0 0 3 re0 0x4 02:00:00:00:00:04 em0 1500 0 0 3 afeb0 0x32 02:00:00:00:00:10 em0 1500 4 0 3 fpc1.pic0 0x111 02:00:10:00:01:11 em0 1500 2 0 3 bcast 0xffffffff ff:ff:ff:ff:ff:ff em0 1500 0 0 3 root@router> start shell user root Password: root@border2% telnet -Ji fpc1.pic0 Trying 126.96.36.199... Connected to core1-bucharest.web365.ro. Escape character is '^]'. login: root --- JUNOS 18.1R1.9 built 2018-03-23 23:36:46 UTC root@ms10% mspdbg-cli -ps MSPMAND-CLI> clear clear command hierarchy debug setup debug info in the system lib MSPMAND Library Information plugin plugin information quit quit from this cli request request services in the system set set information in the system show show information about the system test enable test framework testcrash command to crash the pic and reboot MSPMAND-CLI> testcrash <carriage return> Completes command MSPMAND-CLI> testcrash
The below would do the magic (Please confirm your mic and fpc slot before going ahead)
request chassis mic offline mic-slot 0 fpc-slot 1
request chassis mic online mic-slot 0 fpc-slot 1
Nice tip, thank you for that.