Customer Service Center

When the DUT port is issued with a shutdown command, our link (Spirent TestCenter port side) goes in RED as expected and reports Local Link fault, but, the DUT sees a flood of messages or state changes as follows:

Remote Fault Set -> Alarm Local Fault Set -> Alarm Remote Cleared -> Alarm Local Fault Cleared -> Alarm Remote Fault Set and continuous in that loop. Spirent TestCenter application log is not showing any of there state changes

 

Snippet of DUT console logs:
 

Description : Default System Log
Memory Log contents  [size=500   next event=2130  (wrapped)]
 
2129 2019/10/21 14:37:02.213 EDT MINOR: PORT #2018 Base Port 1/2/1
"Alarm Local Fault Cleared"
 
2128 2019/10/21 14:37:02.213 EDT MINOR: PORT #2017 Base Port 1/2/1
"Alarm Remote Fault Set"
 
2127 2019/10/21 14:37:02.203 EDT MINOR: PORT #2018 Base Port 1/2/1
"Alarm No Frame Lock Cleared"
 
2126 2019/10/21 14:37:02.193 EDT MINOR: PORT #2017 Base Port 1/2/1
"Alarm No Frame Lock Set"
 
2125 2019/10/21 14:37:02.193 EDT MINOR: PORT #2017 Base Port 1/2/1
"Alarm Local Fault Set"
 
2124 2019/10/21 14:37:02.193 EDT MINOR: PORT #2018 Base Port 1/2/1
"Alarm Remote Fault Cleared"
 
2123 2019/10/21 14:37:00.353 EDT MINOR: PORT #2018 Base Port 1/2/1
"Alarm No Frame Lock Cleared"
 
2122 2019/10/21 14:37:00.353 EDT MINOR: PORT #2018 Base Port 1/2/1
"Alarm Local Fault Cleared"
 

This  behavior was expected, and the reason is as follows:

The requirement is that our FW shall attempt to recover link whenever a Local Fault (LF) is detected. To be more precise, 10G PX3 FW executes the following resets at most once every second while LF is detected in an effort to recover the link:
 

Set GT_RX_RESET_100G & GT_RESET_ALL_100G bits.
Set PMA_RESET & PCS_RESET bits.
Issue random sleep time, anywhere from 10ms up to 1 second.
Clear PMA_RESET & PCS_RESET bits.
Clear GT_RX_RESET_100G & GT_RESET_ALL_100G bits.
 

Link recovery has been a central part of our link handling/stability strategy, this functionality is required to maintain link stability between Spirent ports and provides a more reliable link for us internally and also for customers across the board. While the DUT port is "shutdown", the 10G PX3 port attempts link recovery and temporarily causes the link to come up for a brief moment, but since the DUT port is commanding a "shutdown", the link immediately breaks again, and so you will see this periodic behavior (link up to link down) about once every second or two.  This behavior appears to be what's causing the flood of LF/RF on the DUT side.

NOTE: Spirent TestCenter 5.10 Release (May 21st 2020) included a fix for this particular card PX3-100GQ-T2 and10G speed (ONLY) where TX reset has been disabled, this helped the DUT to don't see the messages comming from us.

CR-01410646 
Subject: 4.99 - PX3-10G mode flooding the DUT with local/remote fault alarm messages after shutdown interface on DUT side.
Date Time Opened: 10/22/2019
Date Time Closed: 5/22/2020
Fix: 5.10 Release (May 21st)

Workaround
 

Configure ‘ignore link status’, we added new Rx link recovery logic to ‘ignore link status’ (CIPCD-15617) so the DUT won’t get impacted with Alarm Messages and the link will come up once the DUT side is up again. This should handle the alarm case as well as links not coming up, however, be aware Spirent TestCenter ports will still transmit due to the ‘Ignore Link Status’ configuration. This shouldn’t cause any Alarms on DUT side other than dropped frames.