Printer FriendlyEmail Article Link

Spirent TesteCenter: Slave chassis went out of sync in a chain

Symptoms
  • Two N11U chassis and one master (3 in total), one slave with Ether cable connected, it worked well with synced, but after some time, slave chassis showing out of sync, and MCU controller status LED showing RED .
  • After reboot slave chassis, we see the same issue in the beginning, after insert/pull out the ether cable multiple times, it comes into stable and synced well.
Environment
  • N11U chassis
  • Sync Master
  • Slave move
  • Chassis in chain
Environment:
  • 1 N11U chassis - Master
  • 2 N11U slave chassis

Issue hasn't been reproduced in house
 
Explanation/Resolution
  • Rebooting the slave chassis made the issue gone and issue hasn't seen anymore
  • Rebooting a chassis chain that might have lost sync has a best practice reboot process.
  • Attached is a script which would help avoid the manual step of sequentially rebooting the chained chassis one by one after an upgrade.
  • It would be useful to use this script which make this reboot operation less error-prone. We are looking at how best to improve this and make it more intelligent but the initial version should be helpful as well.
  • The script behavior will check the chassis sync status. If any of the chassis is OutofSync, it starts rebooting in the order​
    1. Master
    2. Direct_Slave
    3. Indirect_Slave’s
  • Script just reboots all the chassis in the sequential order even though only one chassis is OutofSync. So it doesn’t check whether the user is using or not. 
    • This script helps our support team to avoid the manual step of sequentially rebooting the chained chassis one by one after an upgrade.
    • So it’s assumed that we would get an open window for upgrading the chassis and so no user should be using the test setup at that time
  • Then it will check for Test modules in all the chassis and check the status. Again if any of the test modules are down/OutofSync it will reboot it.
    • Reboot wait time: 5mins (300secs)
    • Reboot Retry count: 3
  • How to run the script:
    1. Launch cmd prompt
    2. User can run the script in two ways
      • Method_1: only Master Chassis HostName/IPaddress
        • tclsh rebootChassisChainUpTopology.tcl Master_Chassis
        • ex: tclsh rebootChassisChainUpTopology.tcl 10.109.41.65
      • Method_2: Complete list of chassis in the chain setup
        • tclsh rebootChassisChainUpTopology.tcl Master_Chassis Slave_1 SLAVE_2 .... Slave_N
        • ex: tclsh rebootChassisChainUpTopology.tcl 10.109.41.65 10.109.41.65 10.109.42.213 10.109.42.235 10.109.42.219
        •  
Attachments
Attachment Description
Chassis Chain Reboot script

Product : Spirent TestCenter,Chassis,Chassis