Printer FriendlyEmail Article Link

landslide: how to debug test server "not ready" issue and check the test server status

Environment/Versions
landslide test server on unbuntu 14, kernel 3.13 or 4.8.
this tool doesn't support old OS ubuntu 9 with kernel 2.x.
Also, since the auto start service of TS process on new OS ubuntu 16.04 has been changed, this tool doesn't support OS ubuntu 16.04, if you run it on OS 16.04 you'll see it stop at step3 failed checking auto start service.
Answer
From time to time, we see cases that test server status is not ready and end user had no idea why.

so we wrote a small tool here to automatically check the status of test server and try to tell what's wrong with  test  server.
note this tool doesn't support old OS ubuntu 9 with kernel 2.x.
to make test server registered with TAS and be ready for service, there are below multiple conditions need to meet:
1. TS should  be configured with correct TAS IP and other settings by  ipcfg .
2. the TS and TAS communication should be good, nothing in network/fireball blocks the UDP connection (from TS 889x port to TAS port 8889) and  ssh  connection.
3. TS IP should be configured on TAS.
4. TS and TAS better to be on  same  main release.
5. TS auto-start service is enabled in  linux  system and  ssesh  process was brought up.
6. TAS would send out license info to TS based on big packet fragmented into multiple small UDP packets(8889 to 889x), those fragmented packets should not be blocked by  netowrk  or firewall. we see cases that routers on  network  couldn't handle and re-assemble such dozens of fragmented packets and TS failed to get license info, so always not ready.
7. TS should not be recycled due to  any  unexpected error like nic card issue, or "failed to bind  socket  on configured  ip " issue .etc. If recycled, you would see "CoastAbort" info in test server log and the tool would also check that.

the attached tool would check those above conditions and try to tell you what's wrong with  test  server ,also  help you save all the logs if you choose yes.
to run this tool: 
Put it onto your test server under /home/ cfguser  directory by  ftp  or  scp  or whatever way you prefer to, then “ chmod  755  tsCheck ” to make it executable.
              Then run “./tsCheck” using  cfguser  account.
below is the sample of running this tool:
>>>>
##>./tsCheck
2019-02-05 20:50:32,234 - INFO - starting ts Checking now......
2019-02-05 20:50:33,837 - INFO - currently under cfguser account
2019-02-05 20:50:36,240 - INFO - changed into root account on this TS.
2019-02-05 20:50:36,240 - INFO -
###############################::step1 :chechking ts Config......
 
2019-02-05 20:50:37,241 - INFO - got tsIP successfully,it's:10.71.41.51
 
2019-02-05 20:50:37,742 - INFO - got tasIP successfully,it's:10.71.41.50
 
2019-02-05 20:50:37,742 - INFO - success:ts Config is good!
 
2019-02-05 20:50:37,742 - INFO -
###############################::step 2: checking ping from TS to TAS......
 
2019-02-05 20:50:40,274 - INFO - success :ping  connection between TS and TAS is good! TAS is reachable
 
2019-02-05 20:50:40,274 - INFO -
###############################::step 3.1: checking sshd service on TS......
 
2019-02-05 20:50:40,775 - INFO - success: sshd  service is running well on TS!
 
2019-02-05 20:50:40,775 - INFO -
###############################:step3.2: checking TS auto-start service......
 
2019-02-05 20:50:41,276 - INFO - success: TS auto start service is enabled and good.
 
2019-02-05 20:50:41,276 - INFO -
###############################:step 3.3: checking available disk space......
 
2019-02-05 20:50:41,776 - INFO - your disk size is 396G and you have 308G free space left.
2019-02-05 20:50:41,776 - INFO - you have used 19% of your disk space.
2019-02-05 20:50:41,777 - INFO -
###############################::step 4.1: checking ssh connection from TS to TAS......
 
2019-02-05 20:50:42,878 - INFO - trying to login to TAS now......
2019-02-05 20:50:43,379 - INFO - getting TAS password challenge,starting logging using cfguser default password...
2019-02-05 20:50:44,080 - INFO - logged into TAS by cfguser successfully,checking root account now...
2019-02-05 20:50:47,584 - INFO - login into TAS successfully.
2019-02-05 20:50:48,085 - INFO - success: ssh connection from TS to TAS is good.
2019-02-05 20:50:48,085 - INFO -
###############################:step 4.2: checking ssh connection from TAS to TS......
 
2019-02-05 20:50:49,286 - INFO - success:  ssh connection from TAS to TS is good!
 
2019-02-05 20:50:49,286 - INFO -
###############################:step 4.3: check fragmented big packet from tas to ts,which is usually used by TAS to send info to TS......
 
2019-02-05 20:50:51,792 - INFO - success :fragmented  big packet sent from TAS to TS successfully!
 
2019-02-05 20:50:51,792 - INFO -
###############################:step 5.00:checking whether tas service is started and running......
 
2019-02-05 20:50:52,543 - INFO - success: tas is started and in service!
 
2019-02-05 20:50:52,544 - INFO -
###############################:step 5.01:checking whether mysql database service is running on TAS......
 
2019-02-05 20:50:53,044 - INFO - success:  mysql  database is started and in service!
 
2019-02-05 20:50:53,044 - INFO -
###############################:step 5.02:checking whether ts is configured on TAS......
 
2019-02-05 20:50:54,045 - INFO - success: ts is configured on TAS!
 
2019-02-05 20:50:54,045 - INFO -
###############################:step 5.1: verify UDP 8889 connection between TAS-TS by checking TAS log...
 
2019-02-05 20:50:54,551 - INFO - found last udp status report from TS in TAS log, it happened at TAS local time:2019-02-05 20:45:31
2019-02-05 20:50:55,052 - INFO - the current TAS time is :2019-02-05 20:45:33
2019-02-05 20:50:55,053 - INFO - success: the TAS log shows received udp status report from TS less than 5 minutes ago
2019-02-05 20:50:55,053 - INFO -
###############################:step 5.2: verify UDP 8889 connection between TAS-TS by capturing packets on TAS...
 
2019-02-05 20:50:59,156 - INFO - success: found UDP packet sent by TS by capturing,UDP connection is good.
2019-02-05 20:50:59,156 - INFO -
###############################:step 5.3: checking TS/TAS/test suite version......
 
2019-02-05 20:51:01,159 - INFO - TS version is 17.4.1.3
2019-02-05 20:51:01,160 - INFO - TAS version is 17.5.1.2
2019-02-05 20:51:01,160 - WARNING - warning: ts and tas are on the main release but different branch,you might see mismatched measurement and some other discrepency when running test case! better to sync them
2019-02-05 20:51:01,160 - INFO - Test suite  version is 17.4.1.3
2019-02-05 20:51:01,160 - INFO - success: ts and test suite version are the same !
2019-02-05 20:51:01,160 - INFO -
###############################:step 6: checking TS process ssesh status.....
 
2019-02-05 20:51:02,863 - INFO - Last known Test Server StartTime is at UTC time:2019-02-04 22:19:39.104042
2019-02-05 20:51:03,364 - INFO - Test server sent out registration to TAS at UTC time:2019-02-04 22:19:42.760690
2019-02-05 20:51:03,865 - INFO - TAS authenticated this TS successfully at UTC time:2019-02-04 22:19:42.827990
2019-02-05 20:51:04,365 - INFO - server Initialized successfully at UTC time:2019-02-04 22:20:16.582691
2019-02-05 20:51:04,866 - INFO - the last status report sent by TS happened at UTC time:2019-02-06 02:49:12.409005
2019-02-05 20:51:05,367 - INFO - the number of running ssesh process is:4
2019-02-05 20:51:05,868 - INFO - success:the status of TS is ready and good now!
2019-02-05 20:51:05,868 - INFO -
###############################:step 7: checking whether to save current tsCheck logs and all other test server logs/syslogs.
 
do you want to save  current   tsCheck  log and other test server logs and compress it(Yes/No) ?yes
2019-02-05 20:51:13,484 - INFO - starting to save all logs found on this test server!
2019-02-05 20:51:13,785 - INFO - compressing logs now......please wait.....
2019-02-05 20:51:15,987 - INFO - compressing logs now......please wait.....
2019-02-05 20:51:17,989 - INFO - compressing logs now......please wait.....
2019-02-05 20:51:48,025 - INFO - compressing logs now......please wait.....
2019-02-05 20:51:49,883 - INFO - ls_auto_saved_log-2019-02-05-20-51-13.484657.tar.gz is successfully generated under /home/ cfguser
##>



 
Attachments
Attachment

Product : TAS Manager,Landslide Client