-- AndreaFormica - 26-Jul-2012

Instructions for restarting alignment servers

In the following we describe a set of instructions to restart alignment servers in case it is needed.

Alignment server hosts

• voatlas129 : Barrel Alignment (Asap) and related services
• voatlas157 : Endcap Alignment server, COOL upload schedulers
• voatlas153 : B field monitoring and calibration servers

Instructions

1. connect to voatlasxxx (129[barrel-align],157[ec-align],153[bfield-calibration]) with user jasap
2. go to /data/jasap
3. verify the status of the process
4. stop the server or kill the process(es)
5. restart the server

How to verify process status

In order to verify the process status you can make use of the unix command ps.

       ps aux | egrep jboss 

If nothing comes out, the server is off. If something comes out but you do not see the system working (you cannot access any web page, aligngui does not work, the log file[see below] does not show recent messages), this means that you should kill the process and restart the server. Here an example of an "active" process output:

       [voatlas157] /data/jasap> ps aux | egrep jboss jasap 

       17798 0.0 0.0 63816 1112 pts/1 S 11:19 0:00 /bin/sh /usr/jboss-5/bin/run.sh -b voatlas157.cern.ch -Djava.awt.headless=true  
         -Dhep.aida.IAnalysisFactory=hep.aida.ref.BatchAnalysisFactory -Dasap.path= -Dcorbaref.home=/data/jasap/corba 
         -Dasap.startSchedulers=false -Dcorba.server.isExtern=false -Dorg.jboss.util.propertyeditor.DateEditor.format=yyyy/MM/dd-HH:mm:ss:z  
         -Dalign.static.web.dir=file:/data/jasap/web/ -Djboss.server.base.dir=/data/jasap/jboss -Djboss.server.base.url=file:///data/jasap/jboss  
         -Dbmon.startCalibScheduler=false Dbmon.startSchedulers=false -Dasap.startAlignCherryPyScheduler=true  
         -Dasap.startDAQSummaryScheduler=true -DFelix.LibName=Cfelix -Dalign.login.home=/afs/cern.ch/user/j/jasap -Dlogin.method=jaas -c jbalign jasap  

      17823 7.4 46.2 3342952 1873848 pts/1 Sl 11:19 2:51 java -Dprogram.name=run.sh -server -Xms2048m -Xmx2048m -XX:MaxPermSize=512m 
          -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -XX:+UseParallelGC 
          -XX:NewRatio=2 -Djava.net.preferIPv4Stack=true -Djava.endorsed.dirs=/usr/jboss-5/lib/endorsed -classpath /usr/jboss-5/bin/run.jar 
          org.jboss.Main -b voatlas157.cern.ch -Djava.awt.headless=true -Dhep.aida.IAnalysisFactory=hep.aida.ref.BatchAnalysisFactory 
          -Dasap.path= -Dcorbaref.home=/data/jasap/corba -Dasap.startSchedulers=false -Dcorba.server.isExtern=false 
          -Dorg.jboss.util.propertyeditor.DateEditor.format=yyyy/MM/dd-HH:mm:ss:z -Dalign.static.web.dir=file:/data/jasap/web/ 
          -Djboss.server.base.dir=/data/jasap/jboss -Djboss.server.base.url=file:///data/jasap/jboss -Dbmon.startCalibScheduler=false  
          -Dbmon.startSchedulers=false -Dasap.startAlignCherryPyScheduler=true -Dasap.startDAQSummaryScheduler=true 
          -DFelix.LibName=Cfelix -Dalign.login.home=/afs/cern.ch/user/j/jasap -Dlogin.method=jaas -c jbalign jasap 

      22361 0.0 0.0 61152 708 pts/1 S+ 11:57 0:00 egrep jboss 

As you see you have 2 process, you should be sure to kill the 2 of them.

How to stop the server or kill the process

1. HOW to KILL a process: try to switch off the server in a clean way: ./bin/asap.sh -q or by using directly the kill command: kill -9 PID (where PID stands for Process ID, which you can retrieve from the previous instructions). After killing a process please copy the log file in order for experts to be able to see what went wrong.
2. WHERE to check LOG FILES : Log files are stored in /data/jasap/log There is plenty of "old" logs, plus the "active" log, called jboss.log. Copy the log into another name for later checks. cp log/jboss.log log/jboss.log.BAD
3. RESTART the application server: ./bin/asap.sh then check the file log/jboss.log, after one minute or so it should (at the end) contain something like

11:20:47,062 INFO [SecurityInterceptor] SecurityInterceptor performs authentication... 
11:20:47,062 INFO [SecurityInterceptor] >>> Leaving SecurityInterceptor 
11:20:47,114 INFO [Http11Protocol] Starting Coyote HTTP/1.1 on http-voatlas157.cern.ch%2F128.142.129.158-8080 
11:20:47,135 INFO [AjpProtocol] Starting Coyote AJP/1.3 on ajp-voatlas157.cern.ch%2F128.142.129.158-8009 
11:20:47,161 INFO [Http11Protocol] Starting Coyote HTTP/1.1 on http-voatlas157.cern.ch%2F128.142.129.158-8443 
11:20:47,170 INFO [ServerImpl] JBoss (Microcontainer) [5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)] Started in 1m:16s:869ms