LCLS CMLOG Restart Notes


These notes describe how to restart the CMLOG system processes and the ERR processes.

CMLOG System Processes

The CMLOG processes on lcls-daemon2 are the following:

  1. cmlogServer
  2. cmlogClientD
  3. iocLogAndFwdServer
  4. fwdCliS

Although not a part of the CMLOG system, the ALH and Channel Watcher processes running on lcls-daemon2 must be restarted after the cmlogClientD process is restarted in order for the ALH and Channel Watcher processes to be able to log messages.

Single Button Press Method of Restarting CMLOG System and Related Processes

The usual method of restarting CMLOG system and related (i.e., ALH and Channel Watcher) processes is through a single button press on the Controls UNIX Watchdog GUI. This button press invokes the following script on lcls-daemon2 (where the CMLOG system and related processes run) under the "laci" account:

This script performs the following actions:

  1. Stops the fwdCliS process.
  2. Stops the iocLogAndFwdServer process.
  3. Stops the cmlogClientD process.
  4. Stops the cmlogServer process.
  5. Starts the cmlogServer process.
  6. Starts the cmlogClientD process.
  7. Starts the iocLogAndFwdServer process.
  8. Starts the fwdCliS process.
  9. Stops and then restarts the ALH process
  10. Determines the Channel Watcher type for each running Channel Watcher process. After the type for a Channel Watcher processes has been determined, the associated Channel Watcher process is stopped and then restarted.

Note that this script does not need to be modified when new Channel Watcher instances are added or removed. The script determines which Channel Watcher processes are currently running (from information obtained through a UNIX "ps" command) and restarts them using the associated system startup commands in the lcls-daemon2 /etc/init.d directory.

It is advisable to request permission from the current EOIC before restarting the CMLOG system and related processes. However, experience has shown (confirmed by feedback from EOICs) that restarting the CMLOG system and related processes using the Controls UNIX Watchdog GUI does not adversely affect Operations.

The following steps can be followed to invoke the Controls UNIX Watchdog GUI and to find the button to restart the CMLOG system and related processes:

  1. Logon to a machine with an account name that will allow you access to activate Controls UNIX Watchdog GUI actions. For example:
  2. To start the Controls UNIX Watchdog GUI:
  3. Select the "Controls_UNIX_Watchdog" button appearing in the popup window.
  4. On new popup window that appears (the main Controls UNIX Watchdog window), first select the arrow button to the right of the "Controls_Servers" button.
  5. Scroll down the window until finding the "lcls-daemon2" button. Select the arrow button to the right the "lcls-daemon2" button.
  6. Scroll down the window until finding the "PROC" button underneath the "lcls-daemon2" button. Select this "PROC" button.
  7. On the right pane of the window, locate the "CMLOG SYSTEM RESTART" button. Select the "P" button to the right of "CMLOG SYSTEM RESTART".

uwd -> Controls_UNIX_Watchdog -> Controls_Servers right arrow -> lcls-daemon2 right arrow -> PROC ->CMLOG SYSTEM RESTART "P" button


The actions resulting from restarting the CMLOG system and related processes take approximately three minutes to complete. This is due to the large number of processes that are stopped and then restarted (with waits of 5 seconds after a process is restarted). Most of these processes are Channel Watcher processes.

Manual CMLOG Stop and Restart Procedure

As noted above, the CMLOG system processes (with the ALH and Channel Watcher processes) are usually restarted by a single button push on the Unix Watchdog GUI. This procedure describes a manual procedure for restarting these processes.


  1. ssh lcls-prod02
  2. ssh lcls-daemon2 -l laci
  3. cd /etc/init.d

Stop the CMLOG processes

  1. ./st.fwdCliS stop
  2. ./st.ioclogandfwdserver stop
  3. ./st.cmlogClientD stop
  4. ./st.cmlog stop

Start the CMLOG processes

  1. ./st.cmlog start
  2. ./st.cmlogClientD start
  3. ./st.ioclogandfwdserver start
  4. ./st.fwdCliS start

Restart the ALH process and the Channel Watcher processes:

As noted above, if the cmlogClientD process is restarted, the ALH process and the Channel Watcher processes must be restarted in order for them to log messages.

To stop and restart the ALH process:

There is a separate Channel Watcher system startup file for each Channel Watcher process. Channel Watcher system startup files have names of the form:

where "XX" is a two letter identification for a Channel Watcher process.

ERR Logging Processes Restart

The Err logging processes, errClient and oocCosEventService, run on mccas0 and are required to log Aida messages. They may need to be restarted if Aida messages are not being logged. There are two methods of restarting these processes: (1) using system startup files on mccas0, and (2) using the errmanager script.

Occasionally a problem occurs with the cmlogClientD process on mccas0, which is started by errClient process. If Aida messages are not being logged after restarting the Err logging process, kill the cmlogClientD process(es) on mccas0 and then restart the Err logging processes again.

Restart Using System Startup Files

  1. Logon to the mccas0 machine using the "cddev" account.
  2. cd /etc/init.d
  3. ./st.err restart
  4. ./st.ooc_CosEventService restart

Restart Using errmanager

Perform the following (e.g., on a machine such as flora or tersk):

  1. /afs/slac/g/cd/soft/prod/solaris/bin/errmanager all restart
  2. To verify that the processes were restarted:

Author:  Bob Hall 04-Jun-2008
Revision:  Bob Hall 03-Jan-2008 Added errmanager information.
Revision:  Bob Hall 27-Mar-2012 Updated information including starting CMLOG with button press.

Modified: Jingchen Zhou 01-June-2012