LM - the log manager

TaoNSDumper / TaoNSLookup / OprLMInterrupter / TaoNSUnbinder

How it works

The LM has 2 modes of operation:

take events and distribute these 1 event at a time via CORBA to the OprDaemonApp running on that node.
acknowledge that an even has been processed. There are two types of events here:
- Those that fail a digi filter and are not written to Objectivity.
- good events that are written to Objectivity.

To get data first, the DRS loads the xtc file onto disk. This is then read, event by event, by the LM and then transported by a TCP/IP connection to the OprDaemonApp on client nodes. Here events are written into a shared memory space. Theis is shared with regards the ELF app running on the client node.

Start up

The LM is started from the OprManager/runOprProcesses script. You can track the start up by looking for the runlog* file in a Run directory and looking at the output files that are associated with the LM:

lm.log: this is the log file for the LM itself and shows some preliminary start up info and other 'meaningless' messages [wrt pr-ops] about what the lm is doing. This contains some information that may be useful if the run fails to start properly because of expired tokens etc.
lmm.log: this file is more important than lm.log. The logging manager creates a log file called lmm.log in each run directory. To continuously monitor this file do a tail -200f lmm.log once you are in the run directory of the currently processing run. This log file updates every 60s and shows the connected broncos, how many events each bronco has processed, and how many events have been processed in total for this run. You can monitor this file to see how far along the run is and to look for stuck broncos.
Note: At the end of a run the broncos will finish and drop out of the list with a message:
"Socket BRONCOxxx.SLAC.Stanford.EDU crashed/died since last query"
This just means that they are done. Although a crashing bronco will do this in the middle of the run. To check that a bronco is finished, look in the done directory under the run directory of the currently processing run.
runlog* tells you what runOprProcesses thinks has happened when the run is starting up with diagnostic information in the event that a lm has 'Failed to start even after 120 seconds'.

Normally the LM should start up without any problems. However if the LM fails to start for a run or anything else strange happens the order in which you should search the files [assuming that the exist] is runlog*, lm.log, lmm.log paying note to only the first few lines of lm.log unless you know what this is doing.

Looking for and killing

The utilities that we have to look for and kill running Log Managers are:

where here the [corbaRefName] is the CORBA reference and looks like 'Opr/LM/run#-cb#' for Opr and 'Rep/run#-cb#/LM' for any of the REP farms. The [lookup-corbaRefName] is something like 'Opr LM/run#-cb#' or 'Rep LM/run#-cb#'. For example, if you have a run and you want to check to see if a LM is running for run # 14697 you use:

TaoNSLookup Rep LM/14697-1

The return of a long string like this tells you that there is a LM running. To kill it you would use:

OprLMInterrupter Rep/LM/14697-1

NOTE: the Rep LM/14697-1 -> Rep/LM/14697-1 change. You can check to see if the LM is removed by repeating the above command:

TaoNSLookup Rep LM/14697-1

The other inspection utility that you have to use is TaoNSDumper:

Each entry marked as context corresponds to a LM having started for a given run. The line with 'reference' indicates that there is either a LM still running for that particular run, or that the LM has been killed in a nasty way ...

If the machine running the LM crashes or one does a kill -9, the reference for a name in the NamingService is not removed. If this is the case you will not be able to start a new LM for that run because you need to remove the name from the naming service. To do this you can use TaoNSUnbinder.

Possible problems with running

For the moment you'll need to see the oprshifter page for troubleshooting beyond these words.

If a node crashes, events that are not signaled as done with by the client are then redistributed to other client nodes [the killer events are never redistributed to stop wiping out the farm with corrupt events]. For each client that a LM connects to it has a thread to that cient, plus a few others.

If the LM is 'starving', this means that the memory is filled up and that most events wil be pending. The sum of current + pending events is the number stored in memory at any given time. If you see that this is happening it is a sign that you've got a problem that needs fixing.

Back to Opr Operations Home

Maintained by Adrian Bevan.

Last updated on 13-Aug-2001