Some information about epics
Information about epics can also be found elsewhere:
Make sure you look at some instructions at the bottom
of this page.
To steer and monitor the DIRC we have
The two first hardware IOCs are located in rack #07 in the E.H. (bottom and
middle crates) whereas
drc-gas is on top
of the E.H. in rack #38 (where the N2 rotameters are).
- three 'hardware' (= real boards plugged in VME crates) IOCs:
- one 'software' IOC running on a linux box:
The epics production area is
/nfs/bbr-srv01/u1/babar/boot/apps, directory to which the environment
should be set. Each IOC uses its own package
(called epics-drc-mon/hv/gas/soft respectively).
If you look in this directory you should see a structure similar to the
example below (situation as of 2006/02/12):
lrwxrwxrwx 1 narnaud bfactory 17 Oct 4 13:23 drc-soft -> drc-soft-20051004
lrwxrwxrwx 1 thiebaux bfactory 15 Oct 20 13:42 drc-gas -> drc-gas-20Oct05
lrwxrwxrwx 1 wittgen bfactory 10 Oct 26 13:25 drc-hv -> drc-hv-osi
lrwxrwxrwx 1 narnaud bfactory 15 Jan 19 13:21 drc-mon -> drc-mon-01Nov05
Indeed, there are several versions of the IOC packages in this area. The ones
which are currently used in production are defined in symbolic links pointing
to the 'official' names expected by the BaBar epics system: for us,
Changing the version of the code running in an IOC is straightforward: one just
needs to change the symbolic link and to reboot the IOC. Yet, this procedure
should only be done by the DIRC epics experts (currently Christophe Thiebaux
and Nicolas Arnaud) or by ODC experts (Matthias Wittgen in particular).
The IOC logfiles for the current day and for the past week can be found in the
/nfs/bbr-nfs01/logfiles/Odc aerea. There is one directory per day which contains one file for each
Each IOC package contains several directories:
CVS adl bin configure db dl logs script stl vxWorks.sym
Makefile alh boot data dbd http perl src vxWorks
The main ones -- which will be briefly described below -- are found in all such
This directory contains the code source for all the IOC panels. When the
package gets compiled, binary files loaded when some panels need to be
displayed on a computer screen are stored in the
It contains ascii files which define the contents on the DIRC section in
the alarm handler (ALH). They are loaded each time the main ALH (on
the con01 IR2 pilot console) gets restarted or reloaded. The exact same
configuration appears on all the other alarm handler interfaces which
are started (in read-only mode) by users afterwards.
Hence, to have changes taken into account, the pilot should be asked to
restart the ALH or to reload its configuration.
Files with the label
should be used when some parts of the DIRC are off for a significant
period of time, e.g. during a shutdown. These files are very similar
to the ones which are used when the DIRC is on but most of the alarms
are masked to avoid filling the ALH with fake or 'normal' alarms.
The main DIRC ALH configuration file is
Inside are defined the files which are read by the pilot ALH. That's the file
to edit to replace one config. file by its shutdown version.
Alarms are stored in the
cmlog logfiles. Two
are available on the shifter consoles: the alarm handler logfile and the ORC
logfile. Both can be queried with command lines to get the list of alarms
which showed up in a given time range. Examples of scripts can be found
here. Information is also available in the
file OstErrLogger/README.cmlogQuery or in this page.
More information on the ALH can be found in the
The startup file read when the IOC is rebooted is located here. For an old
VxWorks IOC (drc-mon
and drc-gas), the
file is start.cmd
whereas it is st.sys
for a new RTEMS IOC
(drc-hv). This file
configures the board, load the epics variables defined in the
and starts the state machines.
To check that a modified
db/ file has no
syntax error, one can run the following command from the top directory:
It dumps the list and the characteristics of all epics variables which would
be loaded if the IOC gets rebooted. Redirect the output to a text file and
check if variables you modified were correctly processed.
The .db files contain
the list of epics variables (or records) which will be loaded with
the IOCs. Several fields can be set to setup the variables. Those which
are not explicitely set have default values. Some records can be defined with
a parameter -- like
$(N) -- which is
useful to define similar records, for instance a quantity which is monitored
separately in each of the 12 sectors. As an example, the file
contains the variable
field(INPA,"DRC::ENV:VSM:WRM$(N).SEVR CP NMS")
field(INPB,"CEN:BIP:DAQ:STATUS.VAL CP NMS")
This definition is loaded in the IOC via the file
which contains the following lines:
They mean in particular that 12 'copies' of the record
will be constructed:
(N = 0), ...,
(N = 11).
The two other options
V are used to set
some fields of other variables (in that case scaler minor and major alarm
levels) in the same db
The record definition shown above gives a good idea of the most important
- DESC: short
(limited number of characters! Potential syntax error when the file gets
loaded...) description of the variable.
- SCAN: record
- CALC: field
defining how the record value is computed.
In this example, one tests the value of the field
INPB (defined two
lines below as the value of another variable
and one returns the value of the severity (NO ALAM, MINOR, MAJOR or NOT
CONNECTED) of the record
if B = 1, 0 otherwise.
- HIGH gives the
value of the first level of 'high' (= above nominal) alarm.
- HIHI is the
threshold for the second level of 'high' alarm.
There are similar fields for 'low' (= below nominal) alarm thresholds:
- Finally, the two fields
associate the span style="color: blue; font-family: monospace">HIGH
alarm level to a MINOR (= yellow) alarm and
HIHI to a major
(= red) alarm.
More information about epics records can be found in the Reference Manual.
This directory contains the binary files defining the epics panels and which
are generated when the directory
adl/ gets compiled.
Scripts which are executed via the alarm handler (see
.alh files for
details) when some alarms are raised are located here. For instance the
action triggered by the script can be to page automatically the DIRC oncall
expert in case of a serious alarm like a dead fan in a front-end crate.
This directory contains the state machines. A state machine is a pseudo
C program which monitors some part of the DIRC (for instance the HV).
regularly, it checks the status of this component (by collecting information
from epics records) and decides what to to depending on the state which
has been identified.
Keeping the same example: when the DIRC is runnable, the HV ramping state
is in state 'V0' and stays there until it receives the order to bring the DIRC
to injectable. When this happens, the state machine moves to some 'To V1'
states and finally reaches the state 'V1' where it stays until it is asked to
go to runnable again.
The DIRC uses several state machines for different purposes: HV ramping
up and down, Wiener crate fans and background monitoring, HV set to off or on
via epics etc.
State machines needs to be compiled as well.
- The convention in epics is that any button with an arrow
on the button should just open another panel without performing
any action. In the case of DIRC our main panel (Monitor.dl) has
only such "arrow buttons". An example of an "action button" is
the V0 button on the HV panel (caen.dl) which directly switches
the HV levels (without asking for confirmation).
- The BaBar CVS repository is on the web.
- You can checkout a given tag of a CVS package in a directory with a name different from the package by using the command
cvs co -r <tag> -d <directory name> <CVS package name>
cvs co -r HEAD -d drc-mon-test epics-drc-mon
cvs checkout: Updating drc-mon-test
cvs checkout: Updating drc-mon-test/adl
- Epics command lines are available to follow the evolution of some variables from a terminal without starting any CPU and memory consuming client.
- caget gives the current value of the epics record/field given in parameter. For instance,
[Water level in the SOB in meters]
[Yellow alarm level on the sector 3 scaler]
- camonitor monitors the time evolution of a given record. For instance,
DRC::ENV:VSM:WRM3 06/18/06 10:51:34.857798754 4038
DRC::ENV:VSM:WRM3 06/18/06 10:51:38.857798594 4118
DRC::ENV:VSM:WRM3 06/18/06 10:51:42.873901310 4486
DRC::ENV:VSM:WRM3 06/18/06 10:51:47.873901110 4850
- caput allows you to change the value of a record 'live'. Make sure you know what you're doing...
If the change should be made permanent, don't forget to modify the IOC code accordingly. Otherwise, the modification will be lost next time the IOC gets rebooted.
bbr-dev100|~>caput DRC::ENV:VSM:WRM3.HIGH 451000
Old : DRC::ENV:VSM:WRM3.HIGH 450000
New : DRC::ENV:VSM:WRM3.HIGH 451000
bbr-dev100|~>caput DRC::ENV:VSM:WRM3.HIGH 450000
Old : DRC::ENV:VSM:WRM3.HIGH 451000
New : DRC::ENV:VSM:WRM3.HIGH 450000
- A few words about drc-soft
- The procedure to restart
instance after having introduced some changes in the package) is explained
in the pilot long-term instructions. In short:
- Open a x-term window on con01 (you need to be logged as babar to
restart the IOC);
- Some white/red alarms should pop up on the pilot ALH but they should
go away after a few seconds. If so, you're done!
drc-soft is not
requested for running (having problems with the IOC will not interfere
with data taking!), it is better to test any new code before rebooting
the IOC. The way to do this is the following:
- Log-on to an IR2 subnet machine, e.g.
- Run drc-soft-<yourTag>/boot/start_standalone.cmd [in principle you shouldn't have
modified the currently running version of
have rather checked out the code in a different directory where you would
have introduced your changes...].
- Check for any unexpected message showing up on the console (if you're
not sure of their meaning, ask another epics expert).
- If there are none, check some epics variables with the
(in particular those you have modified/introduced!) to make sure they
are updated correctly and that the values make sense. To avoid mismatch
with the variables belonging to the
drc-soft IOC which
is currently running, the epics names are different in this test mode. The
normal 'BKG' tag
is replaced by
instance, the variable
(scaler rate ratio in sector 3) would be
DRC::TEST:SC_3:HIT_RATIO and so on...
- Troubleshooting: If you have troubles
in standalone mode, look at the following
- Do not make non-trivial changes in the IOC directories used in production.
Check out a new copy of the package, modify it, compile it and finally reboot
the IOC after having changed the symbolic link when you have a good opportunity
to test the changes (at least a few hours without data taking).
- Commit in CVS the changes which have been tested and work.
- Tag frequently the running areas in CVS to have up-to-date snapshots of
- Do not reboot an IOC during data taking unless there is an obvious
problem with it.
- An IOC which has crashed or which is hanging up should not directly
rebooted: page the ODC oncall expert so that he/she could investigate the
problem to try to understand what happened.
Last modified: Sun Aug 13 10:48:37 PDT 2006