Some information about epics

• DIRC Detector Control section of the Operations Manual;
• Rebooting the DIRC IOCs section of the Operations Manual;
• DIRC epics webpage;
• Taking Data and Running the State Machine webpage (for shifters);
• Using the Alarm Handler webpage (for shifters);
• Using EPICS panels webpage (for shifters).

To steer and monitor the DIRC we have

• three 'hardware' (= real boards plugged in VME crates) IOCs: drc-mon, drc-hv and drc-gas;
• one 'software' IOC running on a linux box: drc-soft.

The epics production area is /nfs/bbr-srv01/u1/babar/boot/apps, directory to which the environment variable DAPPDIR should be set. Each IOC uses its own package (called epics-drc-mon/hv/gas/soft respectively).

If you look in this directory you should see a structure similar to the example below (situation as of 2006/02/12):

Changing the version of the code running in an IOC is straightforward: one just needs to change the symbolic link and to reboot the IOC. Yet, this procedure should only be done by the DIRC epics experts (currently Christophe Thiebaux and Nicolas Arnaud) or by ODC experts (Matthias Wittgen in particular).

The IOC logfiles for the current day and for the past week can be found in the /nfs/bbr-nfs01/logfiles/Odc aerea. There is one directory per day which contains one file for each IOC.

Each IOC package contains several directories:

• adl/
  This directory contains the code source for all the IOC panels. When the package gets compiled, binary files loaded when some panels need to be displayed on a computer screen are stored in the dl/ directory.
• alh/
  It contains ascii files which define the contents on the DIRC section in the alarm handler (ALH). They are loaded each time the main ALH (on the con01 IR2 pilot console) gets restarted or reloaded. The exact same configuration appears on all the other alarm handler interfaces which are started (in read-only mode) by users afterwards.
  Hence, to have changes taken into account, the pilot should be asked to restart the ALH or to reload its configuration.
  Files with the label '-shutdown' should be used when some parts of the DIRC are off for a significant period of time, e.g. during a shutdown. These files are very similar to the ones which are used when the DIRC is on but most of the alarms are masked to avoid filling the ALH with fake or 'normal' alarms.
  The main DIRC ALH configuration file is drc-mon/alh/drc.alh. Inside are defined the files which are read by the pilot ALH. That's the file to edit to replace one config. file by its shutdown version.

  Alarms are stored in the cmlog logfiles. Two are available on the shifter consoles: the alarm handler logfile and the ORC logfile. Both can be queried with command lines to get the list of alarms which showed up in a given time range. Examples of scripts can be found here. Information is also available in the file OstErrLogger/README.cmlogQuery or in this page.

  More information on the ALH can be found in the Alarm Handler User's Guide.

• boot/
  The startup file read when the IOC is rebooted is located here. For an old VxWorks IOC (drc-mon and drc-gas), the file is start.cmd whereas it is st.sys for a new RTEMS IOC (drc-hv). This file configures the board, load the epics variables defined in the db/ directory and starts the state machines.
  To check that a modified db/ file has no syntax error, one can run the following command from the top directory:
  processdb boot/<startup file>
  
  It dumps the list and the characteristics of all epics variables which would be loaded if the IOC gets rebooted. Redirect the output to a text file and check if variables you modified were correctly processed.
• db/
  The .db files contain the list of epics variables (or records) which will be loaded with the IOCs. Several fields can be set to setup the variables. Those which are not explicitely set have default values. Some records can be defined with a parameter -- like $(N) -- which is useful to define similar records, for instance a quantity which is monitored separately in each of the 12 sectors. As an example, the file drc-mon/db/scaler.db contains the variable
  
  record(calc,"DRC::ENV:VSM:SM$(N)")
  {
          field(DESC,"combined Scaler/DAQ")
          field(SCAN,"Passive")
          field(PINI,"YES")
          field(CALC,"(B=1)?A:0")
          field(INPA,"DRC::ENV:VSM:WRM$(N).SEVR CP NMS")
          field(INPB,"CEN:BIP:DAQ:STATUS.VAL CP NMS")
          field(HIGH,"1")
          field(HIHI,"2")
          field(HSV,"MINOR")
          field(HHSV,"MAJOR")
  }
  
  This definition is loaded in the IOC via the file drc-mon/db/scaler.tmpl which contains the following lines:
  
  file db/scaler.db
  {
          {N=0,H=300000,V=400000}
          {N=1,H=300000,V=400000}
          {N=2,H=300000,V=400000}
          {N=3,H=300000,V=400000}
          {N=4,H=300000,V=400000}
          {N=5,H=300000,V=400000}
          {N=6,H=300000,V=400000}
          {N=7,H=300000,V=400000}
          {N=8,H=300000,V=400000}
          {N=9,H=300000,V=400000}
          {N=10,H=300000,V=400000}
          {N=11,H=300000,V=400000}
          {N=12,H=300000,V=400000}
          {N=13,H=300000,V=400000}
          {N=14,H=300000,V=400000}
  }
  
  They mean in particular that 12 'copies' of the record DRC::ENV:VSM:SM$(N) will be constructed: DRC::ENV:VSM:SM0 (N = 0), ..., DRC::ENV:VSM:SM11 (N = 11).
  The two other options H and V are used to set some fields of other variables (in that case scaler minor and major alarm levels) in the same db file.
  The record definition shown above gives a good idea of the most important fields:
  • DESC: short (limited number of characters! Potential syntax error when the file gets loaded...) description of the variable.
  • SCAN: record scan frequency.
  • CALC: field defining how the record value is computed.
    In this example, one tests the value of the field INPB (defined two lines below as the value of another variable CEN:BIP:DAQ:STATUS) and one returns the value of the severity (NO ALAM, MINOR, MAJOR or NOT CONNECTED) of the record DRC::ENV:VSM:WRM$(N) if B = 1, 0 otherwise.
  • HIGH gives the value of the first level of 'high' (= above nominal) alarm.
  • HIHI is the threshold for the second level of 'high' alarm.
    There are similar fields for 'low' (= below nominal) alarm thresholds: LOW and LOLO.
  • Finally, the two fields HSV and HHSV associate the span style="color: blue; font-family: monospace">HIGH alarm level to a MINOR (= yellow) alarm and HIHI to a major (= red) alarm.

  More information about epics records can be found in the Reference Manual.

• dl/
  This directory contains the binary files defining the epics panels and which are generated when the directory adl/ gets compiled.
• script/
  Scripts which are executed via the alarm handler (see .alh files for details) when some alarms are raised are located here. For instance the action triggered by the script can be to page automatically the DIRC oncall expert in case of a serious alarm like a dead fan in a front-end crate.
• src/
  This directory contains the state machines. A state machine is a pseudo C program which monitors some part of the DIRC (for instance the HV). regularly, it checks the status of this component (by collecting information from epics records) and decides what to to depending on the state which has been identified.
  Keeping the same example: when the DIRC is runnable, the HV ramping state machine (drc-hv/src/drcvoltage.st) is in state 'V0' and stays there until it receives the order to bring the DIRC to injectable. When this happens, the state machine moves to some 'To V1' states and finally reaches the state 'V1' where it stays until it is asked to go to runnable again.
  The DIRC uses several state machines for different purposes: HV ramping up and down, Wiener crate fans and background monitoring, HV set to off or on via epics etc.
  State machines needs to be compiled as well.

Useful information

• The convention in epics is that any button with an arrow on the button should just open another panel without performing any action. In the case of DIRC our main panel (Monitor.dl) has only such "arrow buttons". An example of an "action button" is the V0 button on the HV panel (caen.dl) which directly switches the HV levels (without asking for confirmation).
• The BaBar CVS repository is on the web.
• You can checkout a given tag of a CVS package in a directory with a name different from the package by using the command
  cvs co -r <tag> -d <directory name> <CVS package name>
  For instance
  cvs co -r HEAD -d drc-mon-test epics-drc-mon
  cvs checkout: Updating drc-mon-test
  U drc-mon-test/Makefile
  cvs checkout: Updating drc-mon-test/adl
  U drc-mon-test/adl/Makefile
  U drc-mon-test/adl/Monitor.adl
  (...)
• Epics command lines are available to follow the evolution of some variables from a terminal without starting any CPU and memory consuming client.
  • caget gives the current value of the epics record/field given in parameter. For instance,
    bbr-dev100|~>caget DRC::WTR:VSM:CALC0
    DRC::WTR:VSM:CALC0 5.41107
    [Water level in the SOB in meters]
    or
    bbr-dev100|~>caget DRC::ENV:VSM:WRM3.HIGH
    DRC::ENV:VSM:WRM3.HIGH 450000
    [Yellow alarm level on the sector 3 scaler]
  • camonitor monitors the time evolution of a given record. For instance,
    bbr-dev100|~>camonitor DRC::ENV:VSM:WRM3
    DRC::ENV:VSM:WRM3 06/18/06 10:51:34.857798754 4038
    DRC::ENV:VSM:WRM3 06/18/06 10:51:38.857798594 4118
    DRC::ENV:VSM:WRM3 06/18/06 10:51:42.873901310 4486
    DRC::ENV:VSM:WRM3 06/18/06 10:51:47.873901110 4850
    (...)
  • caput allows you to change the value of a record 'live'. Make sure you know what you're doing...
    bbr-dev100|~>caget DRC::ENV:VSM:WRM3.HIGH
    DRC::ENV:VSM:WRM3.HIGH 450000
    bbr-dev100|~>caput DRC::ENV:VSM:WRM3.HIGH 451000
    Old : DRC::ENV:VSM:WRM3.HIGH 450000
    New : DRC::ENV:VSM:WRM3.HIGH 451000
    bbr-dev100|~>caput DRC::ENV:VSM:WRM3.HIGH 450000
    Old : DRC::ENV:VSM:WRM3.HIGH 451000
    New : DRC::ENV:VSM:WRM3.HIGH 450000
    If the change should be made permanent, don't forget to modify the IOC code accordingly. Otherwise, the modification will be lost next time the IOC gets rebooted.
• A few words about drc-soft
  • The procedure to restart drc-soft (for instance after having introduced some changes in the package) is explained in the pilot long-term instructions. In short:
    • Open a x-term window on con01 (you need to be logged as babar to restart the IOC);
    • cd $DAPPDIR/drc-soft/boot
    • restartCommand.csh
    • Some white/red alarms should pop up on the pilot ALH but they should go away after a few seconds. If so, you're done!
  • Although drc-soft is not requested for running (having problems with the IOC will not interfere with data taking!), it is better to test any new code before rebooting the IOC. The way to do this is the following:
    • Log-on to an IR2 subnet machine, e.g. bbr-dev100.
    • Run drc-soft-<yourTag>/boot/start_standalone.cmd [in principle you shouldn't have modified the currently running version of drc-soft but have rather checked out the code in a different directory where you would have introduced your changes...].
    • Check for any unexpected message showing up on the console (if you're not sure of their meaning, ask another epics expert).
    • If there are none, check some epics variables with the camonitor command (in particular those you have modified/introduced!) to make sure they are updated correctly and that the values make sense. To avoid mismatch with the variables belonging to the drc-soft IOC which is currently running, the epics names are different in this test mode. The normal 'BKG' tag is replaced by 'TEST': for instance, the variable DRC::BKG:SC_3:HIT_RATIO (scaler rate ratio in sector 3) would be DRC::TEST:SC_3:HIT_RATIO and so on...
    • Troubleshooting: If you have troubles starting drc-soft in standalone mode, look at the following file.

Important instructions:

• Do not make non-trivial changes in the IOC directories used in production. Check out a new copy of the package, modify it, compile it and finally reboot the IOC after having changed the symbolic link when you have a good opportunity to test the changes (at least a few hours without data taking).
• Commit in CVS the changes which have been tested and work.
• Tag frequently the running areas in CVS to have up-to-date snapshots of satisfactory packages.
• Do not reboot an IOC during data taking unless there is an obvious problem with it.
• An IOC which has crashed or which is hanging up should not directly rebooted: page the ODC oncall expert so that he/she could investigate the problem to try to understand what happened.