LCLS Alarms:

overview and maintenance




I.  Description

II. Add items to ALH/Alarm Summary IOCs

III. Add items to EPICS Watcher IOC

IV. Maintenance



Related Links:

LCLS Alarm wiki page

Soft IOC release procedure : general soft IOC release procedure

Helper scripts : Perl scripts to assist with creating alarm groups, generating IOC database files, etc.
EPICS Collaboration ALH page: with links to documentation



I. Description of LCLS Alarm System:


The LCLS Alarm system consists of 5 elements which are described below.


1.     EPICS Alarm Handler (ALH)


Config file location:  /usr/local/lcls/tools/alh/config/*.alhConfig

config is a symlink to the current version, with version numbers corresponding to the alarm release number


This is the alarm tree gui developed by the EPICS collaboration.   Control system PVs are configured into a tree structure, with alarms summarizing up the tree.

There are 2 ways to invoke the alh gui:

o   Click the “Alarm tree…” button on lclshome

o   Run the linux script  lclsalh


Enable/disable alarms: FORCEPVs:

Normally change of alarm status of any control system PV in the tree shows up in the ALH gui.


However, there is a built-in alarm disable/enable feature of ALH, using so-called FORCEPVs.  A FORCEPV is configured for each control system PV in the alarm tree.  When the FORCEPV value is 0,alarming is enabled for the corresponding PV: any change in alarm status appears in the tree.  When the FORCEPV is 1, alarming is disabled for that PV: it always shows a NO_ALARM state.

FORCEPVs are instantiated in the Alarm summary IOCs…see below.


A PV with alarms disabled is indicated in the ALH tree gui:  the PV has a mask valueof <-DATL> in the gui.  This mask value propagates up the tree along with alarm status.  Therefore you can drill down in the tree to find PVs that have been disabled.


2 bash scripts control disabling and enabling alarms at any level of the tree:

o   disableALHGroup

o   enableALHGroup

Please click “Help” on the lclshome display or refer to the LCLS Alarm wiki page above for instructions on using these scripts.


2.     Alarm summary IOCs


IOC application location:  $EPICS_IOC_TOP/Alarms


The alarm summary IOCs summarize alarms from all control system PVs in the ALH tree, with a summary PV created at each level of the tree.  At the top level, summary severities are used to create the colored squares on lclshome.  In other words, the alarm summary PV tree structure directly mirrors the alh gui tree structure.


There are 6 PRODUCTION alarm summary IOCs, divided geographically:

o   sioc-sys0-al00– alarm summaries for global and IN20, VMS interface and odd calculation databases.

o   sioc-sys0-al01– alarm summaries for BSY and LTU, and MCCO databases, AL01 VMS interface database.

o   sioc-sys0-al03 – alarm summaries for li21 - li24, and li20 PVs

o   sioc-sys0-al04 – alarm summaries for li25 – li30

o   sioc-sys0-al05 – alarm summaries for UND

o   sioc-sys0-al06 – alarm summaries for DMP, FEE, NEH, XRT, FEH


On the DEVELOPMENT side (lcls-dev2, where they provide colors for the development lclshome), these are called:

o   sioc-sys0-al0d

o   sioc-sys0-al1d

o   sioc-sys0-al3d

o   sioc-sys0-al4d

o   sioc-sys0-al5d

o   sioc-sys0-al6d


The IOCs operate independently, and are rebooted depending on the specific needs of the release at hand.  It’s ok if the IOCs are running different versions of the application.



There are 3 main types of PVs booted into each Alarm Summary IOC, and these control alarm functions (FORCEPVs, DP PVs, alarm summary PVs).  Name below is the same as the corresponding alhConfig file. 
There are also a couple of special databases involved with including VMS PVs in the alarm system.


And, for the linac upgrade testing period, there is extra PV logic to enable quick enable/disable of upgrade-related PVs (see below0


·         FORCEPVs:



Instantiated from alhFP_alhConfigname.template (e.g. alhFP_vac_li24_vpio.template)



1.      allow alarms in the ALH gui tree to be disabled/enabled, as described in #1 above 

2.      disable/enable alarm summary PVs in the IOC application.


There is one FORCEPV per control system PV in the alarm tree; the naming convention is pvNameFP (for example,BEND:IN20:231:STATMSGFP)

FORCEPVs control alarm enable/disable in the alarm handler gui.  In the alarm IOCs, there is another layer of records to accommodate linac upgrade requirements (see below); the PVs that enable/disable alarms for summary PVs in the alarm IOCs are “SV” PVs.  Normally these mirror the value of the corresponding FP PVs.


·          “DP”PVs:


Also instantiated from alhFP_alhConfigname.template (e.g. alhFP_vac_li24_vpio.template)



1.      mirror the alarm severity of its control system PV. 

2.      are sensitive to the value of its corresponding FORCEPV.  If its FORCEPV is 0, its alarm severity matches that of the control system PV.  If its FORCEPVis 1, the alarm severity is always NO_ALARM, i.e. the alarm is disabled.


There is one DP PV per control system PV in the alarm tree; the naming convention is pvNameDP (e.g.BEND:IN20:231:STATMSGDP)


·         Alarm summary PVs:


Instantiated from alhSP_alhConfigname.db (e.g. alhSP_vac_li24.db, or alhSP_vac_li24_vpio.db)


The alarm summary PVs are calc records that maximize severity for a set of input DP PVs.  There is a summary PV for every group (level) in the ALH tree, summing up to the top level. The severity of the top level for each subsystem/area creates the colorfor the corresponding lclshome box.  As described above FORCEPVs in concert with DP PVs allow summary PV alarms to be enabled/disabled.


Special alarm PVs for VMS:

There are 2 types of PVs for representing VMS alarms in the LCLS alarm system:

1. PVs that can reflect VMS alarm status

Certain primaries in VMS (generally those for which you can caget <pv>.clr) pass a VMS alarm severity to EPICS through the SLCCAS.  This alarm severity is visible to tools that access severity at a low level: edm and alh for example.  Alarms color for these PVs can be seen in edm, and also the alh gui.  Examples:




However, to represent these PVs in the alarm system summary PVs and propagate their status to lclshome, it's necessary to create a set of corresponding PVs to capture the VMS severity in <pv>.SEVR.  There is a parallel set of records that does this, see:

  • alhVMS_sevr.db and .template
  • alhVMS_sevr_asts.db and template
  • alhVMS_sevr_asts_nms.db and template


These parallel records are configured in the alh gui and alarm summaries.  edm displays may show alarms from the original VMS PVs.

2. PVs that can't reflect VMS alarm status

The rest of the primaries in VMS don't send any alarm severity to EPICS.  In this case, it's necessary to create parallel EPICS PVs for setting alarm limits and showing severity.  Examples:




The databases/templates that create these parallel PVs are:

alhAPV_micro.db and .template

alhAPV_step.db and .template

alhAPV_klys.db and .template

These parallel records are configured in the alh gui and alarm summaries, and in edm color PVs.



3.     ALH logging and Channel Watcher daemons (lcls-daemon2)

ALH logging daemon config file location:  /usr/local/lcls/tools/alh/config/DAEMON/*.alhConfig

CW config file location:  /usr/local/lcls/tools/ChannelWatcher/config/SIOC-SYS0-AL0*.cwConfig


·         The ALH logging daemon runs the alarm handler in a display-free logging mode.  All changes in alarm status for all enabled PVs in the alh tree are logged to cmlog.


·         The ChannelWatcher daemon logs and save/restores all changes to the FORCEPVs that enable/disable alarms (see above Alarm Summary IOCs section)  Therefore the enable/disable state of each PV is saved across alarm IOC reboot.



4.     EPICS Watcher IOC, watcher diag display, watcherCUD

The EPICS Watcher IOC – sioc-sys0-al02 – hosts alarm PVs requiring more complex logic. The origin version was an EPICS translation of Joe Frisch’s MatlabWatcher.  More types of alarms have been added over time.


The watcher diag display is /usr/local/lcls/tools/edm/display/lcls/watcher_test.edl, which can be invoked from lclshome by clicking “Matlab GUIs…”/”GOTO Watcher…”/”Watcher Diag Display…”


Robin Gold maintains and updates the watcher CUD display that runs in the control room.



5.     lcls_main.edl, lcls_alarm_summary.cud displays

o        lcls_main.edl is the lclshome display.

o        lcls_alarm_summary.cud is the alarm CUD shown on overhead displays in the control room.


Both displays have 2 layered elements for each box:

1.      Alarm sensitivity to corresponding top-level summary PV (e.g. NTWK:LI23:1:STATSUMY)

2.      A selectively visible “*” character that appears when a PV somewhere in the tree underneath has had its alarming disabled.  The “*” is visible if the corresponding top-level FP summary PV is MINOR, indicating a “1” somewhere down the tree (e.g. NTWK:LI23:1:STATSUMYFP)



6. Special features for Linac Upgrade PVs in the alarm system (automatic disable/enable)

During normal production LCLS operations, alarms for Linac Upgrade PVs need to be invisible to alarm summaries on lclshome, and disabled in the alarm handler gui.  During ROD testing periods, when new devices and controls are tested within the context of production LCLS, the new alarms need to be visible.

For a full explanation of how this is accomplished, see alarmsForTransitionDesign.docx here:{12BDADA6-C4DC-4D91-9C59-FC8BFE2AA27F}


Operationally, this means that databases and templates in the li20-li30 areas are a little more complicated.


Normally, FORCEPVs control alarm enable/disable in the alarm handler gui.  In the alarm IOCs, there is another layer of records to accommodate linac upgrade requirements (see below); the PVs that enable/disable alarms for summary PVs in all of the alarm IOCs are “SV” PVs.  Normally the “SV” PVs simply mirror the value of the corresponding FP PVs.


In the alarm IOCs that handle sectors li20 – li30 (sioc-sys0-al03, sioc-sys0-al04) the “SV” PVs are more complex.  They take the corresponding “FP” PV, and combine it with the status of the PV source IOC (SLC or EPICS, depending on whether the PV is currently coming from its original source: the SLC system, or the new source:  EPICS iocs)   


sioc-sys0-al03 and al04 use different and more complex versions of the basic alhFORCEPV.db, called alhDEVLPV.db.

The CAMAC IOC mode PVs that are used to enable/disable the alarms for the linac upgrade are:
The "constantly 0" PV for other alarm PVs mixed in with linac upgrade PVs is:

for an example of how these PVs look in a .template file see: alhFP_evg_li22_ioc.template




II.  Details

adding elements to the alarm system: ALH/Alarm Summary IOCs/lclshome


For general details on modifying and releasing soft IOCs, see Soft IOC release procedure

Perl scripts to assist with creating alarm groups, generating IOC database files, etc.: Helper scripts

          Checklist to follow when creating an alarm release: CHECKLIST

1.Modify alarm handler gui and auto-generate:

·        IOC db files

·        IOC template files

·        channel watcher config lines


Please note: any small or large syntax error in any part of the tree will RENDER THE ENTIRE GUI COMPLETELY un-displayable.  It’s really important to create a test display of the whole tree before putting changes into production!  (-:



At the top level of the ALH gui, the tree is divided into functional beamlines, which allows folks to evaluate the alarm status of particular beam paths.


The subtree of interest for maintenance purposes is “All LCLS”, which contains ALL alarm PVs, in a tree structure.


The alh tree is (with a few exceptions) composed of groups with this structure like this:





                                    List of alarm PVs

for example:










The alarm tree is constructed from ascii config files, *.alhConfig, with a separate file for each subsystem/device type - i.e. the file's place in the tree is reflected in its name.  (e.g. vac_li24_vpio.alhConfig)  The tree is constructed by nested “include” statements in a hierarchy of alhConfig files. 

For example:

·         vac_li24_vpio.alhConfig has alarms for all VPIO vacuum devices in li24

·         vac_li24.alhConfig includes all vacuum device type files in li24

·         li24.alhConfig includes all subsystem files in li24


The bad news: 

there are hundreds of files. 

The good news:

1.      the filenames tell you exactly what’s in them. 

2.      you can easily operate on small parts of the tree with no effect on the rest of the tree.

(see LCLS Alarm wiki page for excruciating details on naming convention, etc.)

PLEASE NOTE though, there are special cases that deviate from this norm.  When making changes/additions, please locate similar structures in the tree, and emulate them, or cut/paste.



1.      cvs checkout tools/alh/config into a sandbox directory:

cvs co tools/alh/config


2.      Identify where your changes will go, and the relevant alhConfig files.  Are you simply adding PVs to an existing device?  Are you adding devices?  Are you adding a completely new subsystem?  If you are adding alarm groups: please see the appendix A for possible rare but present PV naming collision possibilities to watch out for!


3.      Implement the changes.  For small numbers of PVs, simply cutting and pasting in the alhConfig file of choice works fine.  For adding masses of devices or large chunks of subsystems, there are helper scripts for constructing the file details.  (see Helper scripts for details)  Cutting and pasting from similar files is helpful. 


Warning: even a tiny syntax error in a file will cause the entire tree to fail – i.e. it will not display, and the error message is absolutely unhelpful.  Suggestion: add small numbers of files at a time, and test in between (even one by one)


4.      Test:

Feel free to clone the bash script

/home/softegr/jrock/lclsalhTEST on prod

to point to your own sandbox alhConfig directory.  Run it to see your new tree.   Have the subsystem engineer review the new tree if you’re implementiing many changes – this will save lots of time later!  Check especially for disconnected (white “E”) PVs that should exist – can indicate typos.


5.      cvs alhConfig files, but don’t update in prod yet (this is instantly visible, and shouldn’t be done until the IOCs are rebooted, but you can prepare by checking out the new version):

·         cvs commit in sandbox dir

tag the alh config release:
cvs tag Alarm_Configs-R*same release version as the new IOC application*

·         cd /usr/local/lcls/tools/alh

cvs co -d Alarm_Configs-R*new version* tools/alh/config

when you're finished releasing, testing and verifying the IOC release, then move the config symlink to the new version.
try lclsalh to make sure files are in the right place
cd config, and mkdir DAEMON.
cp *.alhConfig DAEMON/.
cp /usr/local/lcls/tools/alh/specialFilesForDAEMON/.



6.      Generate IOC database and template files from new/modified alhConfig files.

You can do this using perl scripts and in your alhConfig sandbox directory (see Helper scripts for details)  ALL helper scripts are located in /usr/local/lcls/tools/alh/script.


Or, a  more labor-intensive alternative is to make changes manually to the IOC database files (see section 2.2 below)  For small changes this can make perfect sense - use the cut-and-paste method on the alhSP_* and alhFP_* files of choice.


To generate the 2 types of IOC files from the alhConfig files:

·         Summary PV databases:  one of these per alhConfig file:  alhSP_alhfilename.db –fix xx_yy_zz.alhConfig


·         FORCEPV/DP PV template:  one of these per lowest-level alhConfig file(i.e. containing control system PVs): alhFP_alhfilename.template  -fix xx_yy_zz.alhConfig


For sioc-sys0-al03 and al04, li20-li30, take this additional step for linac upgrade:

·         Modify the FORCEPV PV template to add the PV source macro: –fix ALRM:CONSTPV:ZEROAL xx_yy_zz.alhConfig


(ALRM:CONSTPV:ZEROAL is a placeholder PV that will be replace by an IOC modepv when needed)


Take a look at the .db and .template files to make sure they look ok.


7.      Create adds/mods to ChannelWatcher config files:

For most IOCs: –fix FORcw xxx.alhConfig


For sioc-sys0-al03 and 04 for linac upgrade use this syntax instead: DV –fix FORcw xxx.alhConfig


this creates a file called FORcw with the necessary cw lines. which you will use to edit the cwConfig file.



2.  Make a new Alarms IOC Application release


Please refer to Soft IOC release procedure for details on Soft IOC releases.  Only an outline of the cvs and release steps is presented below.



1.      cvs checkout the head version of the IOC application:

cvs co Alarms


2.      Create new .db and .template files in the application’s sandbox Alarms/alhPVApp/Db directory:

·         copy the new .db and .template files which you created in step 1.6 above, from your alh sandbox directory over to your sandbox IOC directory …/Alarms/alhPVApp/Db directory


·         manually modify the .db and .template files corresponding to your modified alhConfig files (alhSP_alhConfigname.db, alhFP_alhConfigname.template)   For small changes this can make perfect sense - use cut-and-paste on the alhSP_* and alhFP_* files of choice.



3.      Determine which alarm IOC(s) you’ll be working with for the release (see geographical designations in the section I.2 above)


4.      If new files were created for the release, in the sandbox:

·         modify the Db/Makefile to add the newly created files.

·         cvs add the new files

·         modify the relevant IOCs’ startup files to add the new alhSP..db and alhFP..template files.

·         try make from the Alarms director and make sure your new files appear in the proper location (db).


5.      Still in the sandbox, cvs the new files, create the new release.  Then checkout and build your new release in the production IOC area, $EPICS_IOC_TOP:

·         cd to the top of the application in the sandbox area

·         make clean uninstall

·         cvs commit

·         cvs tag Alarms-Rx-y-z

cd $EPICS_IOC_TOP/Alarms

cvs co -d Alarms-Rx-y-z  –r Alarms-Rx-y-z Alarms

·         cd Alarms-Rx-y-z

·         make


6.      Point IOC boot dirs of the IOCs involved in the release to new version:

·      cd $IOC/<iocname>

·      change bin symlink to point to new version

·      edit TO_ROLLBACK file for the new version



3.  Modify Channel Watcher config files



1.      cvs checkout the Channel Watcher files:

cvs co tools/ChannelWatcher/config/SIOC-SYS0-AL0x.cwConfig

where x corresponds to the IOC(s) you have changed.


2.      Edit the relevant file(s), merging in the changes from the FORcw file(s) you created in part 1 above.


3.      CVS channel watcher files and move to prod

·         cvs commit in sandbox dir

·         cd/usr/local/lcls/tools/ChannelWatcher/config

·         cvs update SIOC-SYS0-AL0*.cwConfig   (please explicitly cvs update these files instead of running a blanket “cvs update”; otherwise you will inadvertently update other developers’ cwConfig files!!)


4.  If needed, modify lcls_main.edl and lcls_alarm_summary.cud display


If you’ve added or deleted an entire area/subsystem box, you’ll need to modify lcls_main.edl and lcls_alarm_summary.cud



1.      cvs co tools/edm/display/lcls/lcls_main.edl and lcls_alarm_summary.cud


2.      edit the lcls_main.edl display (cut and paste works well for this)


3.      lcls_alarm_summary.cud is difficult to edit (requiring using edm, and then vi to correct) – if you’re not comfortable doing this you can ask Jonathan Warren, the display’s author, for helpful hints.  But if you’re brave, here’s what you do:

·         cp lcls_alarm_summary.cud lcls_alarm_summaryCUD.edl

·         edm lcls_alarm_summaryCUD.edl (edm doesn’t like .cud extensions during editing!)

·         make your changes

·         you may need to move the large black background rectangle aside to click on the canvas, in order to do operations like save and execute.  If so, make note of their positions before you do the move.

·         test, save the display

·         vi lcls_alarm_summaryCUD.edl

·         return the large black background rectangle back to its original position by replacing its X and Y position.

·         finally, cp lcls_alarm_summaryCUD.edl lcls_alarm_summary.cud.


4.      run lclshome and execute the lcls_alarm_summary.cud in your sandbox display dir to see the new versions and make sure you haven’t broken anything (you won’t have summary colors for the new areas yet: new areas will appear white)


5.     CVS everything, put all in production and reboot IOCs


Here’s where you cvs commit everything, and move it into production in several steps, and it all becomes extremely VISIBLE!



1.      Send out a Software Mini-Test Plan.  Rebooting an IOC will cause the corresponding parts of lclshome to go momentarily white, and if PVs in the rebooted IOCs are referenced in other areas, these areas may show purple temporarily.


Once you’re given the go-ahead by the EOIC and the CD:

2.      Reboot sioc-sys0-al0x

·        Use the /etc/inid.d scripts on lcls-daemon1 (usual soft ioc boot procedure)as laci, one at a time:

st.sioc-sys0-al0x restart

where x corresponds to the IOC(s) you have changed.

monitor each reboot with iocConsole.

·         After each reboot, check lclshome carefully for unexplained purple boxes, or other effects.


3.      update alarm handler files in prod and prepare daemon directory:

·         cd /usr/local/lcls/tools/alh/config

·         cvs update

·         test by running lclsalh


prepare the DAEMON alhConfig directory with the new stuff:

·         cd /usr/local/lcls/tools/alh/config/DAEMON

·         cp ../*.alhConfig .

·         cp ./specialFiles/*.* .


4.      Restart daemon processes:

·        on lcls-daemon2 as laci

st.alh restart

st.cwAL restart


5.      If you have a new version of lcls_main and cud, cd to your edm sandbox directory and run lclshome and the lcls_alarm_summary.cud to see the new version of the displays.  If everything looks good:

·         cvs commit

·         cd /usr/local/lcls/tools/edm/display/lcls

·         cvs update lcls_main.edl

·         cp /usr/local/lcls/tools/edm/display/lcls/lcls_main.edl   /usr/local/lcls/tools/edm/Upgrade-EdmDevl/.

(lclshome runs from this new directory temporarily)

·         cvs update lcls_alarm_summary.cud

·         run lclshome to make sure all is well


6.      Log your followup “completed successfully” entry to the software e-log.  If you’ve modified the displays, remind people to restart lclshome to see the changes.


In needed, restart displays in the control room

If you’ve modified lcls_main.edl and lcls_alarm_summary.cud, go to the control room and:

·         restart all the lclshome displays running (the operators will help with this)

·         restart the lcls_alarm_summary.cud displays (operators will help with this)



III.  Details

adding elements to the alarm system: EPICS Watcher IOC


For general details on modifying and releasing soft IOCs, see Soft IOC release procedure

Adding instances of existing logic (simplest case):


If you're adding a new type of logic, you'll need to go through the process of checking out to dev, creating your new databases, testing.

Many changes to the watcher involve simply adding a new instance of an existing logical database. In this case see II. above.  cvs checkout the IOC app, and add rows to the relevant section of watcher.template, then jump to #4 below.

For example, the most common are:

watcherHeartBeatCheck.db: simple incrementor check

watcherFbckCheck.db: check for Diane's feedbacks

1.  Checkout application on DEV

cvs checkout the Alarms application head version on lcls-dev2

2.  Create/mod databases

All watcher databases are:


They are all booted into sioc-sys0-al02 with


Create/mod the databases and template as needed.  vdct may be useful  here (so far I’ve just used vi…)

3.  Test

Test on the dev network; create test input PVs as needed.

For examples of testing, see $EPICS_IOC_TOP/Test/Development

which is booted into sioc-sys0-TEST

4.  CVS changes and make a new Alarms IOC Application release on production


See section II. above and Soft IOC release procedure for details

5.  Modify watcher displays

·         Modify Watcher Diag display

cvs co tools/edm/display/lcls/watcher_test.edl

make mods (cut and paste can be useful here!)

cvs commit

6. Reboot sioc-sys0-al02


1.      Send out a Software Mini-Test Plan.   Restarting the EPICS Watcher IOC has no effect on the beam or other systems.


Once you’re given the go-ahead by the EOIC and the CD:

2.      Reboot sioc-sys0-al02

·        Use the /etc/init.d script on  lcls-daemon1 as laci (usual soft ioc boot procedure)

st.sioc-sys0-al02 restart

monitor the reboot with iocConsole.


·         After the, check the new watcher_test.edl display, esp. the new/changed items.


7. Release displays

cd /usr/local/lcls/tools/edm/display/lcls

cvs update watcher_test.edl

test from lclshome

Ask Robin Gold to modify the EPICS Watcher CUD display


Maintenance activities

In general, the alarm system requires no scheduled care and feeding.  Here is a list of maintenance items though, which I'll keep adding to as they come up.



A.   potential gotchas in creating summary PVs:


- device-level summary PV duplicate

  if a device name is used in > 1 subsystem in alh, the device-level group and summary PV name will be

  duplicated.  Need to modify the device-level summary names to avoid name collision (i.e. add subsystem

  name.)  See EVG IOC device-level summaries for an example, with names changed to avoid collision with NTWK,

  and TEMP device-level summaries with names changed to avoid collision with MGNT.


- top-level summary PV duplicate


    * device type = a subsystem name (e.g. MPS, or BCS)

    * area = one of the alarming areas in lclshome (in20Gr, li22)

    * unit number = 1

  the device-level summary PV name will be equal to the device-top-level summary PV name.  Need to

  modify the name at the device level to avoid collision (see MPS PVs in the vacuum subsystem,

  or BCS:LI24:1 device in the BCS subsystem)



Last modified 29-September-2010  J. Rock