|
LCLS
Alarms: overview
and maintenance |
Contents:
II. Add items to ALH/Alarm Summary IOCs
III. Add items to EPICS Watcher IOC
Related Links:
Soft
IOC release procedure : general
soft IOC release procedure
Helper scripts : Perl scripts to assist with creating alarm groups,
generating IOC database files, etc.
EPICS Collaboration ALH page:
with links to documentation
I.
Description of LCLS Alarm System:
The LCLS Alarm system consists of 5 elements which are described below.
1. EPICS Alarm Handler (ALH)
Config file location: /usr/local/lcls/tools/alh/config/*.alhConfig
config is a symlink to the current version, with version numbers corresponding to the alarm release number
This is the alarm tree gui developed by the EPICS collaboration. Control system PVs are configured into a tree structure, with alarms summarizing up the tree.
There are 2 ways to invoke the alh gui:
o Click the “Alarm tree…” button on lclshome
o Run the linux script lclsalh
Enable/disable alarms: FORCEPVs:
Normally change of alarm status of any control system PV in the tree shows up in the ALH gui.
However, there is a built-in alarm disable/enable feature of ALH, using so-called FORCEPVs. A FORCEPV is configured for each control system PV in the alarm tree. When the FORCEPV value is 0,alarming is enabled for the corresponding PV: any change in alarm status appears in the tree. When the FORCEPV is 1, alarming is disabled for that PV: it always shows a NO_ALARM state.
FORCEPVs are instantiated in the Alarm summary IOCs…see below.
A PV with alarms disabled is indicated in the ALH tree gui: the PV has a mask valueof <-DATL> in the gui. This mask value propagates up the tree along with alarm status. Therefore you can drill down in the tree to find PVs that have been disabled.
2 bash scripts control disabling and enabling alarms at any level of the tree:
o disableALHGroup
o enableALHGroup
Please click “Help” on the lclshome display or refer to the LCLS Alarm wiki page above for instructions on using these scripts.
2. Alarm summary IOCs
IOC application location: $EPICS_IOC_TOP/Alarms
The alarm summary IOCs summarize alarms from all control system PVs in the ALH tree, with a summary PV created at each level of the tree. At the top level, summary severities are used to create the colored squares on lclshome. In other words, the alarm summary PV tree structure directly mirrors the alh gui tree structure.
There are 6 PRODUCTION alarm summary IOCs, divided geographically:
o sioc-sys0-al00– alarm summaries for global and IN20, VMS interface and odd calculation databases.
o sioc-sys0-al01– alarm summaries for BSY and LTU, and MCCO databases, AL01 VMS interface database.
o sioc-sys0-al03 – alarm summaries for li21 - li24, and li20 PVs
o sioc-sys0-al04 – alarm summaries for li25 – li30
o sioc-sys0-al05 – alarm summaries for UND
o sioc-sys0-al06 – alarm summaries for DMP, FEE, NEH, XRT, FEH
On the DEVELOPMENT side (lcls-dev2, where they provide colors for the development lclshome), these are called:
o sioc-sys0-al0d
o sioc-sys0-al1d
o sioc-sys0-al3d
o sioc-sys0-al4d
o sioc-sys0-al5d
o sioc-sys0-al6d
The IOCs operate independently, and are rebooted depending on the specific needs of the release at hand. It’s ok if the IOCs are running different versions of the application.
PVs:
There are 3 main types of PVs
booted into each Alarm Summary IOC, and these control alarm functions
(FORCEPVs, DP PVs, alarm summary PVs). Name
below is the same as the corresponding alhConfig file.
There are also a couple of special databases involved with including VMS PVs in
the alarm system.
And, for the linac upgrade testing period, there is extra PV logic to enable quick enable/disable of upgrade-related PVs (see below0
· FORCEPVs:
Instantiated from alhFP_alhConfigname.template (e.g. alhFP_vac_li24_vpio.template)
FORCEPVs
1. allow alarms in the ALH gui tree to be disabled/enabled, as described in #1 above
2. disable/enable alarm summary PVs in the IOC application.
There is one FORCEPV per control system PV in the alarm tree; the naming convention is pvNameFP (for example,BEND:IN20:231:STATMSGFP)
FORCEPVs control alarm enable/disable in the alarm handler gui. In the alarm IOCs, there is another layer of records to accommodate linac upgrade requirements (see below); the PVs that enable/disable alarms for summary PVs in the alarm IOCs are “SV” PVs. Normally these mirror the value of the corresponding FP PVs.
·
“DP”PVs:
Also instantiated from alhFP_alhConfigname.template (e.g. alhFP_vac_li24_vpio.template)
DP PVs:
1. mirror the alarm severity of its control system PV.
2. are sensitive to the value of its corresponding FORCEPV. If its FORCEPV is 0, its alarm severity matches that of the control system PV. If its FORCEPVis 1, the alarm severity is always NO_ALARM, i.e. the alarm is disabled.
There is one DP PV per control system PV in the alarm tree; the naming convention is pvNameDP (e.g.BEND:IN20:231:STATMSGDP)
·
Alarm summary PVs:
Instantiated from alhSP_alhConfigname.db (e.g. alhSP_vac_li24.db, or alhSP_vac_li24_vpio.db)
The alarm summary PVs are calc records that maximize severity for a set of input DP PVs. There is a summary PV for every group (level) in the ALH tree, summing up to the top level. The severity of the top level for each subsystem/area creates the colorfor the corresponding lclshome box. As described above FORCEPVs in concert with DP PVs allow summary PV alarms to be enabled/disabled.
Special alarm PVs for VMS:
There are 2 types of PVs for representing VMS alarms in the LCLS alarm system:
1. PVs that can reflect VMS alarm status
Certain primaries in VMS (generally those for which you can caget <pv>.clr) pass a VMS alarm severity to EPICS through the SLCCAS. This alarm severity is visible to tools that access severity at a low level: edm and alh for example. Alarms color for these PVs can be seen in edm, and also the alh gui. Examples:
LI21:ASTS:ACC_H2O
LI30:VACV:902:SHUTTER
LI24:QUAD:201:STRG
However, to represent these PVs in the alarm system summary PVs and propagate their status to lclshome, it's necessary to create a set of corresponding PVs to capture the VMS severity in <pv>.SEVR. There is a parallel set of records that does this, see:
- alhVMS_sevr.db and .template
- alhVMS_sevr_asts.db and template
- alhVMS_sevr_asts_nms.db and template
These parallel records are configured in the alh gui and alarm summaries. edm displays may show alarms from the original VMS PVs.
2. PVs that can't reflect VMS alarm status
The rest of the primaries in VMS don't send any alarm severity to EPICS. In this case, it's necessary to create parallel EPICS PVs for setting alarm limits and showing severity. Examples:
LI20:KLYS:51:KPHR
LI21:CSTR:1:STAT
LI29:STEP:96:VACT
The databases/templates that create these parallel PVs are:
alhAPV_micro.db and .template
alhAPV_step.db and .template
alhAPV_klys.db and .template
These parallel records are configured in the alh gui and alarm summaries, and in edm color PVs.
3. ALH logging and Channel Watcher daemons (lcls-daemon2)
ALH logging daemon config file location: /usr/local/lcls/tools/alh/config/DAEMON/*.alhConfig
CW config file location: /usr/local/lcls/tools/ChannelWatcher/config/SIOC-SYS0-AL0*.cwConfig
· The ALH logging daemon runs the alarm handler in a display-free logging mode. All changes in alarm status for all enabled PVs in the alh tree are logged to cmlog.
· The ChannelWatcher daemon logs and save/restores all changes to the FORCEPVs that enable/disable alarms (see above Alarm Summary IOCs section) Therefore the enable/disable state of each PV is saved across alarm IOC reboot.
4. EPICS Watcher IOC, watcher diag display, watcherCUD
The EPICS Watcher IOC – sioc-sys0-al02 – hosts alarm PVs requiring more complex logic. The origin version was an EPICS translation of Joe Frisch’s MatlabWatcher. More types of alarms have been added over time.
The watcher diag display is /usr/local/lcls/tools/edm/display/lcls/watcher_test.edl, which can be invoked from lclshome by clicking “Matlab GUIs…”/”GOTO Watcher…”/”Watcher Diag Display…”
Robin Gold maintains and updates the watcher CUD display that runs in the control room.
5. lcls_main.edl, lcls_alarm_summary.cud displays
o lcls_main.edl is the lclshome display.
o lcls_alarm_summary.cud is the alarm CUD shown on overhead displays in the control room.
Both displays have 2 layered elements for each box:
1. Alarm sensitivity to corresponding top-level summary PV (e.g. NTWK:LI23:1:STATSUMY)
2. A selectively visible “*” character that appears when a PV somewhere in the tree underneath has had its alarming disabled. The “*” is visible if the corresponding top-level FP summary PV is MINOR, indicating a “1” somewhere down the tree (e.g. NTWK:LI23:1:STATSUMYFP)
During normal
production LCLS operations, alarms for Linac Upgrade PVs need to be
invisible to alarm summaries on lclshome, and disabled in the alarm handler
gui. During ROD testing periods, when
new devices and controls are tested within the context of production LCLS, the
new alarms need to be visible.
For a
full explanation of how this is accomplished, see
alarmsForTransitionDesign.docx here: https://slacspace.slac.stanford.edu/sites/controls/specialprojects/S20-BSY/S20BSY%20Documents/Forms/AllItems.aspx?RootFolder=%2fsites%2fcontrols%2fspecialprojects%2fS20-BSY%2fS20BSY%20Documents%2fDesigns&FolderCTID=&View={12BDADA6-C4DC-4D91-9C59-FC8BFE2AA27F}
Operationally,
this means that databases and templates in the li20-li30 areas are a little
more complicated.
Normally, FORCEPVs control alarm enable/disable in the alarm handler gui. In the alarm IOCs, there is another layer of records to accommodate linac upgrade requirements (see below); the PVs that enable/disable alarms for summary PVs in all of the alarm IOCs are “SV” PVs. Normally the “SV” PVs simply mirror the value of the corresponding FP PVs.
In the alarm IOCs that handle sectors li20 – li30 (sioc-sys0-al03, sioc-sys0-al04) the “SV” PVs are more complex. They take the corresponding “FP” PV, and combine it with the status of the PV source IOC (SLC or EPICS, depending on whether the PV is currently coming from its original source: the SLC system, or the new source: EPICS iocs)
adding elements to the alarm system: ALH/Alarm Summary IOCs/lclshome
For general details on modifying and
releasing soft IOCs, see Soft
IOC release procedure
Perl scripts to assist with creating alarm
groups, generating IOC database files, etc.: Helper scripts
Checklist
to follow when creating an alarm release: CHECKLIST
|
1.Modify alarm handler gui and auto-generate: ·
IOC db files ·
IOC template
files ·
channel
watcher config lines |
Please note: any small or large syntax error in any part of the tree will RENDER THE ENTIRE GUI COMPLETELY un-displayable. It’s really important to create a test display of the whole tree before putting changes into production! (-: Intro: At the top level of the ALH gui, the tree is divided into functional beamlines, which allows folks to evaluate the alarm status of particular beam paths. The subtree of interest for maintenance purposes is “All LCLS”, which contains ALL alarm PVs, in a tree structure. The alh tree is (with a few exceptions) composed of groups with this structure like this: Area Subsystem Devicetype Unit List of alarm PVs for example: LI24 Vacuum VPIO VPIO:LI24:730 VPIO:LI24:730:EXTINTLK VPIO:LI24:730:P etc. The alarm tree is constructed from ascii config files, *.alhConfig, with a separate file for each subsystem/device type - i.e. the file's place in the tree is reflected in its name. (e.g. vac_li24_vpio.alhConfig) The tree is constructed by nested “include” statements in a hierarchy of alhConfig files. For example: · vac_li24_vpio.alhConfig has alarms for all VPIO vacuum devices in li24 · vac_li24.alhConfig includes all vacuum device type files in li24 · li24.alhConfig includes all subsystem files in li24 The bad news: there are hundreds of files. The good news: 1. the filenames tell you exactly what’s in them. 2. you can easily operate on small parts of the tree with no effect on the rest of the tree. (see LCLS Alarm wiki page for excruciating details on naming convention, etc.) PLEASE NOTE though, there are
special cases that deviate from this norm. When making
changes/additions, please locate similar structures in the tree, and emulate
them, or cut/paste.
Steps: 1. cvs checkout tools/alh/config into a sandbox directory: cvs co tools/alh/config 2. Identify where your changes will go, and the relevant alhConfig files. Are you simply adding PVs to an existing device? Are you adding devices? Are you adding a completely new subsystem? If you are adding alarm groups: please see the appendix A for possible rare but present PV naming collision possibilities to watch out for! 3. Implement the changes. For small numbers of PVs, simply cutting and pasting in the alhConfig file of choice works fine. For adding masses of devices or large chunks of subsystems, there are helper scripts for constructing the file details. (see Helper scripts for details) Cutting and pasting from similar files is helpful. Warning: even a tiny syntax error in a file will cause the entire tree to fail – i.e. it will not display, and the error message is absolutely unhelpful. Suggestion: add small numbers of files at a time, and test in between (even one by one) 4. Test: Feel free to clone the bash script /home/softegr/jrock/lclsalhTEST on prod to point to your own sandbox alhConfig directory. Run it to see your new tree. Have the subsystem engineer review the new tree if you’re implementiing many changes – this will save lots of time later! Check especially for disconnected (white “E”) PVs that should exist – can indicate typos. 5. cvs
alhConfig files, but don’t update in prod yet (this is instantly visible, and
shouldn’t be done until the IOCs are rebooted, but you can prepare by checking out the new version): · cvs commit in sandbox dir tag the alh config release: ·
cd
/usr/local/lcls/tools/alh
·
6. Generate
IOC database and template files from new/modified alhConfig files. You can do this using
perl scripts makeSummaryPVdbs.pl and makeForcePVdbTemplate.pl in your
alhConfig sandbox directory (see Helper scripts for details)
ALL helper scripts are located in /usr/local/lcls/tools/alh/script. Or, a more labor-intensive alternative is to make
changes manually to the IOC database files (see section 2.2 below) For small
changes this can make perfect sense - use the cut-and-paste method on the
alhSP_* and alhFP_* files of choice. To generate the 2 types
of IOC files from the alhConfig files: ·
Summary PV databases: one of these per alhConfig file: alhSP_alhfilename.db makeSummaryPVdbs.pl –fix xx_yy_zz.alhConfig ·
FORCEPV/DP PV template: one of these per lowest-level alhConfig
file(i.e. containing control system PVs): alhFP_alhfilename.template makeForcePVdbTemplate.pl -fix
xx_yy_zz.alhConfig For sioc-sys0-al03 and al04, li20-li30, take this additional step for
linac upgrade: ·
Modify the FORCEPV PV template to add the PV
source macro: changeLUTemplates.pl –fix ALRM:CONSTPV:ZEROAL xx_yy_zz.alhConfig (ALRM:CONSTPV:ZEROAL is
a placeholder PV that will be replace by an IOC modepv when needed) Take a look at the .db
and .template files to make sure they look ok. 7. Create
adds/mods to ChannelWatcher config files: For most IOCs: makeForcePVcw.pl –fix FORcw xxx.alhConfig For sioc-sys0-al03 and
04 for linac upgrade use this syntax instead: makeForcePVcw.pl DV
–fix FORcw xxx.alhConfig this creates a file called FORcw with the necessary cw lines. which you
will use to edit the cwConfig file. |
|
2. Make a new Alarms IOC Application release |
Please refer to Soft
IOC release procedure for details on Soft IOC releases. Only an outline of the cvs and release
steps is presented below. Steps: 1. cvs checkout the head version of the IOC application: cvs co Alarms 2. Create new .db and .template files in the application’s sandbox Alarms/alhPVApp/Db directory: · copy the new .db and .template files which you created in step 1.6 above, from your alh sandbox directory over to your sandbox IOC directory …/Alarms/alhPVApp/Db directory OR · manually modify the .db and .template files corresponding to your modified alhConfig files (alhSP_alhConfigname.db, alhFP_alhConfigname.template) For small changes this can make perfect sense - use cut-and-paste on the alhSP_* and alhFP_* files of choice. 3. Determine which alarm IOC(s) you’ll be working with for the release (see geographical designations in the section I.2 above) 4. If new files were created for the release, in the sandbox: · modify the Db/Makefile to add the newly created files. · cvs add the new files · modify the relevant IOCs’ startup files to add the new alhSP..db and alhFP..template files. · try make from the Alarms director and make sure your new files appear in the proper location (db). 5. Still
in the sandbox, cvs the new files, create the new release. Then
checkout and build your new release in the production IOC area,
$EPICS_IOC_TOP: · cd to the top of the application in the sandbox area ·
make
clean uninstall ·
cvs
commit ·
cvs tag
Alarms-Rx-y-z
·
· cd Alarms-Rx-y-z ·
make 6. Point IOC boot dirs of the IOCs involved in the release to new version: ·
cd $IOC/<iocname> · change bin symlink to point to new version · edit TO_ROLLBACK file for the new version |
|
3. Modify Channel Watcher config files |
Steps: 1. cvs checkout the Channel Watcher files: cvs co tools/ChannelWatcher/config/SIOC-SYS0-AL0x.cwConfig where x corresponds to the IOC(s) you have changed. 2. Edit the relevant file(s), merging in the changes from the FORcw file(s) you created in part 1 above. 3. CVS
channel watcher files and move to prod · cvs commit in sandbox dir · cd/usr/local/lcls/tools/ChannelWatcher/config · cvs update SIOC-SYS0-AL0*.cwConfig (please explicitly cvs update these files instead of running a blanket “cvs update”; otherwise you will inadvertently update other developers’ cwConfig files!!) |
|
4. If needed, modify lcls_main.edl and
lcls_alarm_summary.cud display |
If you’ve added or deleted an entire area/subsystem box, you’ll need to
modify lcls_main.edl and lcls_alarm_summary.cud Steps: 1. cvs co tools/edm/display/lcls/lcls_main.edl and lcls_alarm_summary.cud 2. edit the lcls_main.edl display (cut and paste works well for this) 3. lcls_alarm_summary.cud is difficult to edit (requiring using edm, and then vi to correct) – if you’re not comfortable doing this you can ask Jonathan Warren, the display’s author, for helpful hints. But if you’re brave, here’s what you do: · cp lcls_alarm_summary.cud lcls_alarm_summaryCUD.edl · edm lcls_alarm_summaryCUD.edl (edm doesn’t like .cud extensions during editing!) · make your changes · you may need to move the large black background rectangle aside to click on the canvas, in order to do operations like save and execute. If so, make note of their positions before you do the move. · test, save the display · vi lcls_alarm_summaryCUD.edl · return the large black background rectangle back to its original position by replacing its X and Y position. · finally, cp lcls_alarm_summaryCUD.edl lcls_alarm_summary.cud. 4. run lclshome and execute the lcls_alarm_summary.cud in your sandbox display dir to see the new versions and make sure you haven’t broken anything (you won’t have summary colors for the new areas yet: new areas will appear white) |
|
5. CVS everything, put all in production and reboot
IOCs |
Here’s where you cvs commit everything, and move it into production in
several steps, and it all becomes extremely VISIBLE! Steps: 1. Send
out a Software Mini-Test Plan.
Rebooting an IOC will cause the corresponding parts of lclshome to go
momentarily white, and if PVs in the rebooted IOCs are referenced in other
areas, these areas may show purple temporarily.
Once you’re
given the go-ahead by the EOIC and the CD: 2. Reboot
sioc-sys0-al0x ·
Use the /etc/inid.d scripts on lcls-daemon1 (usual
soft ioc boot procedure)as laci, one at a time: st.sioc-sys0-al0x restart where x corresponds to the IOC(s) you have changed. monitor each reboot with
iocConsole. · After each reboot, check lclshome carefully for unexplained purple boxes, or other effects. 3. update
alarm handler files in prod and prepare daemon directory: · cd /usr/local/lcls/tools/alh/config ·
cvs
update · test by running lclsalh prepare the DAEMON
alhConfig directory with the new stuff: ·
cd
/usr/local/lcls/tools/alh/config/DAEMON ·
cp
../*.alhConfig . ·
cp
./specialFiles/*.* . 4. Restart
daemon processes: ·
on lcls-daemon2 as laci st.alh restart st.cwAL restart 5. If
you have a new version of lcls_main and cud, cd to your edm sandbox directory
and run lclshome and the lcls_alarm_summary.cud to see the new version of the
displays. If everything looks good: ·
cvs
commit ·
cd
/usr/local/lcls/tools/edm/display/lcls ·
cvs
update lcls_main.edl ·
cp
/usr/local/lcls/tools/edm/display/lcls/lcls_main.edl /usr/local/lcls/tools/edm/Upgrade-EdmDevl/. (lclshome runs from this new directory temporarily) ·
cvs
update lcls_alarm_summary.cud · run lclshome to make sure all is well 6. Log
your followup “completed successfully” entry to the software e-log. If you’ve modified the displays, remind
people to restart lclshome to see the changes. |
|
In needed, restart displays in the control room |
If you’ve modified lcls_main.edl and lcls_alarm_summary.cud, go to the
control room and: ·
restart
all the lclshome displays running (the operators will help with this) ·
restart
the lcls_alarm_summary.cud displays (operators will help with this) |
adding elements to the alarm system: EPICS Watcher IOC
For general details on modifying and releasing
soft IOCs, see Soft
IOC release procedure
Adding instances of existing logic (simplest case):