CRITICAL TEMPERATURE MONITORING DETAILS

In general

Temperatures are monitored with 3 summary PVs - the MMF PVs are on a development soft IOC, the Undulator PV is on a production soft IOC (Arturo Alarcon). An alarm handler daemon process is running on development and production. When the alarm status of either temp monitor PV increases (e.g. goes from NO_ALARM to MAJOR or INVALID), the alh daemon process will send a text message to the phone numbers in the appropriate VMS-based list (Ken Brobeck). The lists are show below.

The Undulator PV is always configured to send alerts. The MMF PV alerts are turned on ONLY during the winter downtime.

For the MMF system, it is not uncommon for the temperature hardware to disconnect briefly and very temporarily. To avoid sending non-temperature-related alarms to everyone, Arturo created a separate PV for this disconnect status, configured in the alh daemon with a command to send a text and e-mail to him only.

alh guis, en/disable actions, monitoring and restarting the daemons

Undulators: Production

Monitor PV is USEG:UND1:100:HVACSTATUS

ALH GUI for the monitor PV:
from lclshome click "Alarm Handler Tree" button
in the alh gui, navigate down to All LCLS/UND1/Temp/Undulator Temp Monitor

Disable/Enable the monitor PV alarm:
Use the usual production disable/enable scripts (see lclshome "Help..." for help)
Log into production host (e.g. as softegr) and run shell script:

disableALHGroup USEG:UND1:100:HVACSTATUS
enableALHGroup USEG:UND1:100:HVACSTATUS
Alarm logging:
alarms are logged to production cmlog

to see if the Alarm handler daemon is running:
Log into lcls-srv01 or lcls-srv02 as eoicdesk
Run the UWD (enter uwd)
Navigate down to Controls_UNIX_Watchdog/Controls_Servers/lcls-daemon2/PROC/lcls-daemon2 AlarmHandler(LCLS)

OR
log into lcls-daemon2 as laci
ps -ef | grep LCLS_All
you should see a process running that is owned by laci.

To restart the alh daemon:
log into lcls-daemon2 as laci
cd /etc/init.d
./st.alh restart

alhConfig file containing undulator temp alarm:
/usr/local/lcls/tools/alh/config/DAEMON/temp_und1_useg.alhConfig

MMF: Development

Monitor PV for Temperature Alarms is RTD:MMF0:114:MEANTEMP
Monitor PV for Disconnect status is RTD:MMF0:114:STATUS

ALH GUI for the monitor PV:
log into lcls-dev2 (bash) and run this script:
~jrock/work_alh/tools/alh/script/alhGUIDEV

Disable/Enable the monitor PV alarm:
(The two monitor PVs can be enabled and disabled independently.)
Log into lcls-dev2
disable: caput RTD:MMF0:114:MEANTEMPFP 1
disable: caput RTD:MMF0:114:STATUSFP 1

enable: caput RTD:MMF0:114:MEANTEMPFP 0
enable: caput RTD:MMF0:114:STATUSFP 0

Alarm logging:
alarms are logged to file /nfs/slac/g/lcls/tools/alh/log/Alarms

to see the Alarm handler daemon status:
log into lcls-dev2 as laci
ps -ef | grep LCLS_All
you should see a process running that is owned by laci.

To restart the alh daemon:
log into lcls-dev2 as laci
cd /etc/init.d
./st.alh restart

alhConfig file containing MMF temp alarm:
/afs/slac/g/lcls/tools/alh/config/DAEMON/LCLS_All.alhConfig

VMS-based phone lists for alarm text messages (Ken Brobeck):

MMF_Alert:
!Ken Brobeck
brobeck@slac.stanford.edu
! Jim Welch
6507148412@txt.att.net
! Ernest Williams
6502158569@tmomail.net
! Roaming Guard
6502846731@tmomail.net
! Roaming Guard
6502846732@tmomail.net
!
!Heinz-Dieter
6508234526@txt.att.net
!William Colocho
6502009193@tmomail.net
!ON CALL HVAC
6505708811@myairmail.com
!Arturo
6503873093@txt.att.net
Und_temp:
!Ken Brobeck
brobeck@slac.stanford.edu
! Jim Welch
6507148412@txt.att.net
! Ernest Williams
6502158569@tmomail.net
! Roaming Guard
6502846731@tmomail.net
! Roaming Guard
6502846732@tmomail.net
!
!Gene Kraft
4086918018@vtext.com
!Heinz-Dieter
6508234526@txt.att.net
!William Colocho
6502009193@tmomail.net
!ON CALL HVAC
6505708811@myairmail.com
!Mike Zurawel
6508469838@myairmail.com
!Arturo 6503873093@txt.att.net
For MMF disconnect alarms only
Messages are sent directly (i.e. do not use a VMS-based phone list):
Arturo: cell phone text message
Arturo: email