EPICS Save/Restore (sr) Upgrade Project

(Channel Watcher)

• Staged Implementation
• Master and Channel Group File Formats
• How to Start Channel Watcher
• Software Distribution Page

Problem: The current version of Save/Restore (sr), also known as Bumpless Reboot, may be interfering with normal IOC operations due to the large number of file writes.

Solution: Move the file writes (i.e. Save functions) to a client computer (i.e. UNIX box).  We may even eventually do away with the file altogether and use a database such as Oracle.  Leave the restore functions on the IOC.

Notes from the EPICS Workshop, Dec. 3&4, 2001, Fairmont Hotel, San Jose: Jeff Hill suggests that we design the software with Object Oriented Plug-ins so users can choose which features to include in their local Channel Watcher builds.  Click here for software design notes.

Staged Implementation:

The software upgrade will be developed in the following stages.  Each stage represents a point at which the software will be tested and released.

Stage 1: Simple move to UNIX with enhancements

Stage 1 Enhancements:

• Toggle channel logging on the fly
• Add and Remove channels on the fly
• /nowrite option for logging only and toggle it on the fly.  Channels with the /nowire option do not cause the Restore Repository to be generated.
• Don't write unknown values to Repository for when Channel Watcher can't connect to IOC, like when it's down
• Allow other Channel Groups inside the Channel Group (i.e. Masterfile).
• Web documentation
• Put Channel Groups and the Master Files in AFS and manage using CVS.  Manually FTP or SCP files to the gateway /usr/local area after a change.
• Add a channel alias name which will be supplied as a comment next to the channel name in some Channel Group files.
• Put Repositories in the /u1 area (different from the area for the Channel Groups) where they will be backed up every night.  Each Repository may be in a different directory so if a Master File is used, it must specify the Channel Group file name, with path, and the path and name of the Repository created for it.
• Write Repository on any channel value change, but no more than every configurable seconds.  See -t option on startup.
• Repository files need to have owner and group write privilege
• Write Repositories in s/r version V1.91 format.  This format is in ascii because it is easily editable by somebody making a software release who needs the IOC to reboot with different values.
• Remove check for string of nulls.
• Create a health summary EPICS Record and additional CWlog messages when this record goes into alarm state.  Record names are CS02:CWPEPII:SUMY:STAT and CS02:CWNLCTA:SUMY:STAT (not yet available).   mbbo enum values are:
  • 0 = OK
  • 1 = DOWN
  • 2 = HUNG
  • 3 = CONNERR
  • 4 = DISKERR
  • probably others up to 15
• Don't tie Channel Watcher to a specific file system such as afs or nfs.
• If, for some reason, the Channel Watcher is prone to crashing, make cron job that will execute every 5 minutes and restart any missing Channel Watcher process (we may have one cron job per Channel Watcher process if that's better).
• Test Plan: In order to ensure the Channel Watcher is properly running we plan on  running it in parallel with the current Save/Restore software on the IOC, using different Repositories, of course, and comparing the Repositories generated by both.  When we're satisfied Channel Watcher works we'll need to add Channel Watcher to the UNIX startup files and remove the create_monitor_sets from IOC startup.

The following tags may be used in CWlog messages:

Tag	Value
text	text of message like what epicsPrintf provides
facility	Channel Watcher
host	IOC associated with the channel
device	Channel Name
message	Channel alias
severity	EPICS severity of the channel
status	EPICS status of the channel
value	Current value of the channel as a string
code	See Channel Watcher file CWlogmsgABC.hh for a list of codes

Stage 3: Statistics

Web interface to list channels by

• Active Channel Group 
• Clickable Inactive Channel Group files just like windows does 
• IOC 
• Alphabetically 
• How often channels are changing and flag channels changing faster than some max rate that we can change on the fly.  Even have a different max update rate per channel and a list of channels by max update rate.
• Time of last change 
• Which channels are not connected (just like the Channel Archiver) 

Stage 4: Wildcarding using regular expressions

Add wildcarding of channel names in the Channel Groups and to the Statistics tool listed above using the same Oracle Database we now know and love.  Steph says that there are epics library tools that already know how to resolve regular expressions.

As long as we're doing that Kristi wants the software designed in such a way as to make it easy to make a tool that will list channels given a regular expression.  Make it and call it ChannelLister.

Stage 5: Channel Groups and Repositories in Oracle Database

Stage 5 Enhancements:

• Real on the fly toggling of /log, /nowrite,  add/remove channel, and maximum update rate on a per channel basis with messages to CWlog.  Make sure to allow wildcarding with regular expressions.
• GUI replaces Channel Groups in files
• Channel values written to Oracle Database, but file Repositories still generated.
• Channels with /nowrite should not update the Oracle Database.
• On demand save via a button push that allows different file names for the Repository, but the Bumpless Reboot functionality still generates the Repository on value change
• CGI scripts in Perl for web interface to data stored in Oracle

.

Stage 6: Restore from Oracle Database

Instead of reading the file Repositories the  IOC could get the channel values directly from the Oracle Database possibly with Aida.  Steph also points out that at this point the Channel Watcher could run on an IOC so make sure the code is generic enough so that it can run on multiple platforms.

Master File Format:

Channel-Group-file-name <white space> Repository_file_name

Master File Notes: 

• Optional items are underlined.  
• Repository file name defaults to Channel-Group-file-name.sav
• Allow macros in the above file names and use the epics macro library.
• Make sure Channel Group file names don't recurse.

Channel Group File Format: optional items in italics

# comment

channel:name <white space> /log <white space> /nowrite <white space> # <white space> channel:alias

channel:name <white space> /LOG <white space> /NOWRITE <white space> # <white space> channel:alias

<blank lines>

How to start the Channel Watcher:

Options for starting channel watcher:

Environment variables used by Channel Watcher:

• CW_CHANNEL_GROUP_ROOT - optional.  Specifies the top of the Channel Group location.
• CW_REPOSITORY_ROOT - optional.  Specifies the top of the Repository location.

Notes:

• Where does Oracle Database live?  Ron suggests gateway, but Oracle isn't installed on the gateway and Jingchen doesn't want it there at least until we're on a more open network.
• What GUI did we decide on for Stage 3?  Perl with CGI.
• How does the current sr restore enums?  By string or by value?  Both are present in the Repositories.  You're not gonna like this: fdbrestore uses the string; the first value that's between the ctrl-A terminators.  reboot_restore uses the number; the second value that's between the second ctrl-A terminator and eol.
• Steph knows how to add the Channel Watcher to the (SLC?) network display.
• Bob Hall and Jingchen know how to manage Channel Watcher tasks with a cron job.

Definitions:

• Channel Groups - User created files, usually by an editor, list of epics channels for sr and now Channel Watcher.  This is the only file not automatically generated.
• Channel Watcher - Replaces save/restore "save" part by migrating its functions to UNIX.
• Common Message Logger - EPICS extension Channel Watcher may use to report errors or channel value changes when logging is requested.  Also known as cmlog. 
• On the Fly - For Stage 1 implementation this means: edit your channel list file then restart the Channel Watcher.  For stage 4 implementation this will be done with a GUI and message passing.
• Repository - Files automatically generated by sr and Channel Watcher that the restore functions reboot_restore and fdbrestore need in order to restore channel values on IOC reboot. A repository may contain 3 different files: 
  • ChannelGroup.sav - Last successful Repository written
  • ChannelGroup.sav.bu - Repository from Last IOC boot
  • ChannelGroup.sav.next - Current working file.  This file may become the .sav file if it passes all validity checks.
• Save Restore (sr) - A way to save and restore epics channels on an IOC. Also known as Bumpless Reboot.

Last Update 10-Oct-2002 by Mike Zelazny zelazny@slac.stanford.edu