Output Data Generator Configuration
System Administrator and Programmer Documentation
Wish List for New Features
The generic Channel Archiver Engine is available to all members of the EPICS collaboration. The Engine is an executable program that runs continually, archiving Channel Access data for selected channels. A major modification of the Channel Archiver Engine was made at SLAC to store data into an Oracle database rather than data files. Currently at SLAC three instances of this program run concurrently to archive NLCTA (TARF), 8-Pack, and PEPII data. The Archive Engine web page provides information about the Engine, including usage and configuration information.
The Channel Archiver Monitor is only available at SLAC, producing summary information using facilities available at SLAC. Three instances of this program run at SLAC (on the same machines the three instances of the Engine run to archive NLCTA, 8-Pack, and PEPII data). The Monitor generates cmlog messages that appear in the SLAC MCC error log to alert Operations regaring the status of the Channel Archiver Engine and the channels it is archiving. Messages are generated indicating when the Engine was started and stopped, when the Engine was detected to be not currently running, and when the Engine's write thread is busy (indicating that the Engine is in a "hung" state). If the Engine process is not currently running, the Monitor process will attempt to start the Engine process (unless it has been unsuccessful in starting the Engine process for a specified consecutive number of attempts). Messages are also generated regarding the channels the Engine is archiving: messages are generated when a group of channels is enabled or disabled and when a channel becomes disconnected. A summary message is also produced showing how many channels are currently disconnected. In addition, the Monitor writes values to an EPICS channel indicating the state of the Engine it is monitoring. Values are written indicating whether the Engine is OK, down, hung, or has archived channels that are currently disconnected. The values of the EPICS channels indicating the state of the PEPII and NLCTA Engines are displayed on the SLAC SCP Network display and the SLAC system CUD.
The Channel Archiver Browser is at present only available at SLAC. The Browser is a GUI tool that provides a 2D plot of Channel Archiver data. The data is read from files generated by the Engine. A plot may contain channels from one of the Channel Archiver data sources (NLCTA, 8-Pack, or PEPII). The Browser allows the user to select the channels to plot, to specify the time span, and to specify the Y axis scaling. Configuration files allow one to save channel and curve color/style information for later loading. The Browser may be used to generate a Matlab file of the data currently being plotted and to generate Postscript output describing the plot image for printing. The Channel Archiver Browser User's Guide describes all of the features of the browser.
The Channel Archiver Output Data Generator is at present only available at SLAC. It is a command line tool for generating a Matlab file or text data information from Channel Archiver data when the channels and time span of interest are known. If a user knows the channels and time span of interest, using the Output Data Generator is a quicker method of generating a Matlab file than using the Browser. Also the Output Data Generator may be used to output text data information including the timestamp, value, and comments associated with each data sample.
|Project||Host||Start Script||Stop Script||PV List|
An archive engine may be stopped by one of two methods. For both methods one first needs to login to the host on which the archiver to be stopped is currently running (see Table 1-1 above) using the cddev account.
The first method of stopping an archive engine is to simply kill the Engine process using the Unix kill command. One may use a command such as
The second method of stopping an archive engine is to use a script in the /afs/slac/g/cd/soft/ref/app/channelArchiver/script directory. After logging in using the cddev account, an archive engine may be stopped by issuing the following commands:
One must be aware that a Channel Archive Monitor process associated with an Engine runs every five minutes via a cron job and will restart its associated Engine if it is not running. This is usually the desired behavior because the reason for stopping an Engine is often to restart it soon after to read a modified pvlist file. Another reason is to have an Engine reconnect with channels from an IOC that has been unavailable for an extended period.
However, on rare occasions it is desired to not have an Engine process running for an extended period. To do this one must prevent the Channel Archive Monitor process associated with an Engine from running via a cron job. The easiest way to accomplish this is to stop the cddev account cron job for the associated machine. After logging into the cddev account and before stopping an Engine, the cddev account cron job may be stopped by issuing the following command:
IT IS VERY IMPORTANT TO RESTART THE CDDEV ACCOUNT CRON JOB WHEN IT IS DESIRED DESIRED TO START AN ARCHIVER ENGINE AGAIN. If the cddev account cron job was stopped on the machine where the NLCTA and 8-Pack archivers run, it may be restarted using the following commands:
Since the Channel Archiver Monitor process associated with an Engine is run every five minutes and will restart its associated Engine if it is not running, it is often not necessary to explicitly start an Engine. However, if is desired to explicitly start an Engine one should be logged onto the machine on which the Engine runs under the cddev account and issue the following commands:
The input file used to startup the Engine is named OraArPVList and is located in the project specific archive data files base directory. This input file specifies the following:
Other information may also be included; see the Archive Engine web page documentation referenced above for more information.
the PV (channel) names to be archived an indication for each specified channel whether a Monitor is to be posted the sampling rate (for non-monitored channels).
A configuration file, scannerConfig.txt, needs to be created in the directory specified by the ARBLOGFILES environment variable. It consists of three lines. The first line specifies the number of seconds during which a channel may be disconnected but the channels is still considered to be OK. A typical value for this line may be 900 (indicating 15 minutes). It is not desirable to report that a channel has disconnected if it reconnects within a reasonably short time period; this line specifies the reasonably short time period during which the channel may be disconnected but considered to be OK. The second line in the configuration file specifies the number of consecutive times that the Monitor process should attempt to start the Engine process if it is not running. If for some reason the Monitor process is unable to start the Engine process, this line limits the number of times the Monitor will make this attempt and produce messages indicating failure. The third line in the file specifies the maximum number of channel disconnect messages that will be generated each time the Monitor is run. This is needed to prevent a flood of error messages being displayed in the error log when many channels become disconnected at approximately the same time (e.g., when an IOC is no longer in service).
Before the Browser is invoked through a UNIX shell command, the user must explicitly perform some configuration tasks. The first step is to define required environment variables by means one of the following commands:
|ARDATAFILES||path containing the OraArPVList pv list file and browser filtering setup files|
|ARBLOGFILES||base path containing log files generated by the Browser|
|ARBPSFILES||path where Postscript files generated by users are stored|
|ARBMATLABFILES||path where Matlab files generated by users are stored|
|ARBCONFIGFILES||path where Configuration files saved by users are stored|
|ARBROWSER||path and name of the Browser executable file|
|ARBCONFIGEXT||the default configuration file extension (e.g., cfg)|
The environment variable BROWSER is also set to specify the browser used to display the Channel Archive Browser User's Guide when the "Help" menu bar item is selected in the Browser. For example, the BROWSER environment variable may be set to "netscape" to display the User's Guide using the netscape browser or "shelp" to display the User's Guide using the chimera browser.
For the environment variables listed above that specify a path where files are stored, the value of the environment variable specifies a path associated with NLCTA, 8-Pack, or PEPII files. The ARBLOGFILES environment variables only specify a base path: there are subdirectories underneath the base path with the names of the 12 months where files are stored depending on the current month.
The ARDATAFILES environment variable value must be a path where one
of the subdirectories if the path is either "tarf", "pack", or "pepii" in order
for the Browser to detect the appropriate channel filtering setup files.
If the ARDATAFILES environment variable specifies
a data directory, the Browser will look for the following channel filtering
|Project||Channel Filtering Setup Files|
Each channel filtering setup file is an ASCII file containing rows each having two fields: (1) a code (usually all uppercase letters), and (2) a description (embedded blanks are represented by underscores). The codes represent fields in SLAC PV channel names, which are separated by colons. The Browser uses these files to dynamically construct filtering list shown in the "Add Channels" dialog box (see the Archive Browser User's Guide for more information and an example). The projects subsystems listed in Table 2-2 have different naming convensions for fields in PV channel names which are commonly used. The channel filtering setup files may be edited by a UNIX text editor to reflect changes and additions the PV channel naming convensions.
If the Browser is invoked through a UNIX shell command, it is important to set the DISPLAY environment variable appropriately to cause the browser window to appear on your display. The DISPLAY environment variable is set using a command of the following form:
setenv DISPLAY machine_name:0.0
where machine_name is the name of the machine on which the browser window should appear.
Typical users will not want to invoke the Browser through a UNIX shell
command. Instead, the Browser may be invoked using the following scripts:
These scripts invoke the Python script archiveBrowser.py. The following options are used in invoking this script (typical users do not need to be aware of these):
The "-c" option is used to specify the name of a channel whose data will be displayed when the Browser is initially started.
The "-f" option is used to specify the file name and path of a configuration file that was previously saved using the Browser. If no file extension appears in the file name, the extension specified by the ARBCONFIGEXT environment variable is used (e.g., "cfg"). Configuration files provide a means for a user to start the Browser to show data for specified channels, a Y span, and curve color/styles. The last day of data for the specified channels is displayed for older configuration files that do not contain time period information. Newer configuration files contain time period information that reflects the last common time span selected by the user (if none have been selected, the default is one day). For these newer configuration files, data is retrieved from (present time - time period) to the present time.
If the "-f" or "-c" option is not used, the Browser will not display any data when it is initially started. In this case, the user may then specify the channels to be displayed or load a configuration file to display data for channels of interest using the Browser.
The "-u" option is used to specify a user name for diagnostic logging. Log information recording configuration file names loaded and saved, start/end times of data being displayed, and channel names being displayed are output to ASCII log files by the Browser. The specification of a user name allows this information to also be logged for a session for diagnostic purposes.
Specifying the "-n" option disables logging activity.
The Browser may be run on any machine that provides access to the NFS file system where the browser filtering setup files and log files are stored.
The Output Data Generator allows the use of the same configuration files saved previously by the using the Browser. The channel names and time period information contained in a configuration file may be overriden by specified channel names or time span information supplied when invoking the Output Data Generator. Channel names and time span information may both be explicitly specified without the use of a configuration file.
Configuration files contain time period information (days, hours, minutes, and seconds) [Note: older configuration files do not contain explicit time period information; the implicit time period for these files is one day]. If a configuration file is specified when invoking the Output Data Generator, data is retrieved for output from (present time - time period) to the present time. This "relative to the present" time period information may also be specified when invoking the Output Data Generator by use of the "-t" option. It is indicated by specifying either (1) days, hours, minutes, and seconds, or (2) a relative time keyword (last2hrs, last4hrs, last12hrs, last24hrs, last48hrs, last3days, last4days, lastweek, or last2weeks). Alternatively, an absolute time (a start and an end time) may be specified by either (1) the use of the "-t" option with the "yesterday" or "today" keyword, or (2) explicity specifying both the start and end times with the "-s" and "-e" options (specifying month, day, year, hour, minute, and second). If time is not specified (by the "-f", "-t", or "-s" and "-e"), the default time period is the last 24 hours.
Two types of output data may be generated: (1) Matlab data in a specified file, or (2) text data information. The default type of output data is text data information; Matlab data may be requested by specifying the "-m" option. Text data information consists of the timestamp, value, and comments associated with each desired data sample as well as informational data, such as when the channel was disconnected or the Archive Engine was shut down.
The Output Data Generator may be invoked by using the following scripts
listed in table 2-4 below.
These C shell scripts set needed environment variables and invoke the outputArchiveData.py Python script.
The options (-m, -f, -c, -t, -s, -e, -R, -o, -a, -A, and -h) are described below. The only required option is "-o" when the "-m" option was specified.
The "-m" option is used to request that Matlab data be generated in a specified output file. If the "-m" option is not specified, text data information is output instead.
The "-f" option is used to specify the file name and path of a configuration file that was previously saved using the Browser. If no file extension appears in the file name, the extension specified by the ARBCONFIGEXT environment variable is used (e.g., "cfg"). If no path is explicitly provided, the environment variable ARBCONFIGFILES will be used to determine the path (however, if this environment variable is not defined the path will be the current directory). Configuration files contain various Browser configuration information, including channel names and time period information that are used by the Output Data Generator.
The "-c" option is used to specify a list of comma separate channel names. It must be specified if a configuration file is not specified by use of the "-f" option. If the "-f" option is used, the use of the "-c" option specifies channel names that are used to override those contained in the specified configuration file.
The "-t" option is used to specify time period information. It may be used to specify one of three types of time period information: (1) days, hours, minutes, and seconds (dd:hh:mm:ss) time previous to the present time, (2) a keyword (last2hrs, last4hrs, last8hrs, last12hrs, last24hrs, last48hrs, last3days, last4days, lastweek, or last2weeks) indicating time previous to the present time, or (3) a keyword ("yesterday" or "today") indicating a absolute time span. If the "-f" option is not used, the "-t" option or the combination of the "-s" and "-e" options must be used to indicate the time period. If the "-f" option is used, the use of the "-t" option specifies a time period that overrides that contained in the specified configuration file.
The "-s" option is used to specify an explicit start time for data retrieval. It must be used in conjunction with the "-e" option, which is used to specify the end time. The start time is specified by providing the month, day, year, hour, minute, and second (mm/dd/yyyy hh:mm:ss). If the "-f" option is not used, the combination of the "-s" and "-e" options or the "-t" option must be used to indicate the time period. If the "-f" option is used, the use of the "-s" and "-e" options specifies a time span that overrides the time period information contained in the specified configuration file.
The "-e" option is used to specify an explicit end time for data retrieval. It must be used in conjunction with the "-s" option, which is used to specify the start time. The end time is specified by providing the month, day, year, hour, minute, and second (mm/dd/yyyy hh:mm:ss). If the "-f" option is not used, the combination of the "-s" and "-e" options or the "-t" option must be used to indicate the time period. If the "-f" option is used, the use of the "-s" and "-e" options specifies a time span that overrides the time period information contained in the specified configuration file.
The "-R" option is used to indicate that repeat count information is desired in the output. If this option is not specified, then by default repeat count information is not included in the output.
The "-o" option is used to specify the file name and path of an output file. This option is required if the "-m" option is specified. If the "-m" option is not specified, the output text data information is shown in the output window if the "-o" option is not specified. If the "-m" option is not specified and the "-o" option is specified, the text data information is stored in the specified file. If no file extension appears in the file name, the default extension ".mat" is used for Matlab output and the default extension ".dat" is used for text data information output. If no path is explicitly provided, the path is the current directory.
The "-a" option is used to cause a comments field (which includes alarm information) to be generated for each channel's sample in the output matlab file. This option may be used when the "-m" option is also specified. The "-a" option will cause the output matlab file to grow significantly larger.
The "-A" option is used to cause a timestamp field containing an ASCII representation of each channel's sample time to be generated in the output matlab file. This option may be used when the "-m" option is also specified. The "-A" option will cause the output matlab file to grow significantly larger.
The "-h" option is used to cause usage information to be displayed, indicating the syntax required to invoke the Output Data Generator including a brief description of each option.
The archive_directory_file_name may also be explicitly specified. If it is not specified, the Output Data Generator will open an archive with a file name of "multi_archive.txt" in the directory specified by the ARDATAFILES environment variable (or the current directory if this environment variable is not defined).
The Output Data Generator may be run on any machine that provides access to the data files.
The Monitor is a C program written by Bob Hall for operations use at SLAC. The source and executable for this program is located in the following directory:
The logScanner.txt file contains information regarding the current state of the Engine log file, the Engine itself, and the number of consecutive times that the Engine process was restarted. The exact contents of this file is documented in source file read_log_scanner_file.c. The connectStatus.txt file contains one line for each channel being archived by the Engine. Each line contains a PV name, the timestamp at which the channel first connected or the timestamp associated with the last change in connection state, and the current connection state (connected or disconnected).
A Monitor process uses the EPICS caput utility to write values to an
EPICS channel for indicating the state of the Engine it is monitoring.
The Monitor process monitoring the Archive Engine uses the EPICS channel
PV name specified by the "#define" symbol listed in Table 2-4 below in
the source file log_scanner.h.
The Browser is a small collection of Python scripts. The main script was based on Kay-Uwe Kasemir's demonstration Python script Plot.py. Major modifications were made by Bob Hall at SLAC to develop the original Python scripts into a browser for operations use at SLAC. The name of the main script was changed to archiveBrowser.py. It along with the other two major Browser scripts, ListSelectedChannels.py and ListSelectedFilteredChannels.py, reside in the following EPICS directory at SLAC:
Tkinter, a Python interface to the Tk GUI toolkit, was used to construct the Browser's GUI. The Tkinter Life Preserver web page provides a helpful introduction to Tkinter.
The Browser's GUI also extensively utilized Pmw, a toolkit for building high-level compound widgets in Python using the Tkinter module. The Pmw Documentation web page is an excellent source of Pmw information (particularily its Reference Manuals link). A link located on this web page may be used to download Pmw at no cost. It is installed in the following base directory:
The main widget utilized by the Browser was the BLT graph widget. The Pmw BLT Documentation web page provides an excellent tutorial and reference covering the Pmw interface to this powerful widget. It is very helpful to print the reference information. This web page also contains sources for several Python demonstration scripts utilizing the BLT widget. The BLT widget seems to contain bugs in the area of logarithmic Y axis scaling (attempts to change the Y span when logarithmic scaling is used often results in the crash of a program). Also a program crash can result from attempting to draw a large number of curve data point symbols.
The Browser uses the Channel Archiver Scripting Interface (CASI) feature of the Channel Archiver to open Channel Archiver directory files and retrieve data. CASI allows one to call Channel Archiver LibIO routines from within a Python, Tcl, or Perl script. It utilizes SWIG, software "glue" that allows one to call C or C++ from within a scripting language. SWIG version 1.1-833 was downloaded from the web and utilized. The following EPICS directory contains the CASI makefile (Makefile, a link to Makefile.Unix) used to build a static version of Python containing the CASI interface to the Channel Archiver LibIO routine:
To support the Browser, changes to value.h and casi.cpp were made in this directory. A new routine, getDoubleTime, was added to directly return the time as a double number of seconds to improve performance when retrieving data timestamps.
The Browser makes UNIX system calls to invoke the Archive Manager utility of the Channel Archiver to obtain the archive data time range for channels being plotted and to produce Matlab output. Changes to the Archive Manager utility were necessary to produce Matlab output. New modules matlab.cc, matlab.cpp, and matlab.h were created and existing files main.cpp and Makefile.Host were modified in the following directory:
Probably the the suggestion most likely to be implemented in the future is to be able to specify an upper limit to the amount of memory a user can obtain when using the Browser. This would be done by limiting the number of data points the user can retrieve from within the Browser.
Channel Archiver Browser User's Guide
Oracle Partitioning User's Guide
Tkinter Life Preserver
Pmw BLT Documentation