Archive Engine Usage

The ArchiveEngine program is run from the command line. It requires a configuration file to run. While running, you can view the configuration and status information via a web server that is built into the ArchiveEngine. The ArchiveEngine should be stopped via that web server.

Configuration File

You have to create an ASCII configuration file that lists the channels which you want to archive and specifies how this should be done. The name of this configuration file is passed to the ArchiveEngine on start.
The basic layout of this file is:

  # Comment. Empty lines are also allowed
  <ChannelName>  <Period> [Monitor] [Disable]

A simple example would be this one, it could be called "excas.grp":

  # Channels from excas
  fred 5
  freddy 5
  jane 1
  janet 0.1 Monitor
  

The channel names used in here (fred, ... janet) are served by excas, the example ChannelAccess server which is part of EPICS R3.13.3. You will of course use the names of process variables which are specific to your system.

• The scan period is given in seconds. In this example, fred and freddy are to be archived every 5 seconds while jane is archived every second.
• A Monitor channel is archived on change.
  This option was chosen for "janet". The scan period should then be the estimated rate of change because it is used to allocate internal buffers. Since we happen to know that "janet" is a 10Hz channel, we specify 0.1seconds.
  When many more values arrive than specified in the estimated rate of change, you get "overrun" warning messages from the running ArchiveEngine. This is not necessarily a bad thing: You might have chosen the estiamted rate of change to limit the amount of data that is filling up the disk.
  See Operation and Timing for more on periodic operation vs. monitors.

Start the ArchiveEngine

When called, the ArchiveEngine displays usage information similar to this:

  Version 1.8.2, built Jul 18 2001, 15:09:09

  USAGE:  ArchiveEngine [Options] <config-file> [<directory-file>]

  Options:
        -port <port>                   WWW server's TCP port (default 4812)
        -description <text>            description for HTTP display
        -log <filename>                write logfile
        -nocfg                               disable online configuration

        Default directory-file: 'freq_directory'         
  

Minimally, the engine is therefore started by simply naming the configuration file: ArchiveEngine excas.grp
After collecting some data, the ArchiveEngine will create a "directory file", per default called "freq_directory", as well as data files. See how to use this directory file when retrieving data. Some sites prefer to use another name for this by specifying
ArchiveEngine excas.grp dir.
Now "dir" will be the name of the directory file.
You might get a "cannot create cfg" warning that we can ignore for now, see persistent configuration in the Web Status section. Details on the remaining options:

• -description <description> : set engine's description for the HTTP display
• -log <log-file> : write all messages not only to the console but also to this log file
• -port <port> : run HTTP-server on given port (default: 4812)
• -nocfg : disable the online-configuration screen

The HTTP display options configure the ArchiveEngine's built-in web server. It is described in more detail in the Web Status section.

The archive_active.lck File

You can only run one ArchiveEngine per directory because it creates the directory file and data files in there. When running, this lock file is created. The ArchiveEngine will refuse to run if this file already exists. After shutdown, the ArchiveEngine will remove this lock file. If the ArchiveEngine crashes or is not stopped gracefully by the operating system, this lock file will be left behind. You cannot start the ArchiveEngine again until you remove the lock file, this is a reminder for you to check the cause of the improper shutdown and maybe check the data files for corruption.

More than one ArchiveEngine

Note that you can run multiple ArchiveEngines on the same computer. But they must be

1. be in separate directories, see archive_active.lck
2. use a different TCP port number for the built-in web server

In practice this means that you have to create different directories on the disk, one per ArchiveEngine, and in there run the ArchiveEngines with different "-p <port>" options.

Stop the ArchiveEngine

While the ArchiveEngine can be stopped by pressing "CTRL-C" or using the equivalent "kill" command in Unix. But the preferred method is via the built-in web server.: Use any web browser and point it to

http://<host where engine is running>:<port>/stop

e.g. http://localhost:4812. The port number defaults to 4812 or can be changed with the "-p" option of the ArchiveEngine.

More Options for the Configuration File

• A Disable channel will disable the current group of channels. This is useful when e.g. all channels related to a Power Supply are put in one group. When the PS is switched off, the group can be disabled in order to avoid archiving the irrelevant noise. As an example, your "powersup.grp" configuration file could be:

        # Power Supply Config
        # Most channels scanned every 30 seconds.
        # Disable this group when "PS_Off" != zero
        PS_Off 1 Monitor Disable
        PS_Setpoint 30
        PS_Readback 30
        PS_Current 30
        

  Usually you will have several "group" config files, each with a disabling channel, and include all those from one master file:

        # My Master file "main.grp"
        # Run ArchiveEngine with this one as the config. file.
  
        !group powersup.grp
        !group vaccuum.grp
        # and many more...
        

  Please note that the disabling channel will stop the group when it has a non-zero value. When more complicated expressions are needed to determine the "disable" state, a CALC record or other database logic can be used on the IOC.
• # ......
  Comment, ignored just like empty lines.
• !group <filename>
  Read given file and treat all channels in there as a group. This allows the grouping necessary for the "Disable" mechanism. Note that his mechanism is not hierarchical! You cannot really define subgroups within groups but only one flat list of groups. When using any of the following configuration options, they will not be restricted to the current group file. Instead, they are valid from the point on where they appear in the config file.
• !write_period <seconds>
  Time between writes to disk
  (default: 30 sec.)

The following options are more advanced. You do not have to use them, they can cause more confision than good if you are not careful.

• !default_period <seconds>
  When no period is provided, this value will be used. (default: 1 sec.)
• !ignored_future <hours>
  Defines "too far in the future" as "now + ignored_future". Time stamps beyond that are ignored.
  Details: For strange reasons, the Engine sometimes receives values with invalid time stamps. The most common example is a "Zero" time stamp: After an IOC reboots, all records have a zero time stamp until they are processed. For passive records, as commonly used for operator input, this time stamp will stay zero until someone enters a value on an operator screen or via a safe/restore utility. The Engine cannot archive those values because the retrieval relies on the values being sorted in time. A zero time stamp does not fit in.
  Should an IOC (for some unknown reason) produce a value with an outrageous time stamp, e.g. "1/2/2035", another problem occurs: Since the archiver cannot go back in time, it cannot add further values to this channel until the date "1/2/2035" is reached. Consequently, future time stamps have to be ignored. (default: 6h)
• !get_threshold <seconds>
  Channel Access offers two options for getting values:
  Issue a CA 'get' request or establish a monitor/callback for each change.
  If a channel is requested to be archived on change with the "Monitor" flag, the ArchiveEngine will obviously use the monitor mechanism in ChannelAccess. For scanned channels, there are now still two options left: For a channel that is to be scanned very slowly, for example every 60 seconds, it is possibly advisable to use a single "get" request every minute. On the other hand, if a channel is requested to be archived every second, it might be better not to issue "get" requests at 1Hz but instead to establish a monitor mechanism so that the ArchiveEngine automatically receives any change in the value. Every second it will then write the most recent value.
  It is unclear where to draw this line: When archiving every 10 seconds, it might be good to use the monitor mechanism of ChannelAccess to reduce the network load unless that channel is really changing at 60Hz, in which case we would receive updates at 60Hz over the network only to through all the values away except for a single one every 10 seconds. The ArchiveEngine does not attempt to guess how your IOCs are configured, instead there is the "get_threshold" configuration parameter.
  If the sample rate of a channel is below the 'get_threshold' or the 'Monitor' flag was specified, a Channel Access monitor will be used. This means that the Engine is passive. It receives every change from the IOC and then decicides whether to save that as a new sample or not. If the sample period is greater than the 'get_threshold', a 'get' call will be used for each sample. In this case the ArchiveEngine actively requests a value each time a channel is to be sampled. (default: 20 sec.)
• !file_size <hours>
  A new data file is created after the given hours. Almost: a new data file is created if a new buffer needs to be created and the file size in hours has been exceeded. So in reality, an existing data file can be added to for quite a while after the "file_size" time has expired. And note that you must not remove, rename or otherwise manipulate any of the data files! See the ArchiveManager description for information on how to deal with directory and data files. There is little reason for users to change this parameter. In priciple, you might get slightly better performance by increasing it, so that you create fewer but bigger data files. (default: 24 hours).
• !buffer_reserve
  To buffer the samples between writes to the disk, the engine keeps a memory buffer for each channel.
  The size of this buffer is
    buffer_reserve * write_period/scan_period
  Since writes can be delayed by other tasks running on the same computer as well as disk activity etc., the buffer is bigger than the minimum required: buffer_reserve defaults to 3.
  If you get "override" messages, you should
  1. Check if the offending channel is tagged Monitor. In that case the period estimate might bee too large.
  2. Increase buffer_reserve. Since this is global for all channels and will dramatically increase the memory consumption of the ArchiveEngine, it should be a lst resort or limited to an ArchiveEngine that handles only few channels.

---

ChannelArchiver Manual