LCLS Channel Archiver PV Change Scripts Guide

This document describes the use of the scripts used to modify LCLS archiver engine configuration files in response to requests to add archiver PVs (and/or change the archiving method for existing archiver PVs) or remove archiver PVs. It is important to use scripts for these activities to minimize the time processing requests, minimize processing mistakes, and to avoid the occurrence of any archiver PV appearing more than once in archiver engine configuration files.

This document also describes the processing of an EDM screen "request to be archived" file by scripts used to determine the responsible IOC engineer for PVs in these files. PVs in a "request to be archived" file are generally not automatically added to archiver engine configuration files. First, it is necessary to contact the responsible IOC engineer for each PV in a "request to be archived" file to determine whether each PV needs to be archived and, if so, how it is to be archived (the archiving method).

All of the scripts are contained in the /nfs/slac/g/archiver/lcls_pv_changes directory. A copy of the current version of each archiver engine configuration file is also kept in this directory for use when adding archiver PVs (and/or changing the archiving method for existing archiver PVs) or removing archiver PVs.

Adding Archiver PVs and/or Change Existing Archiver PV Archiving Methods

Requests to archive PVs from IOC engineers are currently usually made by sending email to the LCLS archiver administrator. There are plans to later replace this procedure by developing an archiver request GUI. These requests are currently made by listing the PVs to be added in the text of an email or in an ASCII attachment file. The archiving method is often specified in the email to apply to all requested archiver PVs in the request. If PVs in the request are to be archived using more than one archiving method, it is helpful if these requests are in the "standard format" described later (Sonya Hoobler and Dayle Kotturi usually submit all of their add PV requests in this "standard format").

Due to the necessity of reducing the load on the LCLS archiver system, requests for large numbers of additional archiver PVs (especially when there is a potential for generating large amounts of archiver data) should be questioned. The goal is to only archive high priority archiver PVs and to choose appropriate archiving methods to not generate excessive archiver data.

Copying Archiver Request PVs into the LCLS Archiver PV Change Processing Directory

Add or remove requested archiver PVs (perhaps already in "standard format" for add requests) need to be copied from the request email text or an ASCII file attachment from a PC environment to the UNIX directory /nfs/slac/g/archiver/lcls_pv_changes. This can be done using the Microsoft Outlook tools to create a file on a PC drive (e.g., the C drive) and use a facility such as WinSCP3 to copy this file to the /nfs/slac/g/archiver/lcls_pv_changes UNIX directory. It is suggested that a standard file name be used, such as arch_pvs.txt.

Converting an Archiver Add PV Request File to Standard Format

To efficiently process an archiver add PV request file, it should be converted to "standard format" (if the request is not already in this format). "Standard format" refers to an ASCII text file containing a line for each PV with indications of the archiving method and interval. Each line of such a "standard format" file has three fields, separated by white space (e.g., a space or a tab):

  1. The PV name starting in the first column.
  2. The letter "m" or the letter "s" (uppercase "M" or "S" is also acceptable). The letter "m" indicates the PV is to be archived using the "monitor" ("on change") archiving method. The letter "s" indicates the PV is to be archived using the "scan" (or "periodically sampled") archiving method.
  3. For a PV being archived with the "scan" archiving method, the archiving sampling interval in seconds. For the "monitor" archiving method, an estimate of the expected interval in seconds between significant changes (often set to 1 if this is unknown).

The following assumes a list of PVs to be archived all using the same archive method is in file arch_pvs.txt in directory /nfs/slac/g/archiver/lcls_pv_changes:

  1. cd /nfs/slac/g/archiver/lcls_pv_changes
  2. ./convert_pv_list_to_standard_format.pl
    The following prompts will appear:
    Please enter the name of file containing PVs to be archived : arch_pvs.txt
    Please enter the sampling mode (s = scan, m = monitor):
         Enter either "s" if the PVs in the file are to be archived in scan
         mode (sampled periodically) or in monitor mode (sampled "on change").
  3. If the scan mode was chosen, the following prompt will appear:
    Please enter the scan interval in seconds ( for default 1 second):

    Otherwise if the monitor mode was chosen, the following prompt will appear:
    Please enter the expected change interval in seconds ( for default 1 second):

The output file in this directory is pv_list.txt, which is in "standard format".

Generate New Versions of Archiver Engine Configuration Files

After a file containing PVs to be added or whose archiving method may be changed exists in the "standard format" described above, the add_arch_pvs.pl script may be run to generate new versions of archiver engine configuration files reflecting these requested changes. This script discards any lines containing PVs that appeared previously in the file. It also searches all of the existing archiver engine configuration files for occurrences of PVs in the input request file of PVs to be added or whose archiving method may be changed. For each PV name match, a new version of the configuration file containing the PV is made if the input request file indicates the archiving method or interval is different in order to reflect the archiving method and interval in the request file. If there there are any PVs in the input request file that are new, the user is prompted for the name of the archiver engine configuration file to be modified and then is asked to choose then archiver group in this file into which the new archiver PVs will be added.

The following assumes a request file pv_list.txt in "standard format" in directory /nfs/slac/g/archiver/lcls_pv_changes:

  1. cd /nfs/slac/g/archiver/lcls_pv_changes
  2. ./add_arch_pvs.pl
    The following prompt will appear:
    Please enter name of file containing add PVs and associated archiving methods : pv_list.txt
  3. Information will then be displayed regarding the number of unique PVs in the request list, the number of PVs in this list that are already being archived, the number of PVs not currently being archived, and the names of any new versions of archiver engine configuration files that were modified (which would occur if there were PVs in the request list that were already being archived but the archiving method or interval for these PVs were different than those in the current version of the archiver engine configuration files). For example, if 169 unique new archiver PVs are to be added and none are already being archived, the following information would appear:
    pv_list_file = pv_list.txt
    number unique PVs in request list = 169
    number PVs in archiver list = 0
    number PVs not in archiver list = 169
    no files changed
  4. The following prompt then will appear:
    Please enter name of Archiver pv list file to which new archiver PVs will be added :
    Any archiver engine configuration file may be entered. It is recommended to add new archiver PVs to a configuration file that contains existing archiver PVs for the same subsystem. For instance, if new archiver feedback PVs are to be added it is recommended that they be added to configuration file lcls_2-group.xml and at the end of the PVs in the lcls_2_feedback archiver group in this file. However, new archiver PVs may be added to any archiver group in any archiver engine configuration file. Grouping PVs by subsystem does help to minimize the impact of a "duplicate PVs" problem. Also it is best to archive approximately the same rough number of PVs in each archive engine (and to limit the number of PVs archived by any engine) for EPICS Channel Access considerations.
  5. A list of "group numbers" and associated group names for the supplied engine configuration file name will then be displayed. The user will be prompted to select a group number corresponding to the group to which the new archiver PVs will be added (to the end of this list). For example, if the chosen archiver engine configuration file to the prompt above was lcls_2-group.xml and it is desired to add PVs to the lcls_2_feedback group in this configuration file, the following information might appear and the user would select group number 0 in response to the "enter group number" prompt (the group number corresponding to group name lcls_2_feedback):
    pv_list_file = lcls_2-group.xml
    group 0: lcls_2_feedback
    group 1: lcls_2_llrf
    group 2: lcls_2_llrf_slow_pac
    group 3: lcls_2_llrf_vme
    group 4: lcls_2_timing_llrf
    group 5: lcls_2_llrf_pad
    group 6: lcls_2_vme_crate
    group 7: lcls_2_bsoic
    group 8: lcls_2_und
    group 9: lcls_2_und_motor
    Please enter group number (0 to 9) : 0

New versions of configuration files will have the extension ".new". For the example above, a new version of the lcls_2-group.xml file named lcls_2-group.xml.new would be created. There may be multiple new version configuration files generated from invoking the add_arch_pvs.pl script (e.g., including new versions of configuration files that contain PVs in the input request file which specify different values of the archive method or interval than was in the current configuration file).

After invoking the add_arch_pvs.pl script, the next action to take is to move the new version of each configuration file that was generated into the current configuration file. For example:

EXCEPTION: If a configuration file was modified due to PVs in this configuration file matching one or more PVs in the request file and the archive method or interval was different, the modification will create a version with the ".new" extension (e.g., lcls_2-group.xml.new). Also if the current version of this file was specified for also adding new archiver PVs, an additional new version of the configuration file will be generated (e.g., lcls_2-group.xml.new.new). In this case, the response should then be to move the "new new" version of this configuration file that was generated into the current configuration file and to delete the plain "new" version (which was modified to create the "new new" version). For example:

After processing, there should be no files with the ".new" or ".new.new" extensions. All modified versions of the current archiver configuration files in this directory should then be copied to their respective current engine data directories (/arch/lcls/lcls_nn, where "nn" is the single or double digit engine number). Before copying, it is important to retain the two previous versions the configuration file for each such current engine data directory in case changes need to be backed out. Afterwards, the current version of each archiver engine configuration file in the /nfs/slac/g/archiver/lcls_pv_changes directory should have the same contents as its respective copy of the configuration file in its /arch/lcls/lcls_nn current engine data directory. After copying, the engines for modified versions of the configuration files should be restarted to use these new current versions.

Removing Archiver PVs

Occassionally, a request is made to remove archiver PVs. An ASCII text request file containing the name of each PV to be removed should be created (each line in this file containing one PV name). It is not important whether the file contains the name of some PVs which are not currently being archived.

First, the names of the archiver engine configuration files containing the archiver PVs to be removed must be determined. The following assumes the file containing the list of archiver PVs to be removed is named arch_pvs.txt and this file is in the /nfs/slac/g/archiver/lcls_pv_changes directory:

  1. cd /nfs/slac/g/archiver/lcls_pv_changes
  2. ./find_engines_of_remove_pvs.pl
  3. The following prompt then will appear:
    Please enter name of file containing PVs to be removed: arch_pvs.txt
  4. Information will then be displayed including PV names in archiver engine configuration files that match PV names in the input request file. The last information displayed is the most significant: the names of archiver engine configuration files that match PV names in the input request file along with a count of the number of matches for each such configuration file. These configuration file names should be noted so that the remove_pvs2.pl script can next be invoked for each of these files.

The following assumes the file containing the list of archiver PVs to be removed is named arch_pvs.txt:

  1. ./remove_pvs2.pl
  2. The following prompt then will appear:
    Please enter name of Archiver pv list file :
         Enter the name of a previously identified archiver engine configuration file.
  3. The following prompt then will appear:
    Please enter name of file containing PVs to be removed: arch_pvs.txt

A new version of the specified archiver engine configuration file configuration file with the extension ".new" will be created. Repeat the invocation of the remove_pvs2.pl script for each configuration file containing PVs to be removed identified by the previous running of the find_engines_of_remove_pvs.pl script.

Follow the directions explained in the previous "Generate New Versions of Archiver Engine Configuration Files" section after the explanation of running the add_arch_pvs.pl regarding moving newly generated versions of the archiver engine configuration files as well as subsequent actions leading up to the final step of restarting engines for modified versions of the configuration files.

Processing EDM Screen "Request to be Archived" File

LCLS EDM screens have a "request to be archived" feature (by selecting a PV and using the "control-middle mouse" to bring up a "request to be archived" menu item). This feature enables MCC Operations to append PV names to a "request to be archived" file, which should be checked daily by the LCLS archiver administrator. The request archiver PVs in this file should not be automatically added to archiver engine configuration files, however. For most PV names in this file it is necessary to try to determine the responsible IOC engineer for the PV and contact this person to decide whether the PV actually needs to be archived and, if so, the archiving method which should be used. Also if a PV in this request list needs to be archived, the reponsible IOC engineer may be reminded to request similar PVs to also be archived.

The current LCLS "request to be archived" file is located in LCLS productions systems: /u1/lcls/tools/ArchiveBrowser/toBeArchivedList. If this file is not present, this indicates that no new requests for PVs to be archived have been made recently. To process this file:

  1. ssh lcls-builder -l softegr
    Then choose an approprite number from the displayed list (e.g., the number corresponding to rdh).
  2. cd /home/softegr/rdh/to_be_archived_list
  3. ls -alt | more
    Note the highest number (most recent file) having a name of the form "toBeArchivedList_nn". The new "nn" number will be the previous higher number plus 1. It is desired to save all versions of the toBeArchivedList files for possible later reference.
  4. ls -l /u1/lcls/tools/ArchiveBrowser/toBeArchivedList
    If this file exists:
    mv /u1/lcls/tools/ArchiveBrowser/toBeArchivedList toBeArchivedList_nn
    where "nn" is the new "nn" number determined previously. Besides moving this file to the /home/softegr/rdh/to_be_archived_list directory for processing, this command removes the toBeArchivedList file from the /u1/lcls/tools/ArchiveBrowser directory so it will be easy to determine whether new "request to be archived" requests have been made by finding whether a new version of this file exists.
  5. Use the "secure copy" facility (scp) to copy the newly created toBeArchivedList_nn file to the /nfs/slac/g/archiver/lcls_pv_changes directory. For example:
    scp toBeArchivedList_22 rdh@lcls-prod02:/nfs/slac/g/archiver/lcls_pv_changes
  6. In another window, login to a flora machine (for example) as yourself. For example:
    ssh flora -l rdh
  7. In the window where you are logged on to the flora machine, setup Oracle environment variables to enable access to the MCCO Oracle instance. For example:
    source ~rdh/oracle/oracle_env.csh
    setenv TWO_TASK MCCO
    If your default shell is bash rather than csh, source a bash Oracle environment source script rather than a C shell script.
  8. In the window where you are logged onto the flora machine:
    cd /nfs/slac/g/archiver/lcls_pv_changes
  9. Remove the timestamp preceding each PV name in the new toBeArchivedList_nn file.
  10. Invoke a script that generates the bash script ioc_engineer_contacts.bash. The invoked script uses the irmisdb Oracle account in the MCCO Oracle instance to attempt to find the IOC associated with each PV in the toBeArchivedList_nn file. For each PV name for which the associated IOC name was found, the generated ioc_engineer_contacts.bash will contain a "caget" statement for finding the name of the IOC engineer associated with the IOC using the CONTACT field. To generate the ioc_engineer_contacts.bash file:
    ./generate_ioc_engineer_contact_script.pl
    For SLC and photon system PVs, this script will print "No IOC found for PV" followed by the PV name since there is no Controls IOC associated with these PVs. For SLC PVs, contact William Colocho regarding whether (and how) the PV should be archived. For photon PVs, contact both Jim Turner and William Colocho.
  11. In the window where you are logged into the lcls-builder machine and have set the present working directory to /home/softegr/rdh/to_be_archived_list, copy the newly generated ioc_engineer_contacts.bash file to this directory using the scp utility. For example:
    scp rdh@lcls-prod02:/nfs/slac/g/archiver/lcls_pv_changes/ioc_engineer_contacts.bash .
  12. In the window where you are logged into the lcls-builder machine, invoke the newly copied version of the ioc_engineer_contacts.bash file:
    ./ioc_engineer_contacts.bash
  13. The screen output of the ioc_engineer_contacts.bash script indicates the name of the responsible IOC engineer associated with each Controls IOC PV. Contact each IOC engineer listed via email to ask whether the requested archiver PVs for which this person is responsible need to be archived (and, if so, how they should be archived).

Author:  Bob Hall 14-Jul-2010