OdcNtupleMaker

Note: Starting with the release 14 (online releases 10.x) the date format accepted by OdcNtupleMaker was changed following the changes in BdbTime class which is used for date/time representation in BaBar. Date/time format and input are still in a kinda messy state in BaBar, beware of various issues, such as differences between platforms and daylight savings problems. (Look closely at the examples below and you'll get wat I mean.)

The OdcNtupleMaker retrieves data for specified channels from the ambient database and writes them to an ntuple file, either HBOOK or ROOT; or formats the data in the tabular representation.

How to run OdcNtupleMaker

OdcNtupleMaker normally comes from releases, so all you need is to define correct PATH and LD_LIBRARY_PATH to run it. Usually it is enough to change to test release or production area and issue 'srtpath' command followed by either 'ir2boot' or 'ambientboot' depending on which federation you want to access.

The OdcNtupleMaker runs in non-lock mode and it does not interfere with the Archivers.

Current policy for the data access is to use ir2boot only for most recent data. There are daily sweeps of ambient data from ir2boot to ambientboot, so if you need data older than one day you better use ambientboot from SCS hosts.

SYNTAX

  OdcNtupleMaker [options] [inputFile [outputFile]]

OPTIONS

General options:

-h

Print help message and exit

-v

Produce more verbose printout, can be used for debugging. Several -v options can appear on the command line, each option increases noise.

-q

Reduce noise level. Several -q options can appear on the command line, each option decreases noise.

Data extraction options:

-b time

Defines begin time of the scanning interval. If present overrides the value specified in input file.

-e time

Defines end time of the scanning interval. If present overrides the value specified in input file.

-C

Fetch the live data from the CORBA server. Live data live in the files written by OdcArchiver before they are saved in the database. Files are typically saved once in two hours, so if you need to see very recent data you might need this option. Do not use this option outside IR2.

-L name

Name of the CORBA server which provides access to the live data. Default name is "live".

Data processing options:

-r time

Defines reference time. The time is stored in ntuple as a number of seconds since this reference time. Default value for reference time is 1Jan1999 00:00:00 local time.

-c

Do not save the status codes in straight or sampling modes.

-s seconds

Defines length in seconds of sampling period in averaging mode. If present overrides the value specified in input file.

-n

Do not average data in case -s option or input file specify non-zero seconds. Instead take only one "good" measurement from the sample.

-d delta

Seconds of "delta" time, default value is 0.01. Delta time is used in "straight" mode to reduce the data noise from too frequent updates. This feature is experimental, and it can lead to data loss, use it on your own risk.

-i number

overrides the value used to represent invalid data. Default value is -9999.

Data output options:

-l size

Gives record length for HBOOK files, default value is 1024.

-R

Write ROOT ntuple. Default is to write HBOOK ntuple.

-T

Write ASCII tables. Options -R and -T are mutually exclusive.

-t string

Defines column separator for ASCII tables format. Default is to use TAB.

-H

Suppresses headers with -T options.

-f format

Defines format for printing floating point numbers with -T option. Default format is "%.15g".

PARAMETERS

inputFile

the name of input file for OdcNtupleMaker (see below). If the name is equal to "-" then the input is read from standard input.

outputFile

the name of the output file with ntuple. If you use -T option to ptoduce ASCII table output you can specify "-" as the name of the output file in which case the table will be printed on standard output (you can use -q option to suppress other information printed on stdout too.)

You can omit last or both file names, in this case they will default to "-" (this can be done only with -T option.)

INPUT FILE

The input file contains begin and end times for the scanning interval, the sampling seconds and the list of channels to be written in the ntuple. The times and sampling seconds can also be specified in the command line, and the command line overrides the values in the input file. The first three lines in the input file which define begin/end time and sampling seconds are optional, you can omit all three of them and specify their values with command line options.

The format of the file is:

• First line: Start time (year month day hour minute second)
• Second line: End time (year month day hour minute second)
• Third line: Number of seconds for sampling (see below)
• The rest of the file is the list of entries (channels), one entry per line, no empty lines. You can break the entries into few separate lines using backslash (\) as the last character on the line.

There are three types of entries in this list, ordinary entry, time entry, and so called "__all__ entry".

Ordinary entries

The format of the ordinary entry is:

	 <column name> <channel ID> [option list]

Where:

• column name - name of the column in ntuple
• channel ID - channel identifier

Channel identifier can be specified in two formats. If you know the EPICS name of the channel you can give it together with the system where this channel is defined in the form sys/EPICS. Here is an example entry in config file which uses this format:

	 lumi cen/CEN:PEP:BBR:LUMVAL

Another way to specify the channel identifier is to give its "path name" in the ambient tree. The ambient tree is defined by the archiver configuration file and can be displayed in the OdcArcConfigEdit or AmbientExplorer. Path name also starts with the system name, but it must be preceded by a slash. Here is an example entry in config file which uses path format:

	 tempr /cen/EHmon/WEATHERmon/Temperature

The option list for the entry can contain following options:

-t type

Redefines type of the ntuple column, type can be one of "bool", "int", "float", "double", "<number>:<number>". By default all columns in ntuple are stored as float type (REAL*4). With this option you can select different type for the column. The "<number>:<number>" notation means that the column type will be int with the range given by two numbers. Examples: "-t bool", "t -1:62".

-x expression

Defines an expression to exclude measurements. If this expression evaluates to true for some measurement then this measurement will not be put into ntuple. By default the -x expression is empty and no exclusions are made.

-i expression

Defines an expression to "invalidate" measurements. If this expression evaluates to true for some measurement then this measurement will be replaced with the measurement with status 300 and value given in -i program option (-9999 by default). By default the -i expression equals to "stat.sev == 3" or that given in __all__ entry.

Expressions in -x and -i options have C++-like syntax. For the description of the expression language consult the PrsParser/README.html file. Expressions containing spaces should be surrounded by quotes ("). OdcNtupleMaker defines following identifiers which can be used in these expressions:

• stat.sev - equals to the status severity of current measurement (0-3)
• stat.code - equals to the status code of current measurement
• stat - the same as stat.sev*100+stat.code
• value - equals to the value of current measurement

For a list of status and severity codes see the ODCstatus.html file. Examples of the expressions:

stat.sev==3
"stat != 114"
"stat in [114,115,120] or stat.sev == 3"

Time entry

The format of the time entry is:

	 Time -t type

__all__ entry

The format of the __all__ entry is:

	 __all__ [option list]

The option list for this entry can contain the same options as the ordinary entry. This entry redefines default values of options for all ordinary entries which appear after the __all__ entry and which do not define their own options. There may be many __all__ entries in the file. Example of the config file:

entry1 sys/CHANNEL1   -x "stat > 199"
entry2 sys/CHANNEL2
__all__               -t bool   -x "stat == 114"
entry3 sys/CHANNEL3
entry4 sys/CHANNEL4   -t int
__all__               -x ""  -i "stat.sev > 1"
entry5 sys/CHANNEL5
entry6 sys/CHANNEL6   -t 0:63
__all__               -t double
entry7 sys/CHANNEL7
entry8 sys/CHANNEL8   -x "stat > 99"

With this input file the entries will have following attributes:

SYNOPSIS

OdcNtupleMaker scans all the specified channels in given time interval and extracts all their measurements. Sometimes OdcNtupleMaker can introduce its own "measurements" to signal some bad condition such as database problems or something inconsistent. These measurements all have status code 300 and value given with -i option (-9999 by default). Here is the list of conditions when these "invalid" measurements appear:

• object handle is invalid. This could be a database-related problem when ambient object was not properly stored,
• channel is not defined in the given time interval,
• number of measurements is zero in specific channel. This is a logic failure, usually all channels have at least one measurement.

The times stored in the ntuple are the differences (in seconds) of the time of the measurements from a reference time. The default reference time is 1Jan1999 00:00:00 in local time zone and can be changed with the -r option. The reference time is written in the title of the ntuple.

The OdcNtupleMaker runs in three different modes depending on the value of sampling seconds parameter in the input file (or given with -s option) and -n option.

Straight mode

In the first "straight" mode, which is used when sampling seconds are 0, the individual measurements are extracted from database and stored in ntuple. There is a slight complication of this simple matter because measurements in different channels appears at different times. In the old version of the program the most "frequent" channel was used to set the time "ticks", and for every time tick the information from all channels was written into ntuple as a single row. In the newer version of the program this has been replaced with new algorithm. Now any measurement in any channel is counted as a "tick". This increases the volume of information written into ntuple (sometimes significantly) but resolves many significant problems in the old design. The picture below summarizes differences between old and new designs.

The newest update (V00-01-05 or later) of the OdcNtupleMaker has one more feature - "delta" time. In the "straight" mode when the algorithm finds a new "tick" on the above picture it also looks ahead for "delta" time to see if there are measurements in other channels which are not further than delta seconds. If there are then all these measurements are included in the same "tick" and ntuple row. The time of this row will be the latest time of all measurements which were included.

In the straight mode along with the values of the measurements the status information is also written (unless option -c is given to the program). For every channel there are two columns in ntuple -- one with the name given in the input file, another is with the same name prepended with "st" string. Second column contains the status of the measurement. For a list of EPICS status and severity values see the ODCstatus.html file.

Because the status information is also stored in ntuple straight mode does not use "invalidate" expressions and does not invalidate any measurements (but invalid values and statuses can appear in ntuple anyway due to the reasons described above.) "Exclude" expressions are used in this mode to throw away any measurements which evaluate to true.

Averaging mode

Averaging mode is turned on when the sampling seconds value is non-zero and option -n is not set. The whole time interval is divided into samples, each sample has a length given in the input file or with the -s option. In each sample the program tries to find the mean value of all measurements. More precisely the algorithm works in following way:

• For every measurement the channel's -x and -i expressions are evaluated. If -x expression evaluates to true then the measurement is thrown away. Otherwise if -i expression evaluates to true then the measurement is replaced with invalid measurement, which has status 300 and value of -i option. (Note that there could be invalid measurements even prior to evaluation of any expressions). Any measurement which is not invalid at this point is considered "good".
• If there is at least one good measurement in the sample, then the value stored in the ntuple is an average of all these good measurements.
• If there are no good measurements in the sample, but there are invalid measurements then the value stored in the ntuple is taken from the -i option.
• Otherwise if there are no any measurements in the sample, the value is taken from the previous sample. This could be either the last good measurement if that sample has any good measurements or the value from -i option if previous interval has only invalid measurements. If previous interval does not have any measurements this logic is extended to interval preceding it.
• In the case when the first sample has no measurements its value is determined by the last measurement before this sample. Sometimes there may not be such measurements, so the invalid value will be taken in such case.

Sampling mode

Sampling mode is turned on when the sampling seconds value is non-zero and option -n is set. The whole time interval is divided into samples, each sample has a length given in the input file or with the -s option. In each sample the program tries to find only one measurement and puts it into ntuple. The algorithm works in following way:

• For each sample its mean time is calculated, this time will be stored in ntuple.
• The -x and -i expressions are evaluated exactly as in averaging mode.
• From all good measurements only one selected which is closest to the sample mean time. If there are measurements below mean time, then one with the highest time selected, otherwise the first one with the time after mean time.
• If there are no good measurements in the sample, but there are some invalid measurements then the value stored in the ntuple is taken from the -i option.
• Otherwise if there are no measurements in the sample, the same logic as in averaging mode works.

In sampling mode the status codes are also saved in separate column for each channel. You can disable saving of the status codes with -c option.

EXAMPLES

Example 1

% cat test1.dat
herI cen/CEN:PEP:HER:I
lumin cen/CEN:PEP:BBR:LUMVAL
dchI dch/DCH::HV:PS_1:IMON_0
% OdcNtupleMaker -b "1Oct2002 00:00:00" -e "1Nov2002 00:00:00" -s 30 test1.dat test1.hbook
Begin time: Tue Oct 01 01:00:00 2002 (local time) 0 ns
End time:   Fri Nov 01 00:00:00 2002 (local time) 0 ns
Ref time:   Fri Jan 01 00:00:00 1999 (local time) 0 ns
Sampling:   30
Number of variables: 3
    heri: cen/CEN:PEP:HER:I     -t void -i "stat.sev==3"
    lumin: cen/CEN:PEP:BBR:LUMVAL       -t void -i "stat.sev==3"
    dchi: dch/DCH::HV:PS_1:IMON_0       -t void -i "stat.sev==3"
[..............................]
89280 entries stored in ntuple
finishing Objectivity transaction
< HepTuple >
    Title: "Ref 01/01/99 00:00:00 from 10/01/02 01:00:00 to 11/01/02 00:00:00 every 30 sec"
    Index: 1
    Directories: //PAWC (memory) => //HBF000 (disk)
    Column list:
        Time:R*4             @ default
        heri:R*4             @ default
        lumin:R*4            @ default
        dchi:R*4             @ default

closing ntuple file
%

Example 2

% cat test1.dat
herI cen/CEN:PEP:HER:I
lumin cen/CEN:PEP:BBR:LUMVAL
dchI dch/DCH::HV:PS_1:IMON_0
% OdcNtupleMaker -R -s 0 -b "1Oct2002 00:00:00" -e "5Oct2002 00:00:00" test1.dat test2.root
Begin time: Tue Oct 01 01:00:00 2002 (local time) 0 ns
End time:   Sat Oct 05 01:00:00 2002 (local time) 0 ns
Ref time:   Fri Jan 01 00:00:00 1999 (local time) 0 ns
Sampling:   0
Number of variables: 3
    heri: cen/CEN:PEP:HER:I     -t void -i "stat.sev==3"
    lumin: cen/CEN:PEP:BBR:LUMVAL       -t void -i "stat.sev==3"
    dchi: dch/DCH::HV:PS_1:IMON_0       -t void -i "stat.sev==3"
[..............................]
21411 entries stored in ntuple
finishing Objectivity transaction
RooTuple: ntuple Ref 01/01/99 00:00:00 from 10/01/02 01:00:00 to 10/05/02 01:00:00 has 7 columns.
Complete printout follows:
******************************************************************************
*Tree    :ntp1      : Ref 01/01/99 00:00:00 from 10/01/02 01:00:00 to 10/05/02 01:00:00 *
*Entries :    21411 : Total =          559850 bytes  File  Size =      43666 *
*        :          : Tree compression factor =  12.82                       *
******************************************************************************
*Br    0 :Time      : Time/F                                                 *
*Entries :    21411 : Total  Size=      79990 bytes  File Size  =      24459 *
*Baskets :       10 : Basket Size=       8000 bytes  Compression=   3.27     *
*............................................................................*
*Br    1 :heri      : heri/F                                                 *
*Entries :    21411 : Total  Size=      79990 bytes  File Size  =       1097 *
*Baskets :       10 : Basket Size=       8000 bytes  Compression=  72.92     *
*............................................................................*
*Br    2 :lumin     : lumin/F                                                *
*Entries :    21411 : Total  Size=      79960 bytes  File Size  =        980 *
*Baskets :       10 : Basket Size=       8000 bytes  Compression=  81.59     *
*............................................................................*
*Br    3 :dchi      : dchi/F                                                 *
*Entries :    21411 : Total  Size=      79990 bytes  File Size  =       8050 *
*Baskets :       10 : Basket Size=       8000 bytes  Compression=   9.94     *
*............................................................................*
*Br    4 :stheri    : stheri/I                                               *
*Entries :    21411 : Total  Size=      79970 bytes  File Size  =       1059 *
*Baskets :       10 : Basket Size=       8000 bytes  Compression=  75.51     *
*............................................................................*
*Br    5 :stlumin   : stlumin/I                                              *
*Entries :    21411 : Total  Size=      79980 bytes  File Size  =       7031 *
*Baskets :       10 : Basket Size=       8000 bytes  Compression=  11.38     *
*............................................................................*
*Br    6 :stdchi    : stdchi/I                                               *
*Entries :    21411 : Total  Size=      79970 bytes  File Size  =        990 *
*Baskets :       10 : Basket Size=       8000 bytes  Compression=  80.78     *
*............................................................................*

closing ntuple file
%

Example 3

% OdcNtupleMaker -T -t " " -q -s 0 -b "1May2003 00:00:00" -e "2May2003 00:00:00" << EOI 
Time -t double
herI cen/CEN:PEP:HER:I
lumin cen/CEN:PEP:BBR:LUMVAL
dchI dch/DCH::HV:PS_1:IMON_0
EOI 
Ref 01/01/99 00:00:00 from 05/01/03 01:00:00 to 05/02/03 01:00:00
Time heri lumin dchi stheri stlumin stdchi
136598400 878.48876953125 3278.05029296875 3.72000002861023 0 114 0
136598401.06627 878.48876953125 3264.4345703125 3.72000002861023 0 114 0
136598403.054596 878.48876953125 3258.71435546875 3.72000002861023 0 114 0
136598405.033532 878.48876953125 3258.71435546875 3.59999990463257 0 114 0
136598405.054588 878.48876953125 3271.92724609375 3.59999990463257 0 114 0
136598408.069576 878.48876953125 3265.32080078125 3.59999990463257 0 114 0
136598411.033518 878.48876953125 3265.32080078125 3.65999984741211 0 114 0
136598411.051231 878.48876953125 3263.951171875 3.65999984741211 0 114 0
136598414.049559 878.48876953125 3274.98876953125 3.65999984741211 0 114 0
[....]
136684796.040429 821.874389648438 1950.23522949219 0.0399999991059303 0 114 0
136684796.070429 821.874389648438 1928.80456542969 0.0399999991059303 0 114 0
136684797.065426 821.874389648438 1901.49255371094 0.0399999991059303 0 114 0
136684798.040424 826.196166992188 1901.49255371094 0.0399999991059303 0 114 0
136684799.040422 829.553039550781 1901.49255371094 0.0399999991059303 0 114 0