Directory and Data File Structures and Formats
Connie Logg
February 1, 2003

Companion Documentation

The Program Logic Manual
Iperf Information
Bbcp Information
Bbftp Information
Bandwidth Tests Home Page

Overview

If the package is installed as at SLAC, there are two required configuration files. One is the Monitoring Host Configuration File (MHCF), and the other is the Target Node Configuration File (historically known as the ALIASFILE). The ALIASFILE was originally named that because its first function was to provide, for security purposes, a way to have aliases for the real node names of the target nodes. It also contains the parameters for each probe for each node, and other node specific information.

At SLAC, the source code and package documentation are kept in AFS and the raw data and analysis results are kept in NFS. In the Monitoring Host Configuration File, $BASEDIR points to the base of the documentation and source code, and $REPORTSDIR points to the base of the log, data, png, summary data, and html files.

$BASEDIR - Source and Package Documentation
- $BASEDIR/html contains the package specific documentation such as this page and the Program Logic Manual
- $BASEDIR/v2src contains the source code. Note that specific types of analyses have their own subdirectories. For example:
  - $BASEDIR/v2src/histograms is the histogram code
  - $BASEDIR/v2src/timeseries contains the code which creates the timeseries plots and the diurnal analysis
  - $BASEDIR/v2src/scatterplots is the scatterplot code
  - $BASEDIR/v2src/history contains the code which creates the weekly and monthly summary data, plts, and html pages
  - $BASEDIR/v2src/predict contains the code which does predictions
  - $BASEDIR/v2src/failures contains the failure analysis code
  - $BASEDIR/v2src/toolkit contains the code for distributing the bw tests package and migrating to the new formats
  - $BASEDIR/v2src/site-customization contains the code used by the html page creation routines to customize the pages to specifc sites.
  - $BASEDIR/slaconly contains the ALIASFILE. The 'slaconly" string prevents the web server from serving this file to anyone outside of SLAC.
$REPORTSDIR - Data, Png, Analysis, and html directories
- $PRIVATEDIR = $REPORTSDIR/slaconly contains the 'rawdata' directory where the actual response from the various probes is logged one file, per node, per day. This data contains the actual nodes names and therefore, it is recommended that this not be publically available.
- $REPORTSDIR/html contains subdirectories for the html output from the various analysis. For example:
  - $REPORTSDIR/html/diurnal contains the diurnal analysis html pages
  - $REPORTSDIR/html/histograms contains the histogram html pages
  - $REPORTSDIR/html/scatterplots contains the scatterplot html pages
  - etc.
- $REPORTSDIR/png contains the png graphics files in subdirectories by the analyses types as for the html directory
- $REPORTSDIR/csvdata contains the pinger format csv files
- $REPORTSDIR/failanal contains the data files for the failure analysis. These are created by $SRCDIR/failures/make-failanal-html
- $REPORTSDIR/summary-data contains the files for the weekly (4 points per day) and monthly (1 point per day) plots
- $REPORTSDIR/killed_processes contains the history of hung processes killed by bw-cleanup
- $REPORTSDIR/plotdata contains the extract data, one file, per probe, per day. There are also subdirectories for data which require special aggregation and or formatting for analysis.
  - $REPORTSDIR/plotdata/combined contains one file per node of the data currently being displayed by the timeseries plots and which is used for the other analyses. These are created by $BASEDIR/v2src/create-analysis-data which is called after each complete measurement run from $BASEDIR/v2src/post-test-processing-script to create the data to be used for the analyses.
  - $REPORTSDIR/plotdata/diurnal contains the files created by the last invocation of $BASEDIR/v2src/create-diurnal-plots.

The Structure of the $REPORTSDIR/plotdata data files

The data which is extracted from the probes is logged to space separated flat files. Each target node has its own subdirectory, and within those subdirectories the files are subdivided by month. That is:
'$REPORTSDIR/plotdata/nodename/yyyy_mm/testtype.yyyy_mm_dd'
For example: '$REPORTSDIR/plotdata/node1.cacr.caltech.edu/2003_01/bbcpmem.2003_01_15' is the name of the bbcpmem probe data for node1.cacr.caltech.edu on 1/15/3002
The first line of each file provides the format of the data within the file (note that the number of decimal points may differ):

#date          time    Mbps    streams    winsize    minMbps    maxMbps      epoch       real   user    sys   testdate testtime  testepoch
01/29/2003 00:17:04      79          8        256         73         89 1043828224      16.53   0.13   0.74 01/29/2003 00:17:57 1043828277                                
01/29/2003 01:17:05      94          8        256         89         98 1043831825      16.64   0.12   0.93 01/29/2003 01:17:47 1043831867                                
01/29/2003 02:17:04     110          8        256         95        111 1043835424      16.51   0.12   0.98 01/29/2003 02:17:47 1043835467                                
01/29/2003 03:17:04     115          8        256         73        115 1043839024      16.41   0.12   1.20 01/29/2003 03:17:57 1043839077                                
01/29/2003 04:17:04     130          8        256         30        130 1043842624      15.78   0.16   1.24 01/29/2003 04:17:45 1043842665                                
01/29/2003 05:17:04     137          8        256        112        138 1043846224      16.29   0.17   1.21 01/29/2003 05:17:45 1043846265                                
01/29/2003 06:17:03     137          8        256        116        138 1043849823      16.16   0.18   1.20 01/29/2003 06:17:55 1043849875                                
01/29/2003 07:17:04     132          8        256        104        132 1043853424      16.02   0.17   1.07 01/29/2003 07:17:45 1043853465                                
01/29/2003 08:17:04     106          8        256        106        116 1043857024      16.61   0.18   1.06 01/29/2003 08:17:45 1043857065                                
01/29/2003 09:17:03      92          8        256         29         95 1043860623      15.99   0.12   0.84 01/29/2003 09:17:55 1043860675                                
01/29/2003 10:17:03      55          8        256         55         68 1043864223      16.44   0.13   0.58 01/29/2003 10:17:46 1043864266                                
01/29/2003 11:17:03      39          8        256         39         65 1043867823      16.95   0.11   0.39 01/29/2003 11:17:46 1043867866                                
01/29/2003 12:17:06      52          8        256          6         61 1043871426      18.19   0.21   0.52 01/29/2003 12:18:05 1043871485                                
01/29/2003 13:17:04      39          8        256          0         54 1043875024      19.10   0.07   0.55 01/29/2003 13:17:57 1043875077                                
01/29/2003 14:17:04      35          8        256          0         42 1043878624      19.79   0.14   0.36 01/29/2003 14:18:02 1043878682

Note that there are two different time stamps. The time stamps early in the line are the time that the "run" for this node was started. The time stamps at the end of the file are the "exact" time the specific probe was run. The first line is very important, as it tells the analysis code which column which data is in.

Also note that all bandwidth measurements are in Mbps.

The variables $DATATOANALYZE and $DAYSTOANALYZE provide the information to the analysis code as to which data is to be analyzed by the general analysis code. $BASEDIR/v2src/create-analysis-data uses these parameters to create the proper data files. For example: if $DAYSTOANALYZE = 28; and $DATATOANALYZE = "ping.avg=w i lt 1;ping.min=w i lt 0;bbcpdisk.Mbps=w p lt 16; "bbcpmem.Mbps=w p lt 12;bbftp.Mbps=w p lt 5;iperf.Mbps=w p lt 8;bbcpdisk.eff-Mbps;trace.numhops"; then "combined data" file will contain the last 28 days data for the above variables. The combined data file is what is used as input by the standard analysis scripts. Note again, that the first line contains the data format.

#date          time  ping.avg   ping.min     bbcpdisk.Mbps      bbcpmem.Mbps        bbftp.Mbps   iperf.Mbps    bbcpdisk.eff-Mbps  trace.numhops 
01/13/2003 00:35:08        25         25            101.74            100.82             99.09       103.46                86.03             25                
01/13/2003 02:05:15        25         25            118.07            114.85            108.00       114.12                98.07             25                 
01/13/2003 03:35:18        32         25            137.04            136.82            124.78       112.54               112.02             91                 
01/13/2003 05:05:13        25         25            150.40            149.80            136.31       150.60               120.68             26                   
01/13/2003 06:35:08        51         25            158.61            159.16            143.65       158.80               125.96             97                   
01/13/2003 08:05:04        25         25            138.14            132.59            121.63       134.90               112.54             25

Note that in the $DATATOANALYZE string the {probetype.columntitle} pair indicates the data to be included in the file. Now of course the rest of the question is: "What does 'ping.avg=w i lt 1' or 'bbcpdisk.Mbps=w p lt 16' mean?" The characters after the "=" is the gnuplot specification of how the data is to be plotted and identified in the timeseries plots. 'ping.avg=w i lt 1' means plot the ping average with impulses and color 1. 'bbcpdisk.Mbps=w p lt 16' means plot the bbcpdisk Mbps as points with color 16. The gnuplot figure exhibits the types and colors available for png format plots. Note that bandwidth data is expected to be 'Mbps' where M = 10E6.

Please provide to Designing Author: Connie Logg

Directory and Data File Structures and FormatsConnie Logg February 1, 2003

Companion Documentation

Overview

The Structure of the $REPORTSDIR/plotdata data files

Directory and Data File Structures and Formats
Connie Logg
February 1, 2003