Data File Directories and Formats
Connie Logg
June 16, 2002

Overview

As the bandwidth tests are run, the entire output is logged to a file (in the directory "rawdata", one file per node per day. At the end of a "run" a program called post-test-processing-script is invoked to extract the results from the log files and write them to blank separated files under "plotdata". The extracted data is written to an http accessible directory
http://www.slac.stanford.edu/comp/net/bandwidth-tests/antonia/plotdata for the monitoring node ANTONIA and http://www.slac.stanford.edu/comp/net/bandwidth-tests/pharlap/plotdata for the monitoring node PHARLAP.
The structure of the data directory is as follows:
http://www.slac.stanford.edu/comp/net/bandwidth-tests/plotdata/nodename/yyyy_mm/testtype.yyyy_mm_dd

where testtype = alldata | iperf | bbcpmem | bbcpdisk | bbftp | ping
For specific testtype (except for "alldata"), the format of the data file is "date time kilobits/second" for the bandwidth measurements. There may be other information in the data file, but the kilobits/second will always be the 3rd value on the line. For alldata, the kilobits/sec for each test run in a specific "run" is stored in the file. For example: For bbcpmem: 
#date       time      avgKb/s   minKb/s   maxKb/s   streams  winsize   
02/10/2002 01:44:07  140697.6    117972    147456         ?         ?
02/10/2002 03:16:07  142531.2  119626.4    147456         ?         ?
02/10/2002 04:47:16  139769.6   90798.4    144384         ?         ?
02/10/2002 06:17:17    140928         0  148945.6         ?         ?
02/10/2002 07:43:29  141896.8  117907.2    147456         ?         ?
02/10/2002 09:10:56  143901.6    106852  151400.8         ?         ?
02/10/2002 10:45:26  141865.6   88344.8  147751.2         ?         ?
02/10/2002 12:15:49  141604.8    113936    147456         ?         ?
02/10/2002 13:45:14  143292.8         0    147604         ?         ?
02/10/2002 15:16:48  140493.6         0    147456         ?         ?
02/10/2002 16:47:16  140871.2  106785.6    147456         ?         ?
02/10/2002 18:19:08  142807.2   71890.4    147456         ?         ?
02/10/2002 19:45:32  138802.4   61593.6    151040         ?         ?
02/10/2002 21:18:54  132634.4     93464    147456         ?         ?
02/10/2002 22:48:45  137930.4         0    147456         ?         ?
and for alldata: 
#date       time     pingloss     iperf   bbcpmem  bbcpdisk     bbftp    pingAverage 
06/15/2002 00:57:53         0    332459  198940.8  227318.4 145000.00        25                    
06/15/2002 02:58:23         0    351773  179276.8  198233.6 120000.00        25                    
06/15/2002 04:58:42         0    361428  268459.2  211066.4 130000.00        25                    
06/15/2002 06:57:37         0    355030  242472.8  137680.8 149000.00        25                    
06/15/2002 08:58:39         0    335487    188288  143342.4 135000.00        25                    
06/15/2002 11:00:18         0    271448  123754.4    106148 142000.00        26                    
06/15/2002 12:59:24         0    366265  182067.2  185077.6  87500.00        26                    
06/15/2002 15:00:14         0    357611  175543.2  141485.6 132000.00        25                    
06/15/2002 17:00:43         0    251398    110800  131360.8         ?        25                    
06/15/2002 19:00:33         0    370616  192841.6  273751.2 112000.00        25                    
06/15/2002 20:58:20         0    375238    278068  140273.6 186000.00        25                    
06/15/2002 23:00:09         0    270836  126626.4         ? 181000.00        25
A "?" indicates that there is no data for the specific test during the run which occurred on date & time. For the future, any additional information concerning a run will be added to the end of each line.

The plotdata/alldata directory is a temporary directory where the data being used for the 28 days is stored. This is the used by the histogram, scatterplot, and the other analysis code which performs analysis for the current "28 days". The "28" is specified by the $DAYSTOANALYZE parameter in the Monitoring Host Configuration File.

Passive Monitoring Data Directories

At SLAC we run netflow on our DMZ switch. This generates netflow records for all traffic coming into or leaving SLAC. This data requires upwards of 20 hours of processing time (on an old Sparc 5). The extraction of the bandwidth tests netflow data is done late every night for the previous day.

Currently there are 3 directories which are created for the passive data:

It is important to note that the passive throughput is actually calculated in 3 different ways at this time. This is because we are not really sure how it should be calculated. The first was is the
sum_over_all_streams_I [megabits(I)/activetime(I) ].
The next way is the
(sum_over_all_streams_I [megabits(I)])/((sum_over_all_streams_I [activetime(I)])/number_of_streams),
and the third way is
(sum_over_all_streams_I [megabits(I)])/(total_active_time_of_the_transfer). Note this is not the sum of the active times.
Which column is which is indicated in the heading in thefiles.
The files in $REPORTSDIR/act-pass have the following format: 
#startdate starttime   sum_mbits/sec  totalbytes    #S  activetime  totalbits/ port&protocol totalbits/ p-epoch   act-mbits  a-epoch
#                                                                   totaltime/#s             totaltime
06/15/2002   00:58:18     341.24       434161012     8      81528   340.82      5000 IPERF   315.75   1024127898    332.46  1024127898
06/15/2002   02:58:47     361.37       459862128     8      81544   360.92      5000 IPERF   334.45   1024135127    351.77  1024135127
06/15/2002   04:59:08     370.22       471928044     8      81595   370.16      5000 IPERF   377.54   1024142348    361.43  1024142347
06/15/2002   06:58:02     364.03       463774592     8      81504   364.17      5000 IPERF   337.29   1024149482    355.03  1024149482
06/15/2002   08:59:04     343.97       437954656     8      81470   344.04      5000 IPERF   350.36   1024156744    335.49  1024156744
06/15/2002   11:00:55     278.55       354717908     8      81562   278.34      5000 IPERF   283.77   1024164055    271.45  1024164053
06/15/2002   12:59:50     374.64       478532756     8      81845   374.20      5000 IPERF   382.83   1024171190    366.26  1024171189
06/15/2002   15:00:39     366.81       467297964     8      81597   366.52      5000 IPERF   373.84   1024178439    357.61  1024178438
06/15/2002   17:01:18     257.72       328305868     8      81504   257.80      5000 IPERF   262.64   1024185678    251.40  1024185678
06/15/2002   19:00:58     380.34       483829988     8      81418   380.32      5000 IPERF   351.88   1024192858    370.62  1024192857
06/15/2002   20:58:45     384.42       489494128     8      81489   384.44      5000 IPERF   391.60   1024199925    375.24  1024199925
06/15/2002   23:00:45     277.75       356056824     8      82097   277.57      5000 IPERF   284.85   1024207245    270.84  1024207245

p-epoch = epoch seconds when netflow detected the transfer began
a-epoch = epoch seconds when the active test began
activetime = active time in milliseconds of the flow; seconds=activetime/1000

Please provide Feedback to: Connie Logg