• Install The data base with mysql_install_db to create the database directories (path picked from /etc/my.cnf). Change the permissions on the files within the mysql directory
  chown mysql:mysql /scratch/mysql/mysql/*.

If you installed the MySQL RPMs as suggested in MySQL Install, a mysql server should already be running. You may confirm using ps -ax | grep -i mysql. If not, start the mysql server via mysqld_safe &. If this fails, look at the file datadir/monHost.err for hints as to what the problem is. Note the monHost is from /etc/iepm.cnf and datadir is from /etc/my.cnf. In the case of the example defined in this writeup, the error file will be /scratch/mysql/nettest.slac.stanford.edu.err. The most common reason for failure is that the ownership on the mysql directories is not set to mysql:mysql.

• Set the mysql root password on the data base with the commands:
  mysqladmin -u root password "new password"

• Verify that all permissions are correct. For example:
  ls -l /scratch
  drwxrwxr-x 9 mysql mysql 4096 Aug 22 16:34 mysql/
  cd /scratch/mysql
  Note: anything owned by "mysql:mysql" is created by the mysql processes and the other accounts must be owned by "updateAcct:acctGroup"
  ls -l

  drwxrwxr-x	2	cal	sf	8192	Sep 30 10:07	data
  -rw-rw----	1	mysql	mysql	25088	Jun 7 12:04	ib_arch_log_0000000000
  -rw-rw----	1	mysql	mysql	5242880	Aug 22 16:34	ib_logfile0
  -rw-rw----	1	mysql	mysql	5242880	Jun 7 12:04	ib_logfile1
  -rw-rw----	1	mysql	mysql	10485760	Aug 20 17:15	ibdata1
  drwx------	2	mysql	mysql	4096	Jul 12 14:49	iepm/
  drwxrwxr-x	2	cal	sf	4096	Jul 6 14:41	keepalives/
  drwxrwxr-x	2	cal	sf	16384	Sep 29 23:58	logs/
  drwx------	2	mysql	mysql	4096	Jun 7 11:44	mysql/
  srwxrwxrwx	1	mysql	mysql	0	Aug 22 16:34	mysql.sock=
  -rw-rw----	1	mysql	mysql	6193	Aug 22 16:34	nettest.err
  -rw-rw----	1	mysql	mysql	5	Aug 22 16:34	nettest.pid
  drwxr-xr-x	2	cal	sf	4096	Jul 6 15:45	pids/
  -rw-r--r--	1	cal	sf	9	Jun 7 13:37	pws
  drwx------	2	root	root	4096	Jun 7 11:44	test/

• Now create the iepm data base. Invoke mysql with the following command:
  mysql -u root -p (and enter the mysql root password when prompted)
mysql> create database iepm;

  See the MySQL Manual for information on granting access rights. The following sections have some examples.

• Grant access rights to the administive accounts installAcct and updateAcct; both found in /etc/iepm.cnf. For this example the installAcct is "cal". Note that the updateAcct and installAcct must be valid unix accounts. The following grant commands must be followed for each of your two administrative accounts.
  mysql> grant all on mysql.* to cal@localhost identified by 'password' with grant option;
  mysql> grant all on mysql.* to cal@[dataBaseHost] identified by 'password' with grant option;
  mysql> grant all on iepm.* to cal@localhost identified by 'password' with grant option;
  mysql> grant all on iepm.* to cal@[dataBaseHost] identified by 'password' with grant option;
  mysql> grant all on iepm.* to cal@localhost identified by 'password' with grant option;
  mysql> grant all on iepm.* to cal@[dataBaseHost] identified by 'password' with grant option;
  mysql> grant file on *.* to cal@localhost;
  mysql> grant file on *.* to cal@[dataBaseHost];

• Grant readonly privileges for the readonly account.
  mysql>grant select on iepm.* to 'readonly'@'%';

• Grant read, write, and update access to your users who need those privileges on the system
  mysql> grant select, insert, update on iepm.* to 'mysqlacctname'@'localhost' identified by "a_password"; or for another system:
  mysql> grant select, insert, update on iepm.* to 'mysqlacctname'@'othersystemname' identified by "a_password";
• You can review grants by issuing the following commands:
  mysql> show grants for 'mysqlacctname'@'othersystemname';
  mysql> show grants for 'mysqlacctname'@'localhost';
• Be sure to grant access for select to your web servers, for example:
  mysql> grant select on iepm.* to apache;
  mysql> grant select on iepm.* to nobody;
  If your web server is a separate machine, you may have to grant access for the specfic machines:
  mysql> grant select on iepm.* to apache@hostname;
  mysql> grant select on iepm.* to nobody@hostname;

**It is suggested that you also grant the above priviledges using the fully qualified machine names as well. For instance, if you want to grant readonly priviledges, on machinex, in addition to this command:
mysql>grant select on iepm.* to 'readonly'@'%';
The following should also be executed:
mysql>grant select on iepm.* to 'readonly'@'machinex.slac.stanford.edu';
mysql>grant select on iepm.* to 'readonly'@'machinex';

Install the Source Code

If you used the automatic installer above, IEPM-BW source code should already be installed at the iepmSrcDir specified in /etc/iepm.cnf. If not, download v3src to /myspace/v3src and unzip and untar it to install it.

Updating the Database

update-NODES-info-from-csv

$iepmSrcDir/mysql/update-NODES-info-from-csv
The user should run `$iepmSrcDir/fetch-table-data-into-csv -t NODES` before running this code. 'fetch-table-data-into-csv' will return the path to where the current CSV file can be found: ex. '/tmp/NODES.csv'.
Updates necessary to the NODES table should be added into this CSV file. Once updates are entered, the update-NODES-from-csv script is executed and reads the CSV file and uses this file to update the NODES table in the current database on the system this script is being called from.

Configure the Data base

You are now ready to create the database tables with $iepmSrcDir/mysql/createthetables "installAcct_Password". createthetables is installed with illegal comments so that it can not accidently be run. You will need put a "#" sign in front of the comment lines in order to run it. After you have run it please remove the "#" so that someone does not accidently destroy your data base. Issue the command, for example: /myspace/v3src/mysql/createthetables mypassword It must be called with the installAcct Mysql password.

Run the script $iepmSrcDir/report-table-structure. This script creates a web page with the details of the Database structure. You will be allowed to verify that you have established the correct table structure.

Load the Configuration Information into the IEPM Data Base Tables

There are CGI scripts for loading the system control database tables, however, first an entry must be made for the monitoring host. There are two ways to do this. It can be done with a MySQL command or loaded by creating csv (comma separated files) and loading them with $iepmSrcDir/mysql/[load|update|insert]--from-csv -f filename***. A small sample of nodes is included in the file $iepmSrcDir/mysql/example1-NODES.csv which can be substituted for 'filename' in the execution of 'load-table-from-csv'. NOTE: '##' in the first column will stop the loading of monhost data into the database. Any nodes that are being entered as monitoring hosts must be labeled iepm-bw.domain name. You must load the data base tables in the following order.
1. Load the monitoring host into the NODES table. The NODES table is used to define all the nodes (including the monitoring host). The list of NODES fields is described in APPENDIX A. The absolutely required fields are: (aliasv4, ipv4name, ipv4addr, active,mastertime2run, and domain). To do this in MySQL, log into MySQL with the installAcct or updateAcct as defined in /etc/iepm.cnf configuration file. You must have granted priviliges to these accounts.
   mysql -u updateAcct -p enter password when prompted
mysql>insert into NODES (aliasv4,ipv4name,ipv4addr,active,mastertime2run,
    ->domain,gnuplotpath, ploticuspath)
    -> values (aliasName, monHost, monHostIp, 1, 20, domain,
    ->'/usr/bin/gnuplot','/usr/bin/ploticus');
   where aliasName, monHost, and monHostip are as they are defined in /etc/iepm.cnf. Domain is like "slac.stanford.edu" for the example as derived from monHost in /etc/iepm.cnf.
   After making the entry in the NODES table, issue the following command to get the nodeId for use in loadingthe MONHOST table.
   select * from NODES where aliasv4 = 'aliasName'; You will need this nodeId for the next step.

2. Load the monitoring host information in the MONHOST table. The absolutely required fields are: (nodeId, srcDir, reportsDir, webPath, siteName, daysToAnalyze, dataToAnalyze, cgiPath, and cgiDir). I will use as an example, the information relevant to the node defined in /etc/iepm.cnf. To do this in MySQL, log into MySQL with the installAcct or updateAcct as defined in /etc/iepm.cnf configuration file. You must have granted priviliges to these accounts.
   mysql -u updateAcct -p enter password when prompted
mysql>insert into MONHOST (nodeId, srcDir, reportsDir,
    ->webPath, siteName, daysToAnalyze, dataToAnalyze, cgiPath, and cgiDir)
    -> values (nodeId, '/myspace/v3src', '/var/www/html',
    ->'http://nettest.slac.stanford.edu', 'SLAC Monitoring site', 28,
    ->'ping:avg=w i lt 1;ping:min=w i lt 0;
    ->abwe:avdbcap=w p lt 5;abwe:avabw=w p lt 12;abwe:avxtr=w p lt 8',
    ->'http://nettest.slac.stanford.edu/cgi-bin', '/var/www/cgi-bin'); .
3. To verify this setup, run report-toolspecs to see that the probe specs are correct.
***Options for the updating scripts at $iepmSrcDir/mysql/[load|update|insert]--from-csv include:

update-NODES-info-from-csv
update-PLOTSPECS-info-from-csv
load-MONHOST-from-csv
insert-PLOTSPECS-info-from-csv

Install the IEPM-BW CGI Scripts and Icons

Most of the source will auto install when the source is extracted from the source tar file. However there are a few things which will need manual installation.

• The IEPM icons need to be saved where they are accessible from the web pages. They are distributed in the v3src subdirectory iepmicons. The web directory path will be reportsDir/aliasName/, so in the case of our example, the command is cp -pr /myspace/v3src/iepmicons /var/www/html/node6.slac.stanford.edu/.
• Installing the cgi-bin scripts
  There are 3 important cgi-bin scripts which provide a mechanism for loading the data base. Since they are dependent on the configuration of the system, they must created after the host node is defined in the NODES and MONHOST tables.
  • Create the cgi-bin script to add a node to the NODES table
    cd $iepmSrcDir/cgi-bin
make-add_node_html
    will install the cgi-bin script for adding a node into the cgi-bin directory. You must also make sure to copy the $iepmSrcDir/cgi-bin/add_node.pl file to cgi-bin directory
  • Create the cgi-bin script to update a node in the data base
    cd $iepmSrcDir/cgi-bin
make-update_node_html
    will install the cgi-bin script for updating a node into the cgi-bin directory.
  • Input the specifications for defining the probes
    cd $iepmSrcDir/cgi-bin
make-toolspec_html
    will install the cgi-bin script for adding probe specifications to the TOOLSPEC data base
  • cp $iepmSrcDir/cgi-bin/v3graphem.cgi /var/www/cgi-bin/

Installation Startup Checkout

When the data base and system have been configured, before you load the crontab and start it all up, run the following checks:

1. $iepmSrcDir/monhost-getcfg - If this fails because of a DBI error, please verify your /etc/iepm.cnf file is correct and that the appropriate grants have be setup in MySQL for access by these accounts and the monitoring host node. Also verify the following manually by going into MySQL:
   1. select * from NODES where aliasv4='$aliasName'; - Note the nodeid number.
   2. select * from MONHOST; - Verify that the nodeid for the monhostid entry is in fact the same as fetched in the previous statememt. See monhost-getcfg for more information.
2. $iepmSrcDir/get-nodelist - This should return a list of the target hosts and what tools they are monitoring. See get-nodelist for more information.
3. Generate the gnuplot test plot: $iepmSrcDir/utils/make-gnutest-plot. This will create a test plot of the gnuplot colors and characters for referencing when deciding what colors to plot the data in. This will be accessible from the master IEPM-BW page.
4. Execute $iepmSrcDir/keep-em-alive. This code starts the IEPM-BW test daemons for testing purposes
5. Test that the system will work properly by executing $iepmSrcDir/schedule-load. This execution loads one pass into the schedule table. The daemons should begin working at this point and will store the acquired data in the mysql/data directory.
6. Once $iepmSrcDir/schedule-load is working and data is going into the mysql/data directory, test loading the collected data into the database by using the load scripts.

   The current load data scripts available are:
   load-iperf-data
   load-pathchirp-data
   load-pathload-data
   load-ping-data
   load-tlaytcp-data
   load-trace-data
   

   Based on the tests that you are running, and the data seen in mysql/data, execute $iepmSrcDir/load-[type]-data to simulate uploading the collected into the database. This also proves that the scripts will work when the system is automated. NOTE: The above steps can also be simulated by running $iepmSrcDir/schedule-load -T [type], verifying that the data from this execution can be seen from mysql/data, followed by executing the associated $iepmSrcDir/load-[type]-data script to simulate the data upload to the database.
7. Start the $iepmSrcDir/load-datad daemon.

Database Overview

The iepm database contains all the configuration and control information for the running of tests. It also is where the probe results are stored. The following are the current tables in the IEPM database.

Table Name	Purpose	Notes
NODES	Contains node definitions	Each and every node used in the system must be in this table
MONHOST	Contains the monitoring host specifications	All hosts logging data to the database, must be defined in this table.
TOOLSPECS	Contains the specification for any tests to be run on a regular basis
SCHEDULE	Contains 1 entry for each test everytime it is run except for ABWE
ABWEDATA	Storage for the ABWE data
PINGDATA	Storage for the PING data
ROUTEDATA	Storage from the TRACEROUTE data
ROUTENO	Storage from the route number information	Each unique route is given a route number. They are stored in this table.
BWDATA	Storage for bandwidth tests data	Not yet used
COMMENTS	Storage for comments
GROUPS	Storage for names of grouped nodes	Not yet used
NODEGROUP	Contains the list of nodes and the groups that they belong to	Not yet used

Tests are scheduled by inserting the specification for a test into the SCHEDULE table. The daemon $iepmSrcDir/load-scheduled runs continually. It checks the TOOLSPEC table to see what probes are due for scheduling. Probes should not be run more than every 10 minutes. Note that ABWE has its own daemon "abwed" and runs as often as it can.

The currently available background tests are abwe, ping, and trace(route).

Ping and Traceroute each have a daemon ("pingd" and "traced") which run continually and periodically inquire the SCHEDULE table for any tests that they are to run. They write the data files from their probes into the data directory $dataBaseDir/data, the data base directory (datadir in /etc/my.cnf and dataBaseDir in /etc/iepm.cnf). The scripts $iepmSrcDir/load-ping-data and $iepmSrcDir/load-trace-data are called from the database loading daemon $iepmSrcDir/load-datad and they load the data into the mysql data base.

The abwe daemon "abwed" runs in the backgound doing abwe tests as frequently as possible. Abwe has its own directory where it stores the data. load-abwe-data called from $iepmSrcDir/load-datad loads the awbe data into the database.

A note here on why it is done this way...Originally I had separate data base loading daemons for each type of data, and they loaded the data into the MySQL database concurrently. At high probe rates, they got into lockup conditions where the data base was locked and no one could do anything. Entering data into the data base is actually quite quick, so I wrote $iepmSrcDir/load-datad which calls the loading scripts sequentially and thus avoids the lock up problem.

There are two other IEPM-BW related directories in datadir: logs and keepalives. Any messages (errors or otherwise) are written to the appropriate log file for each of the daemons. The logs are maintained by a script called "copylogs". It renames "yesterday's" logs and starts a new one for "today". Currently 30 days worth of logs are kept. Each daemon also has a file in datadir/keepalives which is updated after each pass. This is checked by a crontab entry "keep-em-alive" every 10 minutes to make sure that the daemons are alive and well. If they are not, they are killed and retarted.

The Crontab

There is a crontab file which contains the calls to load the scheduled tests, maintain the log files and data base backup files, backup the data base, and generate reports on the system. This crontab should be set up under the updateAcct in /etc/iepm.cnf

## copy and date the logs for the day
58 23 * * * /afs/slac/package/netmon/bandwidth-tests/v3src/copylogs /scratch/mysql/logs
## run keepalive check
5,15,25,35,45,55 * * * * /afs/slac/package/netmon/bandwidth-tests/v3src/keep-em-alive
## copy and date the backups
0 1 * * * /afs/slac/package/netmon/bandwidth-tests/v3src/copylogs /nfs/slac/g/net/iepm-bw/bandwidth-tests/mysql-backup
## back up the data base
15 0 * * * /afs/slac/package/netmon/bandwidth-tests/v3src/backup-iepm-mysql-database /nfs/slac/package/netmon/bandwidth-tests/mysql-backup
## generate data logging report
15 2 * * * /afs/slac/package/netmon/bandwidth-tests/v3src/report-logging
## run the regular analysis
23 * * * * /afs/slac/package/netmon/bandwidth-tests/v3src/post-test-processing-script

Notes:

58 23 * * * /afs/slac/package/netmon/bandwidth-tests/v3src/copylogs /scratch/mysql/logs
performs the log maintenance on the /scratch/mysql/logs directory. Note that the path "/scratch/mysql/" to "logs" is the dataBaseDir path in /etc/iepm.cnf

5,15,25,35,45,55 * * * * /afs/slac/package/netmon/bandwidth-tests/v3src/keep-em-alive
checks that all the daemons are running. If one is not, then it tries to kill it and restarts it. If the system has been installed as indicated above, when the crontab is loaded, it will start the daemons automatically.

0 1 * * * /afs/slac/package/netmon/bandwidth-tests/v3src/copylogs /nfs/slac/g/net/iepm-bw/bandwidth-tests/mysql-backup
performs the maintenance on the backup copies.

15 0 * * * /afs/slac/package/netmon/bandwidth-tests/v3src/backup-iepm-mysql-database /nfs/slac/package/netmon/bandwidth-tests/mysql-backup
backs up the iepm mysql database

15 2 * * * /afs/slac/package/netmon/bandwidth-tests/v3src/report-logging
generates the report of what was logg for the past 2 weeks

23 * * * * /afs/slac/package/netmon/bandwidth-tests/v3src/post-test-processing-script
analyzes the data, graphs it, and generates the results page.

The database is backed up nightly to /nfs/slac/g/net/iepm-bw/bandwitch-tests/mysql-backup. If you need to restore it, try "/usr/bin/mysql iepm < full_name_of_backup". 'copylogs' is also used to maintain the backups. They are kept for 30 days at this point.

IEPM-BW System Scripts

These sections describe the scripts which are a part of the IEPM-BW system code.

MySQL Database Loading Scripts

The scripts in this section can be used to insall and load the database initially.

$iepmSrcDir/mysql/createthetables "installAcct_password"

This script creates the MySQL database tables. It deletes the existing tables and recreates them. It is used for installation and development purposes only and is the definitive table description. If you modify your tables, please keep this up to date. If you have to restore your database from a backup, do NOT use this to create the tables.

load-aliases -f aliasfilename -h aliasName

This script was used by us at SLAC to load our IEPM-BW Version 2 alias file into the MySQL database. It loads various tables in the MySQL database. It is presented here as an example of sql statement and how they can be written in perl to load the database tables. Remember, every node, whether it is a monitoring host or target host must have an entry in the NODES table. Note that all fields in the NODES table for a node need not be defined, however any that are going to be needed, must be defined (more on this later).

load-table-from-csv -f filename

This loads a comma separated value (csv) flat file into the appropriate data base. The first line must be the tablename to be loaded, The next line are the comma separated field names, and then following that is the comma separated data, representing on record(row) per line.

Note that the fields can be empty. For example:

TABLENAME=NODES
#aliasv4,aliasv6,ipv4name,ipv4addr,ipv6name,ipv6addr,domain,contactName,contactEmail,contactphone,contactfax,contactaddr,hosttype,siteurl
node6.slac.stanford.edu,,nettest4.slac.stanford.edu,134.79.240.28,,,slac.stanford.edu,Connie Logg,cal@slac.stanford.edu,650-926-2879,650-926-3329,SLAC; 2575 Sand Hill Road; Menlo Park; ca; 94024,,
node1.slac.stanford.edu,,nettest5.slac.stanford.edu,134.79.240.50,,,slac.stanford.edu,Connie Logg,cal@slac.stanford.edu,650-926-2879,650-926-3329,SLAC; 2575 Sand Hill Road; Menlo Park; ca; 94024,,

Note that these files can be created with a spreadsheet program like excel.

MySQL Database Retrieval Scripts

These scripts use used to fetch information out of the mysql data base.

monhost-getcfg

$iepmSrcDir/monhost-getcfg [-h monitoring host alias] reads the monitoring host information out of the MONHOST and NODES mysql tables. This is a particularly useful tool for validating that the data base and configuration files are set up correctly. If called from the unix command line, it prints its results on the terminal. The '-h' is optional and defaults to aliasName. For example: $iepmSrcDir/monhost-getcfg will return a string of the form:
("SRCDIR='/afs/slac/package/netmon/bandwidth-tests/v3src'", "REPORTSDIR='/var/www/html'", "PRIVATEDIR='/nfs/slac/g/net/iepm-bw/bandwidth-tests/litetest/slaconly'", "WEBPATH='http://nettest2.slac.stanford.edu'", "DAYSTOANALYZE='30'", "DATATOANALYZE='ping:avg=w i lt 1;ping:min=w i lt 0;abwe:avdbcap=w p lt 5;abwe:avabw=w p lt 12;abwe:avxtr=w p lt 8'", "MASTERTIME2RUN='10'", "GNUPLOTPATH='/usr/local/bin/gnuplot'", "PLOTICUSPATH='/afs/slac/package/netmon/bandwidth-tests/ploticus/pl'", "PINGPATH='/bin/ping'", "TRACEPATH='/usr/sbin/traceroute'", "GREPPATH='/bin/grep'", "CGIPATH='http://nettest2.slac.stanford.edu/cgi-bin'", "CGIDIR='/var/www/cgi-bin'","SITENAME='SLAC IEPMLITE'")

It is primarily called from perl scripts by:
@ans= `$iepmSrcDir/monhost-getcfg `;
chop @ans;
#now define the parms
eval("\@parms = $ans[0];");
grep (eval("\$$_;"),@parms);
and the following perl variables are defined.

• $SRCDIR - the code source directory (usually the same as iepmSrcDir)
• $REPORTSDIR - which is the directory path to the top of the directory where the reports and webpages are stored.
• $PRIVATEDIR - which is to hide information that you want ot keep from the user
• $WEBPATH - which is the http string for getting to the top of the report and results directory
• $DAYSTOANALYZE - how many days worth of data to analyze with the report scripts
• $DATATOANALYZE - which is a character string that specifies what data values are to be analyzed and how they are supposed to be plotted. For example: $DATATOANALYZE = 'ping:avg=w i lt 1;ping:min=w i lt 0;abwe:avdbcap=w p lt 5;abwe:avabw=w p lt 12;abwe:avxtr=w p lt 8' indicates to the system that it is to analyze:
  • ping "avg" data and plot it with a gnuplot spec of "w i lt 1"
  • ping "min" data and plot it with a gnuplot spec of "w i lt 0"
  • abwe "avdbcap" data and plot it with a gnuplot spec of "w p lt 5"
  • abwe "avdbw" data and plot it with a gnuplot spec of "w p lt 12"
  • abwe "avxtr" data and plot it with a gnuplot spec of "w p lt 8"
  Note that the value to plot is actually the column heading in the data file. The data fetch scripts are described shortly.
• MASTERTIME2RUN - is the longest in seconds a probe is to be allowed to try to execute
• GNUPLOTPATH - is the path to gnuplot
• PLOTICUSPATH - is the path to ploticus plot command
• PINGPATH - is the path to the ping command
• TRACEPATH - is the path to the traceroute command
• GREPPATH - is the path to the grep command
• CGIPATH - is the http path to the cgi-bin directory
• CGIDIR - is the directory path to the cgi-bin directory
• SITENAME - is the site name from the MONHOST table

get-nodelist

$iepmSrcDir/get-nodelist returns a string which when evaluated, defines an associative array %TESTLIST. For example:
@ans = $iepmSrcDir/get-nodelist
eval("$ans[0]";)
where %TESTLIST=('node2.rhic.bnl.gov','abwe,ping,traceroute','node1.clrc.ac.uk','abwe,ping,traceroute')
Upon evaluation, $TESTLIST{'node2.rhic.bnl.gov'} = "abwe,ping,traceroute" is the tests performed for the particular node. The list of active nodes can be extracted by @nodes = keys(%TESTLIST). This script is used heavily in the code for going through all the nodes.

create-abw-list

$iepmSrcDir/create-abw-list creates the IEPMLITE configuration file for ABWE monitoring processes. It is saved as $iepmSrcDir/config/IEPMLITE.conf. There are 3 columns which are legacy, and can be ignored although they must be provided for parsing purposes. They are "db", "poll", and "shortname". It must be manually stored into the ABWE source directory at: $iepmSrcDir/abw/Conf/IEPMLITE.conf. This is the table the abwed daemon will use to perform abwe measurements. When new nodes are added to this process, this manual store must be repeated, and the daemon must be restarted. The ABWE daemon is started by $iepmSrcDir/abwed IEPMLITE. The abwe_rfl is started by executing $iepmSrcDir/keep-servers-alive. However, the keepalive system generally starts both of these actions and keeps it running, so the issuance of this manual command should not be needed.

fetch-monhosts

return a list of the monitoring hosts which have been defined in the database.

fetch-table-descriptions -t tablename

returns a list of the field names in a table. This can be useful for populating CGI web pages with the list of selectable fields. Tablenames currently include: NODES, MONHOST, PINGDATA, ABWEDATA, ROUTEDATA, TOOLSPECS, SCHEDULE, ROUTENO, BWDATA, COMMENTS

fetch-table-data-into-csv -t tablename

returns a CSV list of all current database entries for the specified tablename. This is used in conjunction with an Update tablename script to easily update missing database data. In particular, fetch-table-data-into-csv -t NODES is run before executing $iepmSrcDir/mysql/update-NODES-info-from-csv to quickly update the NODES table.

fetch-target-hosts -t datatable -h monhost -s starttime -e endtime

"fetch-target-hosts" returns the aliasnames of the hosts monitored by monhost which have data in the data table (PINGDATA, ABWEDATA, ROUTEDATA) for the period of time. It is returned as a list (node1.binp.nsk.su=45,node1.cacr.caltech.edu=45,node1.cesnet.cz=42, node1.desy.de=38,node1.dl.ac.uk=41,node1.ece.rice.edu=43,node1.fnal.gov=39, node1.fzk.de=41,node1.indiana.edu=46,node1.internet2.edu=47,node1.kek.jp=42) where the number after the "=" is the number of points found. "-h" is optional, and if not provided the "aliasName" parameter in /etc/iepm.cnf is assumed.

Database Validation Scripts

These scripts create html pages of the information they fetch from the mysql database. They can be useful for validating and verifying what is loaded in the data base. Note that many need some more work on them. Please note that the links provided here are to a version of the package which is in heavy development, and they may fail. Sorry for any inconvenience. A temporary web page linking to some of the output of these scripts is available.

report-logging

"report-logging" generates a report of all the test and logging activity for a few weeks.

report-nodes

generates a report prints out all the nodes listed as target nodes for the specified monitoring host.

report-monhost

generates a report prints out all the nodes in the monitoring host data base MONHOST.

report-toolspecs

generates a report prints out all the probe specs for the monitoring host specified.

report-toolspecs [-h monitoring-host-alias]

generates a report prints out all the probe specs for the monitoring host specified.

Utility Scripts

traceanal/makegifs -f [character or number] -t number -o output directory - generates a red, white and orange gif image icon for a number or character in the specified output directory. If -t is not provided, -t = -f.

Data Fetching Scripts

These all have the same arguments, and defaults are provided. "-t" is the testtype, and the default is "background"
"-o" is the directory where the output is written, and /tmp is thye default
"-n" is the aliasname of the desired node, but all are fetched if it is not provided.
"-s" is the starttime, either in epoch seconds or standard date format
"-e" is the endtime, either in epoch seconds or standard date format
"-h" is the monitoring host alias name and defaults to the host the code is executed on "-t" is the testype. This is "background" for the normal data acquisition. If "-e" and "-s" are not provided, the data for the last 24 hours is returned. If the scripts are being run on the monitoring host, only -n, -s, and -e are required.

fetch-ping-data [-n aliasname] [-o output_directory] [-s starttime] [-e endtime] [-t testtype] [-h monitoring_host_alias]

The only testtype currently is "background". If "-o" is not provided, /tmp is assumed. "fetch-ping-data" returns the name of the file where the output is stored.

fetch-abwe-data [-n aliasname] [-o output_directory] [-s starttime] [-e endtime] [-t testtype] -d ['to' or 'from']

The only testtype currently is "background". If "-o" is not provided, /tmp is assumed. "fetch-abwe-data" returns the name of the file where the output is stored. There are 3 types of ABWE data stored in the data base. There is the overall throughput measurement (fetch with NO -d parameter), there is the raw measurement from the monitoring host to the target hosts (fetch with '-d to') and the raw measurement from the target host to the monitoring host (fetch with '-d from'). The 'to' data has the string 'abweto' in the name, the 'from' data has the name 'abwefrom' in the name, and the averaged data has the string 'abwe' in the name.

fetch-trace-data [-n aliasname] [-o output_directory] [-s starttime] [-e endtime] [-t testtype] [-h monitoring_host_alias] -a [0 | 1]

The only testtype currently is "background". If "-o" is not provided, /tmp is assumed. "fetch-trace-data" returns the name of the file where the output is stored. The raw traceroute data is also fetched currently, and is stored also in the "-o" directory. Note the -a parameter. The tracelogs are anonomized is -a is not provided or is 1. For the raw tracelogs -a 0 must be specified. The text of the traceroute output in in the file ending with ".tracelog".

fetch-routeno-data -n aliasname [-o output_directory] [-h monitoring host alias]

- Creates a blank separated file of the route numbers and paths for the specified host and target nodes. The file name is returned. Note that -o can be used to specify the directory where the file should be stored.


fetch-routeno-data-html -n aliasname [-o output_directory] [-h monitoring host alias]

- Creates an html page of the route numbers and paths for the specified host and target, The file name is returned. Note that -o can be used to specify the directory where the file should be stored. nodes


Data Display Routines

plotts -p plotspecs -o output_file_name

This generates a plot of the data specified by plotspecs and stores the output in output_file_name.png. "-p" is the specification for what is to be plotted. The format is: filename1:var1=gunplotspec1,var2=gnuplotspec2;filename2:var3=gunplotspec1,var4=gnuplotspec2 For example:

"fetch-ping-data -n node1.cacr.caltech.edu" will fetch the ping data for the last 24 hours for this node
"fetch-abwe-data -n node1.cacr.caltech.edu" will fetch the awbe data for the last 24 hours for this node
Note that ther name of the data files are returned from each statement.
Lets call them "pingfile" and abwefile.

The "-p" option for abwe (plotting xtr and abw) would be:
"abwefile:xtr=w p lt 3,abwefile:abw=w p lt 1";


Useful Code Snipits
utils/process-epoch.pl
converts dates to epoch and epoch times. To use it:

require "$iepmSrcDir/utils/process-epoch.pl";
$startime = toepoch("07/06/2004 09:52:59");  # sets $starttime to 1089132779
$endtime = fromepoch("1089132779");  # sets $endtime to "07/06/2004 09:52:59"

utils/seeperlmods
displays the perl modules that are in located in the perl library on your machine. To use it:

10$iepmSrcdir/utils> seeperlmods
::afs::slac.stanford.edu::package::perl::lib::5.6.1::i386_linux22::.packlist
AFS
Apache::AuthCookie
Apache::DBI
Apache::Session

...

libwww-perl
libxml-perl
mod_perl
11$iepmSrcdir/utils>

Maintenance Scripts
copylogs directory
Copylogs shoule be run after midnight. This script examines the directory passed 
in for files of the form "somename.today". If it finds nay
it saves then as "somename.yesterday" and "somename.yesterday's_date'. It is currently set up to remove
files older than 30 days.

backup-iepm-mysql-database directory
This script creates a backup copy of the iepm mysql database and saves it in the specified directory
with the name of the name "aliasName.iepm.today". This directory can be maintained by "copylogs".

Adding a New Probe
Adding a new probe to the Iepm-BW test is "complex" because one must understand all the aspects of the probe:
What part of the result is relevant
What kind of table stucture is needed to store the results. Can one that is defined already be used 
or does it need a custom table?
What are the characteristics of it performance and how can it be controlled - Does it hang - if so, 
how and how can it be detected and remedied? Does it require a responder on the target? If so how does 
it work and what are its performance characteristics (does it hang..how to fix it)?
How can one get ahold of the results? Can one direct the output to a file, or does one need to "capture" 
the output and write it out to a file.
What kind of info does the probe return to you? Does it return RTT? BW? Packet loss?  Etc.? How does one 
want to display the probe results? Time series (if so of what), scatterplots? Plot if against other specific data? 
Is this a probe that can be run via a daemon asynchronously, or does it need to be synchronized to not 
run at the same time as certain other probes?
Are there customized plots one would like to see of this probe versus others? See:
  http://www.slac.stanford.edu/comp/net/iepm-bw.slac.stanford.edu/html/miscplots/iepm-bw.cacr.caltech.edu.html. 
If so then these specs must be defined in the plotspecs table. 

Each probe requires customized configuration and code. 
The probe configuration information (parameters) is put in the toolspecs table. Each probe is different, 
and the configuration for each node it is targeted to can be different (window sizes, streams, duration, run interval, 
A "do_test" script must be written. This is the script that actually performs the test. It can be modeled 
after the existing ones, but needs customization depending on the probe characteristics.
A "load-test-data" script must be written to pick out the relevant/interesting data returned from the probe 
and store it in the data base. 
The specs for the desired plots must be entered in the plotspecs (miscellaneous plots), datatoanalyze (master 
results page), and toolspecs (individual and Maxims code)  tables once one knows what they are. This is not difficult, 
but one must decide what one wants.
The master results page must be modified to include the probe into (controlled by "datatoanalyse" for each probe.
Possible a fetch-test-data script must be written, unless the probe data fits the pattern of one that is already 
written. I have one data table "bwdata" which hopefully is general enuf now to hold the data from new probes, in 
which fetch-bw-data can be used for most data.

As usual, the devil is in the analysis. It is extremely important as one does not want to hang or crash 
the system, and if the probe has certain problems characteristic of it, one MUST take care of them and recognize them!
Fetching Data from Iepm-BW from 'iepm-resp'
The scripts necessary for performing the data fetches are located in /afs/slac/package/netmon/bandwidth-tests/v3src. From this directory, the script to fetch thrulay, iperf and pathchirp data and it's calling convention is:
fetch-bw-data -n [aliasname] -T [datatype]  -s [start-date] -e [end-date] -o [outputdir]
NOTE: 
The values for -T, [datatype], are tlaytcp, iperf, or pathchirp 
      
-o is NOT required. The script will put the datafile in the /tmp directory if omitted.
 
The scripts return the name of the file which was loaded with the data you requested. 


Fetching Data from Iepm-BW from 'iepm-bw'
To gather data from the Iepm-BW database on 'iepm-bw', follow the following steps:
setenv IEPM_CNF /afs/slac/package/netmon/bandwidth-tests/v3src/config/iepm.cnf.iepm-bw.slac.stanford.edu
Preface your call to the fetch routines vis "/usr/local/bin/perl" 
/usr/local/bin/perl fetch-abwe-data -n iepm-bw.cacr.caltech.edu -s 2/1/05 -e today 


It is necessary to preface this call via /usr/local/bin/perl on this machine.  This has to do with the way SLAC in general provides perl via AFS. The code for Iepm-BW Monitoring on other remote sites includes the preface /usr/bin/perl in the first line for compliance reasons with our collaborators.
Code Documentation
ScriptName($iepmSrcDir)
Description
Options
CalledFrom($iepmSrcDir)
Contact

General Scripts

bw-cleanup

Cleans up hung bwtests processes.
This looks for processes which match 
its list @procs and are running under 
the MNUSER account. It is simplistic. 
Ifthe current hour minus the hour is 
the process is > 1, the the process is
killed.

N/A
post-test-processing-script
Connie Logg

keep-servers-alive

Restarts servers necessary to keep 
testing to target sites up and run-
ning.  This script depends on the 
'$iepmSrcDir/config/servers.alive'
file to know which servers to "keep-alive".

N/A
Crontab
Connie Logg

kill-processes

Looks for processes running under the 
logged in ID and kills them if they are 
more than 'n' minutes old. This is intended 
to clean up after run-bw-tests processes 
that don't terminate normally. The default 
for -t is 10

-p "typestr1,typesrt2" -t n -i [yes|no]
Crontab
Connie Logg

Pingloss Scripts

plot-ping

Executes $iepmSrcDir/get-pingplot-data
to gather the necessary ping data
formatted to include pingloss info.
This script graphs that data showing
moments of packetloss, out of order
or duplicate packets.  The graphs can
be found by clicking
here.
-n aliasname -X x-axis_label -Y y-axis_label
overnight-processing-script
Jerrod D. Williams

get-pingplot-data

Executes $iepmSrcDir/fetch-ping-data
to gather ping data for the past 28
days for each node.  This data is 
formatted to display the results of
tests for out-of-order packets as well
as duplicate packets in the final two 
columns of the output.  Results of its
execution can be found at /tmp/ping-data.

N/A
plot-ping
Jerrod D. Williams



Trouble Shooting Hints
The following are a list of problems which we have run into, what the causes might be and how to
possible fix them.

Main results page:
The summary table has red timestamps even though they are current - This may be caused by the
TOOLSPECS record parameter "runInterval" for the given test not being set or set at too small a value.
If (the current time) - (the last run time) > 2*runinterval, the date and time are set red.

ABWE is not working - 
Check that the node ip address is correct in the NODES table.
    
Run "create-abw-list" to create an updated configuration file
    
Kill abwed
    
It should auto restart in about 15 minutes.

    
 

Data Analysis Scripts

There are data analysis scripts which are used by the iepm-bw system, however, we are attempting to code
them such that they can be used by other users with their own data.
Traceroute analysis for Non-IEPM-BW Trace Data


Decide on the base directory ($iepmSrcDir) where you want to put the data and execution scripts.
Ensure that $iepmSrcDir contains a directory traceanal with the following files/directories:

ampdata : Directory containing Non-IEPM-BW Trace Data with filenames of the form, 
sourcehost-nodename.yyyy_mm_dd
traceanal.pl : Configuration file
traceanal : Trace analysis script
fetch-amp-data : Calls convert-amp on data available in directory 'ampdata' to 
create trace and routenumbers data in a format required for part of traceanal processing
convert-amp : Create trace and routenumbers data in a format required for part 
of traceanal processing. Invoked by fetch-amp-data. 'ampdata' serves as the input 
data directory for this script
fetch-amp-routeno : Fetches route number for a given route using the file 
ampdata/sourcehost-nodename.yyyy_mm_dd.routenumbers
get_as.pl
hop_diff.pl
process-epoch.pl
stutter.pl
makegifs

Update $iepmSrcDir in traceanal.pl to reflect the full path of the chosen base directory. Update other values ($WEBPATH, $REPORTSDIR, etc.) if required.
Usage: 

traceanal -d 8/26/04 -n utah -h alaska -c ./traceanal.pl
traceanal -d 8/26/04 -h alaska -c ./traceanal.pl
This iterates over all nodes specified in %TESTLIST parameter in traceanal.pl

Error and Diagnostic Messages

Error messages are of the form:

• Testing bad write connection call - Sample error message: DBIERR: mysql-errtst: all parameters are not defined - dataBase=iepm, dataBaseHost=nettest2.slac.stanford.edu,updateAcct = ,
• Testing sql write error - Sample error message: DBIERR: mysql-errtst: sql = 'UPDAE NODES set active=0 where aliasv4="node5.clrc.ac.uk" ': You have an error in your SQL syntax. Check the manual that corresponds to your MySQL server version for the right syntax to use near 'UPDAE NODES set active=0 where aliasv4="node5.clrc
• Testing bad read connection - Sample error message: DBIERR: mysql-errtst: Read connect failed (database=iepm,dataBaseHost=nettest2.slac.stanford.edu) - Access denied for user: 'any@nettest2.slac.stanford.edu' (Using password: NO)
• Testing sql read error - Sample error message: DBIERR: mysql-errtst: sql = 'SELECT * from nodes' - Read request failed: Table 'iepm.nodes' doesn't exist"

Note that the message "(Using password: NO)" or "(Using password: YES)" simply tells whether a password is passed to the connect statement. It is not saying what the password was.

MySQL (Upgrade / New Install)

• Shutdown the database
• Remove production crontab and any daemons (including all IEPM-BW daemons)
• Check for existing MySQL RPMs with: rpm -qa | grep -i mysql
• Remove each of the RPMs with: rpm -e rpm_name

• Open MySQL Downloads
• Go to Linux x86 RPM Downloads section
• Download the following files

- Server

- Client

- Libraries and header files