Computing at SLAC
Search SLAC      
SLAC Home | Computing Home | Networking Group| Help

About

Installation

Requirements

CGI Installation

Documentation

Results

Original PLM

Installation

It is recommended that the installation be done in the order indicated. As you go through the installation procedure in the order indicated, you will be defining variables for the /etc/iepm.cnf and /etc/my.cnf configuration files. You will need to have root privileges to perform the installation.

  1. Decide which unix accounts you want to be able to perform the installation and database update tasks. These can be the same account, however they must be valid unix accounts. These are the variables installAcct, updateAcct and acctGroup in the /etc/iepm.cnf file. The acctGroup in /etc/iepm.cnf must be set to the unix group the accounts belong to. These must also be configured as valid MySQL accounts with the appropriate privileges, as described below.

  2. Decide on the directory structure where you want to put the code. Please put the src in a subdirectory named "v3src". For example: If I want to put the source under the directory /myspace, the directory would be /myspace/v3src. This will be the variable iepmSrcDir in the configuration file /etc/iepm.cnf.

  3. Decide where you want to put the database. The directory should have at least 3 gigabytes available. (Use df -k to find a suitable mysql partition). In the example below I have chosen the /scratch partition and so the variable dataBaseDir in /etc/iepm.cnf will be /scratch/mysql. In /etc/my.cnf the configuration parameter datadir will also be /scratch/mysql. Allocate the directory with the command mkdir /scratch/mysql.

  4. Add a user "mysql" and a group "mysql" to the system if they do not exist. You must be in root to do this.
    groupadd -g 20000 mysql
    useradd -u 20000 -g mysql mysql

  5. Allocate the directory with "mkdir" while in root
    mkdir /scratch/mysql
    NOTE: You must create and allocate this directory even if logging to a data base on another system. Various IEPM-BW control and system logs are created and stored here.

  6. Set the directory permissions to "mysql:mysql".
    chown mysql:mysql /scratch/mysql

  7. Decide on an aliasName for your monitoring host. This will be called the aliasName in /etc/iepm.cnf. We usually use alias names of the form "node1.slac.stanford.edu".

  8. Allocate the following IEPM-BW control and system log directories in dataBaseDir. They should be owned by the account the system is going to run under. In this example the updateAcct is "cal". If the account "cal" is in account group "sf" (acctGroup in /etc/iepm.cnf), these directories must be owned by "cal:sf". Note that in our example dataBaseDir is /scratch/mysql

    • mkdir /scratch/mysql/data. This is the directory where the data is temporarily logged until it can be loaded into the data base via the appropriate daemon.
    • mkdir /scratch/mysql/keepalives - The keepalive system (for keeping daemons running) stores its flags here. Create the following files with touch to prime the daemon watch script keep-em-alive.
      • touch /scratch/mysql/keepalives/abwed.alive
      • touch /scratch/mysql/keepalives/traced.alive
      • touch /scratch/mysql/keepalives/pingd.alive
      • touch /scratch/mysql/keepalives/bw-synched.alive
      • touch /scratch/mysql/keepalives/pathchirpd.alive
      • touch /scratch/mysql/keepalives/load-datad.alive
      • touch /scratch/mysql/keepalives/load-scheduled.alive
    • mkdir /scratch/mysql/pids creates the directory where the daemon pids are stored.
    • mkdir /scratch/mysql/logs creates the directory where the various IEPM-BW logs are stored. Note that there is a script copylogs which maintains this directory so that it does not grow out of control.

  9. Now change the ownership of all the files and directories created in this section to be owned by the user:group under which the system daemons are going to run. This should not be root. For example, in this case user is "cal" group is "sf"
    chown cal:sf /scratch/mysql/data
    chown cal:sf /scratch/mysql/pids
    chown cal:sf /scratch/mysql/logs
    chown cal:sf /scratch/mysql/keepalives
    chown cal:sf /scratch/mysql/keepalives/*

  10. Create the password file for the updateAcct. With an editor, type the password for the mysql updateAcct defined in /etc/iepm.cnf on one line and save it as dataBaseDir/pws. Set the owner to be the unix updateAcct user.

  11. Configure and save /etc/iepm.cnf
    Example of /etc/iepm.cnf
    # required iepm configuration parameters
    # /etc/iepm.cnf
    # the data base name - must be "iepm"
    dataBase = "iepm"
    # the data base host is the "fully qualified machine name"
    dataBaseHost = "nettest1.slac.stanford.edu"
    # directory for the mysql database, logs, and workspace
    dataBaseDir = "/scratch/mysql"
    # the iepm-bw source
    iepmSrcDir = "/myspace/v3src"
    # the master management account for Mysql (does not have
    to be a unix computer account)
    # This is for granting privileges and managing the Mysql user accounts
    installAcct = "cal"
    # The account for updating database
    updateAcct = "cal"
    # a readonly account for fetching data
    readAcct = "readonly"
    # the unix group for the account
    acctGroup = "sf"
    # the monitoring host name
    monHost = "nettest.slac.stanford.edu"
    # the aliasname for the monitoring host
    aliasName = "node6.slac.stanford.edu"
    # the IP address of the monitoring host
    monHostip = "134.79.243.12"
    mysqlport = "3306"
    #Connie Logg 5/12/04 cal@slac.stanford.edu <----YOUR NAME PLEASE!!

  12. Save a copy of /etc/iepm.cnf with the file name iepm.cnf.aliasName in the directory $iepmSrcDir/config. For example with the above example, a copy of /etc/iepm.cnf would be saved as /myspace/v3src/config/iepm.cnf.node6.slac.stanford.edu. This file is passed to the CGI scripts to provide the necessary configuration information for the CGI's to run.

  13. Configure and save /etc/my.cnf. This provides the information required for the allocation of the data base and related control files.
    Example of /etc/my.cnf
    [mysqld]
    datadir=/scratch/mysql
    socket=/scratch/mysql/mysql.sock
    port=3306
    [mysqld_safe]
    datadir=/scratch/mysql
    socket=/scratch/mysql/mysql.sock
    port=3306
    [mysql.server]
    datadir=/scratch/mysql
    socket=/scratch/mysql/mysql.sock
    port=3306
    [client]
    socket=/scratch/mysql/mysql.sock
    port=3306
    Note that datadir is the same as dataBaseDir in /etc/iepm.cnf.

  • Install the MySQL database. You must have root privileges to install the database. Ensure that /etc/my.cnf is setup correctly, as the installation will use the information for setting up database files.

  • Change the permissions on the mysql directory datadir/mysql/mysql> to mysql:mysql. That is, in this example
    chown mysql:mysql /scratch/mysql/mysql.
  • Now install the data base and bring the mysql system up.
    • Install The data base with mysql_install_db to create the database directories (path picked from /etc/my.cnf). Change the permissions on the files within the mysql directory
      chown mysql:mysql /scratch/mysql/mysql/*.

      • If you installed the MySQL RPMs as suggested in MySQL Install, a mysql server should already be running. You may confirm using ps -ax | grep -i mysql. If not, start the mysql server via mysqld_safe &. If this fails, look at the file datadir/monHost.err for hints as to what the problem is. Note the monHost is from /etc/iepm.cnf and datadir is from /etc/my.cnf. In the case of the example defined in this writeup, the error file will be /scratch/mysql/nettest.slac.stanford.edu.err. The most common reason for failure is that the ownership on the mysql directories is not set to mysql:mysql.

    • Set the mysql root password on the data base with the commands:
      mysqladmin -u root password "new password"

    • Verify that all permissions are correct. For example:
      ls -l /scratch
      drwxrwxr-x 9 mysql mysql 4096 Aug 22 16:34 mysql/
      cd /scratch/mysql
      Note: anything owned by "mysql:mysql" is created by the mysql processes and the other accounts must be owned by "updateAcct:acctGroup"
      ls -l
      drwxrwxr-x 2 cal sf 8192 Sep 30 10:07 data
      -rw-rw---- 1 mysql mysql 25088 Jun 7 12:04 ib_arch_log_0000000000
      -rw-rw---- 1 mysql mysql 5242880 Aug 22 16:34 ib_logfile0
      -rw-rw---- 1 mysql mysql 5242880 Jun 7 12:04 ib_logfile1
      -rw-rw---- 1 mysql mysql 10485760 Aug 20 17:15 ibdata1
      drwx------ 2 mysql mysql 4096 Jul 12 14:49 iepm/
      drwxrwxr-x 2 cal sf 4096 Jul 6 14:41 keepalives/
      drwxrwxr-x 2 cal sf 16384 Sep 29 23:58 logs/
      drwx------ 2 mysql mysql 4096 Jun 7 11:44 mysql/
      srwxrwxrwx 1 mysql mysql 0 Aug 22 16:34 mysql.sock=
      -rw-rw---- 1 mysql mysql 6193 Aug 22 16:34 nettest.err
      -rw-rw---- 1 mysql mysql 5 Aug 22 16:34 nettest.pid
      drwxr-xr-x 2 cal sf 4096 Jul 6 15:45 pids/
      -rw-r--r-- 1 cal sf 9 Jun 7 13:37 pws
      drwx------ 2 root root 4096 Jun 7 11:44 test/

    • Now create the iepm data base. Invoke mysql with the following command:
      mysql -u root -p (and enter the mysql root password when prompted)
      mysql> create database iepm;

      See the MySQL Manual for information on granting access rights. The following sections have some examples.

    • Grant access rights to the administive accounts installAcct and updateAcct; both found in /etc/iepm.cnf. For this example the installAcct is "cal". Note that the updateAcct and installAcct must be valid unix accounts. The following grant commands must be followed for each of your two administrative accounts.
      mysql> grant all on mysql.* to cal@localhost identified by 'password' with grant option;
      mysql> grant all on mysql.* to cal@[dataBaseHost] identified by 'password' with grant option;
      mysql> grant all on iepm.* to cal@localhost identified by 'password' with grant option;
      mysql> grant all on iepm.* to cal@[dataBaseHost] identified by 'password' with grant option;
      mysql> grant all on iepm.* to cal@localhost identified by 'password' with grant option;
      mysql> grant all on iepm.* to cal@[dataBaseHost] identified by 'password' with grant option;
      mysql> grant file on *.* to cal@localhost;
      mysql> grant file on *.* to cal@[dataBaseHost];

    • Grant readonly privileges for the readonly account.
      mysql>grant select on iepm.* to 'readonly'@'%';

    • Grant read, write, and update access to your users who need those privileges on the system
      mysql> grant select, insert, update on iepm.* to 'mysqlacctname'@'localhost' identified by "a_password"; or for another system:
      mysql> grant select, insert, update on iepm.* to 'mysqlacctname'@'othersystemname' identified by "a_password";

    • You can review grants by issuing the following commands:
      mysql> show grants for 'mysqlacctname'@'othersystemname';
      mysql> show grants for 'mysqlacctname'@'localhost';

    • Be sure to grant access for select to your web servers, for example:
      mysql> grant select on iepm.* to apache;
      mysql> grant select on iepm.* to nobody;
      If your web server is a separate machine, you may have to grant access for the specfic machines:
      mysql> grant select on iepm.* to apache@hostname;
      mysql> grant select on iepm.* to nobody@hostname;

      • **It is suggested that you also grant the above priviledges using the fully qualified machine names as well. For instance, if you want to grant readonly priviledges, on machinex, in addition to this command:
      mysql>grant select on iepm.* to 'readonly'@'%';
      The following should also be executed:
      mysql>grant select on iepm.* to 'readonly'@'machinex.slac.stanford.edu';
      mysql>grant select on iepm.* to 'readonly'@'machinex';

      Install the Source Code

      If you used the automatic installer above, IEPM-BW source code should already be installed at the iepmSrcDir specified in /etc/iepm.cnf. If not, download v3src to /myspace/v3src and unzip and untar it to install it.

      Updating the Database

      update-NODES-info-from-csv

      $iepmSrcDir/mysql/update-NODES-info-from-csv
      The user should run `$iepmSrcDir/fetch-table-data-into-csv -t NODES` before running this code. 'fetch-table-data-into-csv' will return the path to where the current CSV file can be found: ex. '/tmp/NODES.csv'.
      Updates necessary to the NODES table should be added into this CSV file. Once updates are entered, the update-NODES-from-csv script is executed and reads the CSV file and uses this file to update the NODES table in the current database on the system this script is being called from.

      Configure the Data base

      You are now ready to create the database tables with $iepmSrcDir/mysql/createthetables "installAcct_Password". createthetables is installed with illegal comments so that it can not accidently be run. You will need put a "#" sign in front of the comment lines in order to run it. After you have run it please remove the "#" so that someone does not accidently destroy your data base. Issue the command, for example: /myspace/v3src/mysql/createthetables mypassword It must be called with the installAcct Mysql password.

      Run the script $iepmSrcDir/report-table-structure. This script creates a web page with the details of the Database structure. You will be allowed to verify that you have established the correct table structure.

      Load the Configuration Information into the IEPM Data Base Tables

      There are CGI scripts for loading the system control database tables, however, first an entry must be made for the monitoring host. There are two ways to do this. It can be done with a MySQL command or loaded by creating csv (comma separated files) and loading them with $iepmSrcDir/mysql/[load|update|insert]--from-csv -f filename***. A small sample of nodes is included in the file $iepmSrcDir/mysql/example1-NODES.csv which can be substituted for 'filename' in the execution of 'load-table-from-csv'. NOTE: '##' in the first column will stop the loading of monhost data into the database. Any nodes that are being entered as monitoring hosts must be labeled iepm-bw.domain name. You must load the data base tables in the following order.
      1. Load the monitoring host into the NODES table. The NODES table is used to define all the nodes (including the monitoring host). The list of NODES fields is described in APPENDIX A. The absolutely required fields are: (aliasv4, ipv4name, ipv4addr, active,mastertime2run, and domain). To do this in MySQL, log into MySQL with the installAcct or updateAcct as defined in /etc/iepm.cnf configuration file. You must have granted priviliges to these accounts.
        mysql -u updateAcct -p enter password when prompted
        mysql>insert into NODES (aliasv4,ipv4name,ipv4addr,active,mastertime2run,
            ->domain,gnuplotpath, ploticuspath)
            -> values (aliasName, monHost, monHostIp, 1, 20, domain,
            ->'/usr/bin/gnuplot','/usr/bin/ploticus');
        where aliasName, monHost, and monHostip are as they are defined in /etc/iepm.cnf. Domain is like "slac.stanford.edu" for the example as derived from monHost in /etc/iepm.cnf.
        After making the entry in the NODES table, issue the following command to get the nodeId for use in loadingthe MONHOST table.
        select * from NODES where aliasv4 = 'aliasName'; You will need this nodeId for the next step.

      2. Load the monitoring host information in the MONHOST table. The absolutely required fields are: (nodeId, srcDir, reportsDir, webPath, siteName, daysToAnalyze, dataToAnalyze, cgiPath, and cgiDir). I will use as an example, the information relevant to the node defined in /etc/iepm.cnf. To do this in MySQL, log into MySQL with the installAcct or updateAcct as defined in /etc/iepm.cnf configuration file. You must have granted priviliges to these accounts.
        mysql -u updateAcct -p enter password when prompted
        mysql>insert into MONHOST (nodeId, srcDir, reportsDir,
            ->webPath, siteName, daysToAnalyze, dataToAnalyze, cgiPath, and cgiDir)
            -> values (nodeId, '/myspace/v3src', '/var/www/html',
            ->'http://nettest.slac.stanford.edu', 'SLAC Monitoring site', 28,
            ->'ping:avg=w i lt 1;ping:min=w i lt 0;
            ->abwe:avdbcap=w p lt 5;abwe:avabw=w p lt 12;abwe:avxtr=w p lt 8',
            ->'http://nettest.slac.stanford.edu/cgi-bin', '/var/www/cgi-bin'); .
      3. To verify this setup, run report-toolspecs to see that the probe specs are correct.
      ***Options for the updating scripts at $iepmSrcDir/mysql/[load|update|insert]--from-csv include: 
      update-NODES-info-from-csv
update-PLOTSPECS-info-from-csv
load-MONHOST-from-csv
insert-PLOTSPECS-info-from-csv

      Install the IEPM-BW CGI Scripts and Icons

      Most of the source will auto install when the source is extracted from the source tar file. However there are a few things which will need manual installation.

      • The IEPM icons need to be saved where they are accessible from the web pages. They are distributed in the v3src subdirectory iepmicons. The web directory path will be reportsDir/aliasName/, so in the case of our example, the command is cp -pr /myspace/v3src/iepmicons /var/www/html/node6.slac.stanford.edu/.
      • Installing the cgi-bin scripts
        There are 3 important cgi-bin scripts which provide a mechanism for loading the data base. Since they are dependent on the configuration of the system, they must created after the host node is defined in the NODES and MONHOST tables.
        • Create the cgi-bin script to add a node to the NODES table
          cd $iepmSrcDir/cgi-bin
          make-add_node_html
          will install the cgi-bin script for adding a node into the cgi-bin directory. You must also make sure to copy the $iepmSrcDir/cgi-bin/add_node.pl file to cgi-bin directory
        • Create the cgi-bin script to update a node in the data base
          cd $iepmSrcDir/cgi-bin
          make-update_node_html
          will install the cgi-bin script for updating a node into the cgi-bin directory.
        • Input the specifications for defining the probes
          cd $iepmSrcDir/cgi-bin
          make-toolspec_html
          will install the cgi-bin script for adding probe specifications to the TOOLSPEC data base
        • cp $iepmSrcDir/cgi-bin/v3graphem.cgi /var/www/cgi-bin/

      Installation Startup Checkout

      When the data base and system have been configured, before you load the crontab and start it all up, run the following checks:

      1. $iepmSrcDir/monhost-getcfg - If this fails because of a DBI error, please verify your /etc/iepm.cnf file is correct and that the appropriate grants have be setup in MySQL for access by these accounts and the monitoring host node. Also verify the following manually by going into MySQL:
        1. select * from NODES where aliasv4='$aliasName'; - Note the nodeid number.
        2. select * from MONHOST; - Verify that the nodeid for the monhostid entry is in fact the same as fetched in the previous statememt. See monhost-getcfg for more information.
      2. $iepmSrcDir/get-nodelist - This should return a list of the target hosts and what tools they are monitoring. See get-nodelist for more information.
      3. Generate the gnuplot test plot: $iepmSrcDir/utils/make-gnutest-plot. This will create a test plot of the gnuplot colors and characters for referencing when deciding what colors to plot the data in. This will be accessible from the master IEPM-BW page.
      4. Execute $iepmSrcDir/keep-em-alive. This code starts the IEPM-BW test daemons for testing purposes
      5. Test that the system will work properly by executing $iepmSrcDir/schedule-load. This execution loads one pass into the schedule table. The daemons should begin working at this point and will store the acquired data in the mysql/data directory.
      6. Once $iepmSrcDir/schedule-load is working and data is going into the mysql/data directory, test loading the collected data into the database by using the load scripts. 
        The current load data scripts available are:
load-iperf-data
load-pathchirp-data
load-pathload-data
load-ping-data
load-tlaytcp-data
load-trace-data
        Based on the tests that you are running, and the data seen in mysql/data, execute $iepmSrcDir/load-[type]-data to simulate uploading the collected into the database. This also proves that the scripts will work when the system is automated. NOTE: The above steps can also be simulated by running $iepmSrcDir/schedule-load -T [type], verifying that the data from this execution can be seen from mysql/data, followed by executing the associated $iepmSrcDir/load-[type]-data script to simulate the data upload to the database.
      7. Start the $iepmSrcDir/load-datad daemon.
    •   Send Us Feedback  
      SLAC