CM2 at Tier 2 UK farms
Introduction
At the steering committee I (Fergus) rashly admitted that I had transferred the
CM2 data to the Bristol farm and was successfully running on it. Here
is my blog of how I did it. It took me a few days as I went down a few
blind alleys and the software was changing as I went. I also moved
the mysql database from our sun to the linux system as our system
manager has cleverly upgraded the kernel to a version that crashes
with afs.
If you have mysql happily running you can skip the section on
Installing mysql. I have
upgraded to v4.0.18 although I think Babar only approves 3.56. I have had
no problems and v4.0 does come with a GUI admin interface.
It will take about a morning or less to install the software. Updating
the database the first time takes a few days and transferring over the
files took about another 36 hours for ~200 gbytes. You mileage may
vary.
Much of what follows could be superfluous but it got the job done.
There is good documentation on the bookkeeping in the here.
You can easily destroy any existing database such as skimData
stuff if you follow these instructions. So make sure you know what you
are doing especially if your farm is set up differently. I've included
instructions on backing up the database.
Some of the problems I encountered are due to be fixed.
Bookkeeping updates
Check
here for updates to the bookkeeping tools and schema.
Versions and releases
You should update your $BFROOT/bin to V00-00-86 and use release
14.5.2d (i.e. analysis-21 with all the bookkeeping updates) or
whatever is
the currently recommended release..
Try to link a BaBar program. If you find that the liboo libraries can
not be found then you may need to edit SRTstartup.csh and
SRTstartup.sh. This appears to happen if you link your release
directories from RAL over afs. This is because the new linking method assumes
that the objectivity libraries are in the release tree but at RAL they
are links to nfs files that are invisible from outside. Find "set
newobjy=1" and change it to "set newobjy=" in two places in
SRTstartup.csh; in SRTstartup.sh comment out line 918 to 924.
I've no doubt this hack is not approved... but it works for me.
Where to store the CM2 data
The location (and access rights to the files) is controlled by
"$BFROOT/kanga/config/KanAccess.cfg" at your site. Check out the HOWTO
package and look at HOWTO-Setup-a-CM2-Kanga-Eventstore
for more details. At Bristol it looks
like:
read /store/SP/BkgTriggers/* file /data/bfb/babar/mywork/mcprod/
read /store/SP/* file /babardisk10/kanga/EventStore
read /store/PR/* file /babardisk9/kanga/EventStore
read /store/PRskims/* file /babardisk9/kanga/EventStore
write /store/PR/* error
read /work/* file /babardisk10/
write /work/* file /babardisk10/
You can ignore the first line as it deals with SP production (which
all your farms are doing anyway, right?). The last 2 lines decide
where users collections will go. babardisk9 and babardisk10 are links
to our disk servers. So just change the last entry on each line to
point to where you want to keep the files (but keep the
/directory/kanga/EventStore form for lines 2, 3, and 4).
Backing up old mysql tables
If you have the old skimdata running you may wish to backup the tables
before proceeding. So stop the server, dump and compress the bbrora
meta-tables. This will take some time and needs a few gigabytes
depending on what you have in it.
> /etc/init.d/mysql.server stop
> mysqldump --quick bbrora | gzip > bbrora.compress.gz
Installing mysql
If you need to install mysql do the following; if not skip this
section and go on to Installing CM2 Book-keeping tables.
Create a mysql user and group to run the database:
> groupadd mysql
> useradd -g mysql mysql
Copy over the mysql.tar from www.mysql.org. I've used version
4.0.18. Unpack in a likely area.
> cd /usr/local/opt
> tar -xvf /path/to/mysql-standard-4.0.18-pc-linux-i686.tar
> cd /usr/local
> ln -fs opt/mysql-standard-4.0.18-pc-linux-i686 mysql
Now start the configuration
> cd /usr/local/mysql
> scripts/mysql_install_db --user=mysql
Set the correct privileges
> chown -R root .
> chown -R mysql data
> chgrp -R mysql .
You'll probably need to change some settings in /etc/my.cnf. Most of
these are in fact defaults (but I don't know which!):
[client]
socket=/tmp/mysql.sock
[mysqld]
port=3306
user=mysql
tmpdir=/tmp
max_allowed_packet=32M
socket=/tmp/mysql.sock
[mysql.server]
user=mysql
basedir=/usr/local/mysql
[safe_mysqld]
err-log=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
By default mysql will put its meta tables in the same area you
installed it which probably isn't a good long term solution as they can get
quite big and could fill your system disk (the meta-tables are
1.9 Gbytes currently on the bristol farm). So create an area for these
files on a big disk e.g.
> cd /data/bfa/babar/
> mkdir -p mysql/data
> chown -R mysql mysql
> chgrp -R mysql mysql
copy over the files needed to start mysql on reboot.
> cp support-files/mysql.server /etc/init.d/mysql.server
> chmod +x /etc/init.d/mysql
> chkconfig --add mysql
Now start mysql server
> ./bin/mysqld_safe --user=mysql &
Now you need to do some security admin. Remove insecure access:
> ./bin/mysql -u root mysql
mysql> delete from user where host='localhost' and user='';
mysql> flush privileges
mysql> quit
Now create a password for root (newpass1 is the password):
> ./bin/mysqladmin -u root password 'newpass1'
Install bfactory user and give it a password (newpass2 is a
different password to newpass1)
> ./bin/mysql -u root -p < SkimTools/remoteinit.mysql
>./bin/mysqladmin -u bfactory password 'newpass2'
Now load the table schema
> ./bin/mysql -u root -p < SkimTools/remotebbrora.mysql
Now you have a table schema for bbrora. But it will be in system area
so we need to move it to the big disk and make a link (see above):
> cd /usr/local/mysql/data
> mv bbrora /data/bfa/babar/mysql/data/
> ln -fs /data/bfa/babar/mysql/data/bbrora .
> chown mysql bbrora
> chgrp mysql bbrora
Now do some simple testing
> ./bin/mysqladmin -u root -p version # show mysql version
> ./bin/mysqladmin -u root -p variables # show the parameters
> ./bin/mysqladmin -u root -p shutdown # shutdown server
> ./bin/mysqld_safe --user=mysql & # start server
> ./bin/mysqlshow # show tables
> ./bin/mysqlshow mysql # show schema of mysql table
Now do more extensive testing (about 30 minutes I think). I think
there was one error which was minor.
> cd test-sql
> ./mysql-test-run
If everything is working, give root the necessary privileges. I had to
do this twice as "bfa" and "bfa.phy.bris.ac.uk" are interpreted as two
different machines.
> ./bin/mysql -u root -p mysql
mysql> select host,user from user;
mysql> grant all privileges on *.* TO 'root'@'bfa.phy.bris.ac.uk' identified by 'newpass1' WITH GRANT OPTION;
mysql> grant all privileges on *.* TO 'root'@'bfa' identified by 'newpass1' WITH GRANT OPTION;
Restoring old mysql tables
To reinstall the old skimdata files, restart the server (if stopped)
> ./bin/mysqld_safe --user=mysql &
You need to drop the bbrora tables first. I think this is where you
lose all the skimData information at your site so make sure you have
the database backed up (see above):
> ./bin/mysql -u root -p < dropbbrora.mysql
where dropbbrora.mysql contains
use bbrora;
DROP TABLE IF EXISTS shift_run;
DROP TABLE IF EXISTS skim_configs;
DROP TABLE IF EXISTS skim_tclcmd;
DROP TABLE IF EXISTS skim_groups;
DROP TABLE IF EXISTS skim_jobstats;
DROP TABLE IF EXISTS skim_releases;
DROP TABLE IF EXISTS skim_requests;
DROP TABLE IF EXISTS skim_streams;
DROP TABLE IF EXISTS skim_streamstats;
DROP TABLE IF EXISTS skim_files;
DROP TABLE IF EXISTS skim_archive;
DROP TABLE IF EXISTS PROD_JOB;
DROP TABLE IF EXISTS PROD_MODES;
DROP TABLE IF EXISTS PROD_REQ;
DROP TABLE IF EXISTS PROD_RUN;
The restore the tables. This will take many hours:
> gunzip < bbrora.compress.gz | mysql bbrora
Installing CM2 Book-keeping tables
create a 14.5.2d directory:
Check out BbkTools as recommended by here
> srtpath
> setboot
> mysql -u bfactory -p < BbkTools/bbk_table_schema.mysql
> mysql -u bfactory -p < BbkTools/data_table_schema.mysql
Installing CM2 Book-keeping tables contents
Now you need to copy over the contents of the database. This requires
setting up some access rights:
> mkdir -p ~/.bbk/sites/local/anyuser
> mkdir -p ~/.bbk/sites/local/bfactory
In directory anyuser, create a files "anyuser" with the one-line entry
of "query":
>cat ~/.bbk/sites/local/anyuser/anyuser
query
Similarly, create a file ~/.bbk/sites/local/bfactory/bfactory
containing the password you used to create the bfactory mysql account:
>cat ~/.bbk/sites/local/bfactory/bfactory
newpass2
Now copy over the meta data; this can take days.
Add the -v option to what is happening. It may not be
necessary to klog at all.
> srtpath
> klog fwilson@slac.stanford.edu
> BbkDbMirror -v -ss slac -su anyuser -ts local -tu bfactory
Updating CM2 Book-keeping tables contents
To keep the contents of the database uptodate, issue the following
command regularly. Seems to take about 30 minutes if done once a day:
> BbkDbMirror -ss slac -su anyuser -ts local -tu bfactory --alsoupdate --skipreplica
Selecting the CM2 files to import
You need to choose which files to import. Use BbkDatasetTcl to get a
list. Here I assume you wish to import all the 3-charged track 3-body
data skims.
Set the status of any 3-charged track 3-body files not held locally to
"to-be-imported":
> BbkFiles --dbsite=local --dbuser=bfactory --dataset='BCCC3body*' --remote='2' --setremote='1C'
Display what is waiting to be imported:
> BbkFiles --dbsite=local --dbuser=bfactory --dataset='BCCC3body*' --remote='1C'
Setting up ssh-agent
To transfer from SLAC you'll need to setup ssh-agent. You only need do
this once. Create a key with a password:
localfarm> ssh-keygen -t rsa
log in to bbr-xfer01, copy over the key and add to authorized users:
localfarm> ssh fwilson@bbr-xfer0x.slac.stanford.edu
bbr-xfer01> scp fwilson@localfarm:~/.ssh/id_rsa.pub .
bbr-xfer01> cat id_rsa.pub >> .ssh/authorized_keys
bbr-xfer01> rm id_rsa.pub
bbr-xfer01> exit
Importing the CM2 files
Now import files. The options are very similar to SkimImport. How you
import the data is up to you.
> ssh-agent tcsh
> srtpath
> cd workdir
> ssh-add ~/.ssh/id_rsa
> ssh-add -l
Now you can transfer the files:
> BbkImport -v --ftp-type=bbftp --dbsite=local \
--dir-local=/usr/local/bin \
--dbuser=bfactory --nstreams=6 \
fwilson@bbr-xfer0x.slac.stanford.edu
"--dir-local" is where you have your bbftp binary. The time will
depend on your network speed and the amount of data to be imported. At
Bristol it took about 36 hours for 200 Gbytes.
At Bristol, I had to do some tricks to get through our firewall, which
involved the use of non-standard bbftp and the use of the
"--remote-program" option. Also, to use ssh-agent I had to tell the
code to use ssh protocol 2 with the --ssh-flag="-2" option as this
does not seem to be the default at Bristol.
Keeping everything uptodate
To keep things uptodate just go through the steps:
- Updating CM2 Book-keeping tables contents
- Selecting the CM2 files to import
- Importing the CM2 files
Problems
The one problem I had was that the files held locally were not
recognized as being local when I tried the GreenCircle dataset. This
may not be a problem for you as I was creating the database before the
alias was created. This was corrected using an Adye magic command:
> BbkModifyDseFlags --dbsite=local --dbuser=bfactory --autosetislocal --all
Fergus (2nd August 2004)
