skimBackup
skimBackup is part of SkimTools
package. It serves to do a backup on tape of kanga files. Which file is
to backup and what is already backed-up are information stored in BBRORA
database. So skimBackup, as skimImport,
skimSqlSelect
and skimDelete, works mainly directly
querying and updating BBRORA.
How skimBackup works
skimBackup works in two different way
-
List-driven: skimBackup gets a list
of files to backup directly from a text file, backups them and updates
BBRORA. If some file from the list doesn't exist, skimBackup
skips it and logs a WARNING message (see log files
later); if some file isn't present on BBRORA, skimBackup
skips it and logs a WARNING message, if the file is already backed up skimBackup
notifies that and skips it and, finally, if a file has size=0 bytes or
size different from that stored in database skimBackup
skips it and logs a WARNING message. When a file has been put on tape its
corresponding entry in BBRORA is tagged as "0B".
-
Database-driven: skimBackup gets all
BBRORA's entries tagged as "0" (means "on disk"). Each entry is supposed
to correspond to a physical file (else skimBackup
logs a WARNING message) so skimBackup puts
it on tape and, after, updates BBRORA tagging that entry as "0B".
skimBackup has internally a routine "BackupFile"
that perform the putting of files on tape. Each site must have its
own BackupFile routine that is strongly site-dependent (of course). See
BackupFile
routine implementation for details.
How to use skimBackup
The sintax of skimBackup is:
skimBackup --help
to get help on available options
skimBackup --version
to get info on version/author/creation date
skimBackup --sql-pwdfile <PWD_FILE> [--fake] [--sql-host <SQL_HOST>]
\
[--sql-user <SQL_USER>] [--sql-db <SQL_DB>]
[--sql-driver <SQL_DRIVER>] \
[--numitems N1] [--numtapes N2] \
[--log-file <ALTERNATE_LOGFILE_BASENAME>]
[--log-dir <ALTERNATE_LOGDIR>] \
[--stat <SOURCE_STATUS>] [--newstat <TARGET_STATUS>]
\
[--bkplist <LIST_FILE>]
As you can see there's only one mandatory argument; a simple "skimBackup
--sql-pwdfile <PASSWORD_FILE>" command will backup maximum
10000 files tagging the corresponding BBRORA's entries as "0B" and logging
its work in directory $BFROOT/kanga/log. Of course you can change the maximum
number of jobs (default is 10000 in order to get a non-huge activity-logfile).
You can also do a "FAKE" run, just to see what it should happen; and so
on...
see options.
Options meaning
-
--sql-host specifies the machine running the
sql server
-
--sql-driver specifies the sql engine type (for now USE
only "mysql", or don't use at all because DEFAULT="mysql")
-
--sql-user specify a user name having read/write
permission on BBRORA (BBRORA's administrator)
-
--sql-pwdfile specify the file containing the BBRORA administrator's
password
-
--sql-db
specify an alternative different from "bbrora" for BBRORA database
-
--numitems specify the maximum number of files
that you want to stage
-
--numtapes specify the number of contemporary
threads writing on tape (in order to use more than one tape unit)
-
--log-file speficy a different file-name for
the log files (default is "skimBackup", just
the program's name)
-
--log-dir specifies a different directory
where put the log files (DEFAULT="$BFROOT/kanga/log")
-
--fake don't really
do anything, just shows wath it should do
-
--stat specifies the
initial status of files to backup
-
--newstat specifies the new status of
files backed up
-
--bkplist specifies a list file containing
the file-names to backup; the format of each row MUST be:
$BFROOT/kanga/EventStore/groups/...
<$BFROOT>/kanga/EventStore/groups/...
EventStore/groups/...
-
--version
shows version/history/author's information and exits
-
--help
shows command's help and exits
Attention:
* The default value for --stat's argument
is "0", and the default value for --newstat's
argument is "0B". But you can change these initial and final status as
you want. Remember that doing this you "re-define" the "ON-DISK" status
and "ON-DISK AND ON TAPE" status for those entries and when you will use
skimData,
you will have to use --remote option with
a proper argument . So it could be reasonable to change the initial status
of files you want to backup, but the final status should be always "0B"
or "B", because by default skimData returns
kanga files tagged as "0" or "0B".
* If you use --remove option, the final
status will be set to "B" (="ONLY ON TAPE").
Log files
By default, the log files are put in $BFROOT/kanga/log,
unless you re-define it with --log-dir; there's
a log file:
skimBackup.YYYYMMDD.<RUN>
and for each file backed-up on tape there's a detailed log file
in the subdirectory:
skimBackup.YYYYMMDD.<RUN>.d/skimBackup.YYYYMMDD.<RUN>.#######
In this log file you can find information on which file it's working
about, under copying, under staging (putting on tape), under updating (in
bbrora, transaction `0'=>`0B', see import_status
description); ####### is an incremental number from 0 to the total
number of files backed-up (limit to 10,000,000 files !).
The logfile's base name "skimBackup"
can be changed using --log-file="NEWNAME".
BackupFile routine implementation
Each site can have a different way to get access to tapes. For example,
some site (like caspur) must copy each file in a special area and after
it can put it on tape. So each site that want use skimBackup have to create
a its own BackupFile routine. This routine is the interface between skimBackup
utility and staging system. BackupFile can
be implemented any possible way, but it must return "1" in success case
and "0" on failure, because skimBackup needs
to handle these two different situations: proceed with BBRORA update or
stop. So SkimTools package now have a subdir
"site/". In this subdirectory each site have
to create its subdir "$BFSITE". In SkimTools/site/<$BFSITE>
each site should put a support-file named "localStage.pl"
containing the implementation of BackupFile
and some other configuration variables useful for that site.
It doesn't matter the implementation of BackupFile
routine; is MUST accept the path/filename in the physical format
and the file size as arguments, for example:
/babar/bfroot/kanga/EventStore/groups/skims/MasterSP/0038/R8.8.0c/00383782-00383806/Skim880gMC/P0000/Charmonium/JpsitollKanga-micro.root
and MUST return 1 succesfully and 0 in error case. Take into
account that BackupFile is called by skimBackup
in this way:
BackupFile($filename, $filesize);
where $filesize is measured in bytes.
localStage.pl have to contain the definition
of the following variable:
-
$KANGALOGDIR (for us at caspur it is $BFROOT/kanga/log;
it is any directory that will host the log files; probably you already
have it for log files of your kanga-import tools)
or you can specify it using --log-dir="<KANGALOGDIR>"
command line option.
localStage.pl CAN contain the definitions
of the following variables:
-
$SQL_HOST (the machine running a sql server
daemon)
-
$SQL_DRIVER (the type of sql-server, mysql/oracle
etc.)
-
$SQL_USER (the user having read/write permission
on BBRORA)
-
$SQL_DB (the alternate name for BBRORA, defalut
= "bbrora" built-in skimBackup script)
If these variables are not defined in localStage.pl you can specify them
by command line options (see above).
You can read SkimTools/site/caspur/localStage.pl
as example
Page written by Alvise Dorigo
Alvise.Dorigo@pd.infn.it
Web Page
Last update: 17 February 2001.
|
