skimSqlMirror
skimSqlMirror is the base program of the mirroring tools. It actually
copies the data from a source database to a target
database.
The information copied are:
-
bbrora all skim_* tables
-
bbrora shift_runs columns
-
bbrmdc some PROD_* tables
At SLAC these tables are in two Oracle databases bbrora, and
bbrmdc for the SP information. At remote sites instead all the
tables have been put in a single db: bbrora. This has been done
to avoid possible conflicts within bbrmdc if it is used for
local MC production.
A relational database is organised in tables connected by
relations
or keys.
These tables have different sizes, so different techniques have been
used to speed up the mirroring.
-
Smaller tables (skim_configs,skim_groups,skim_releases,skim_streams)
are checked and if something has changed they are
rebuilt from scratch, unless --nosmallertables option is used.
-
Only few columns of SP tables are copied every time from scratch.
-
Big tables (skim_jobstats,skim_requests,shift_run,skim_streamstats) are
incrementally updated following the chain below where req_id
and run are the keys that connect the tables:
job_id (incremental index)
|
|
skim_jobstats -----> skim_requests ---> shift_run
| req_id
run
|
|-----> skim_streamstats
req_id
-
Checks are made to see if the number of lines is consistent at
the end of the import. skim_requests.status_code flag has a
special meaning. At SLAC it says how is the status of a production job
(pending,queued,running,done,failed,ok); only jobs with status_code ok
are visible to skimData. At remote sites this flag can have two
values: ok file successfully produced at SLAC and failed
file deleted at SLAC. Consistency checks on the number of rows
is based on entries with status_code=ok.
examples:
-
The target database must be built. skimSqlMirror runs on the source host
(tipically a slac machine), the name of the database administrator user
(i.e. bfactory) is different from the user who is running and the password
file is not what expected: ($BFROOT/work/<beginning-of-$USER>/$USER/$USER.sqp)
skimSqlMirror --targethost < MY_HOST > --user
< MY_ADMIN_USER> --pwdfile < MY_PASSWORD_FILE>
--all
-
The target database must be built but the source is another mysql database.
skimSqlMirror can be executed on the target machine.
skimSqlMirror --sourcehost < SOURCE_HOST > --sourcedriver < SOURCE_HOST > --user
< MY_ADMIN_USER>
--pwdfile < MY_PASSWORD_FILE>
-
The target database must be updated. skimSqlMirror runs on the source host
(tipically a slac machine), the name of the database administrator user
(i.e. bfactory) is different from the user who is running and the password
file is not what expected: ($BFROOT/work/<beginning-of-$USER>/$USER/$USER.sqp)
skimSqlMirror --targethost < MY_HOST > --user
< MY_ADMIN_USER> --pwdfile < MY_PASSWORD_FILE>
HELP
usage: skimSqlMirror [OPTIONS]
| OPTION |
ARGUMENT |
DESCRIPTION |
| -H,--help |
|
print help and exit |
| -V,--version |
|
print version info and exit |
Copy Options:
| OPTION |
ARGUMENT |
DESCRIPTION |
DEFAULT |
| -a,--all |
|
copy the whole database |
unset |
| --nosmalltables |
|
do not check and update of smaller tables |
unset |
Database connection parameters:
| OPTION |
ARGUMENT |
DESCRIPTION |
DEFAULT |
| -th,--targethost |
TARGETHOST |
target database host |
unset |
| -sh,--sourcehost |
SOURCEHOST |
source database host |
unset |
| -td,--targetdriver |
TRGDRV |
database engine on target host |
mysql |
| -sd,--sourcedriver |
SRCDRV |
database engine on source host |
Oracle |
| -u,--user |
SQLDB_USER |
administrator user of the target database |
anyuser |
| -d,--db |
SQLDB_NAME |
database name |
bbrora |
| --pwdfile |
PASSWORDFILE |
file containing SQL password for the admin user |
$BFROOT/work/<beginning-of-$USER>/$USER/$USER.sqp |
skimSqlSelect
skimSqlSelect is the import_status
column management tool. It reads and writes from the same database
(differently from skimSqlMirror that reads from a source database and writes
into a target one). Suppose you have built your database at your institute
and you want to use it to import different streams with different priorities,
then the import_status flag must be properly set.
examples:
-
To change import_status flag from 2 to 1 ONLY for BPCMuonKanga skimmed with the release
S863cP1cA and for ALL Stream14Kanga entries
skimSqlSelect --skimData "--stream BPCMuonKanga --skimrel S863cP1cA:--stream
Stream14Kanga"
- To change import_status flag from 1 to E for ALL the kanga entries
BUT Stream14Kanga and Stream15Kanga
skimSqlSelect --kanga --remote 1 --setremote E --exclude Stream14Kanga,Stream15Kanga
- To change many different streams selected with skimData from one value of import_status
(i.e. 1) to a different value for each stream
skimSqlSelect --remote 1 --skimoptfile <FILE> (*)
(*) example of
file
to use
HELP
usage: skimSqlSelect [OPTIONS]
| OPTION |
ARGUMENT |
DESCRIPTION |
DEFAULT |
| -H,--help |
|
print help and exit |
unset |
| -V,--version |
|
print version info and exit |
unset |
| -v,--verbose |
|
print different level of output 0,1,2 |
0 |
Connect Options:
| OPTION |
ARGUMENT |
DESCRIPTION |
DEFAULT |
| -h,--host |
HOST |
database host |
unset |
| --driver |
DRIVER |
database engine |
mysql |
| -u,--user |
DB_USER |
database administrator user |
anyuser |
| --database |
DB_NAME |
database name |
bbrora |
| --pwdfile |
PASSWORDFILE |
file containing SQL password for the admin user |
$BFROOT/work/<beginning-of-$USER>/$USER/$USER.sqp |
File existence check:
| OPTION |
ARGUMENT |
DESCRIPTION |
DEFAULT |
| --check |
|
only change status for existing files |
unset |
| --nocheck |
|
do not check whether file exists |
set |
Set import_status flag options:
| OPTION |
ARGUMENT |
DESCRIPTION |
DEFAULT |
| --remote |
FLAG |
select files with import_status=FLAG |
1 if --check, 2 if --nocheck |
| --setremote |
FLAG |
set import status to FLAG value |
0 if --check, 1 if --nocheck |
Include options:
| OPTION |
ARGUMENT |
DESCRIPTION |
DEFAULT |
| --skimData |
SKIMDATA OPTIONS |
sequence of colon separated skimData options |
unset |
| --skimoptfile |
FILE |
read skimData option from file
Format: <final import_status> <skimData options> |
unset |
| --kanga |
|
update ONLY and ALL kanga entries |
unset |
| --objy |
|
update ONLY and ALL objectivity entries |
unset |
| --reset |
|
set ALL the entries back to 2 |
unset |
Exclude options:
| OPTION |
ARGUMENT |
DESCRIPTION |
DEFAULT |
| --exclude |
STREAMS |
sequence of comma separated list of streams to be excluded |
unset |
| --excludefile |
FILE |
files containing a list of streams to be excluded
Format: <stream_name> |
unset |
| --exclrem |
FLAG |
select files with import_status=FLAG to excluded |
unset |
Back
to Kanga Data Distribution
Page maintained by : aforti@mail.hep.man.ac.uk
|
