Using a Local Database
The procedure is as follows
- Install MySQL
- Create BaBar bookkeeping databases
- Select files to import: BbkFiles
- Import files: BbkImport
Steps 3-5 can be done in a cron job, augmented by one-time BbkFiles
selections (allows additional imports to be queued, changes in
priority, etc).
Please see: Setting up local Bookkeeping Databases for
the latest documentation.
Select files to import
Now you can query your new database. Initially all files will have
remote status=2, so you need to specify that on the BbkFiles command
line (by default it just shows the files on disk, ie. with status=0).
Or to select everything, use --remote='*'
% BbkFiles --dbsite=local --dataset=AllEventsRun3 --remote='*'
New file
(status=2)
Skim Release Stream Components Files Events GBytes
============ ========= ========== ===== ========= ======
12.4.0g AllEvents E 860 120897863 491.8
12.4.0g AllEvents HCBA 860 120897863 312.0
12.4.0h AllEvents E 681 87246705 360.0
12.4.0h AllEvents HCBA 681 87246705 225.6
12.4.0j AllEvents E 2145 265955780 1152.2
12.4.0j AllEvents HCBA 2144 265406972 695.9
============ ========= ========== ===== ========= ======
Totals 7371 947651888 3237.6
database: 7371 (3476346311340 bytes, 3.2 TB)
BbkFiles can also be used to define which of these files to import. For
example, to select all files containing the aod component
% BbkFiles --dbsite=local --dbuser=bbrora --dataset=AllEventsRun3 \
--components=A --remote=2 --setremote=1C
New file
(status=2)
Skim Release Stream Components Files Events GBytes
============ ========= ========== ===== ========= ======
12.4.0g AllEvents HCBA 860 120897863 312.0
12.4.0h AllEvents HCBA 681 87246705 225.6
12.4.0j AllEvents HCBA 2144 265406972 695.9
============ ========= ========== ===== ========= ======
Totals 3685 473551540 1233.5
change import_status to '1C' for 3685 files
database: 3685 (1324509062440 bytes, 1.2 TB)
--remote=2- Selects just those files that haven't already had their status defined. This
allows BbkFiles to specify the disposition of all new data.
--components=A- select only files with
A in the components list; ie files containing the aod component (use E for files containing esd). If --components is not specified, then all components are selected.
--dbuser=bbrora- Allow write access to the database
--setremote=1C- New status flag. Statuses starting with 1 (selectable
with --remote='1*')
are marked "to import". The second letter gives the import priority
(A=top, Z=bottom),
defining the order in which BbkImport will transfer the files.
Import files
BbkImport, by default, will only import files with statuses '1*', so normally no further selection is required on the BbkImport command line
% BbkImport --dbsite=local --dbuser=bbrora --ftp=bbftp --maxcopy=2 \
bbrdist@bbr-xfer01.slac.stanford.edu
04/02/17 05:35:33 2 files (590439421 bytes, 563.1 MB) selected for transfer
04/02/17 05:35:33 checking stop-files BbkImport.stop.11729 BbkImport.pause.11729 ...
04/02/17 05:35:48 get 2 files (590439421 bytes, 563.1 MB):
bbr-xfer04.slac.stanford.edu:/kanga/store/PR/R12/AllEvents/0003/91/12.4.0j/
AllEvents_00039191_12.4.0jV00_C14.2.0bV00.01.root
(188160487 bytes, 179.4 MB onto /stage/bdata-top1)
bbr-xfer04.slac.stanford.edu:/kanga/store/PR/R12/AllEvents/0003/91/12.4.0j/
AllEvents_00039190_12.4.0jV00_C14.2.0bV00.01.root
(402278934 bytes, 383.6 MB onto /stage/bdata-top1)
04/02/17 05:50:58 copied 2 / 2 files, 590439421 bytes (563.1 MB), 15:08.66 mins,
4.96 Mbits/s (15:24.28 mins, 4.87 Mbits/s with overhead) (OK)
In this example we have specified --maxcopy=2 to limit the number of files imported here.
Collections will be marked local (is_local=1) only when all component files marked for import are transfered (none with status='1*,2', at least one with status='0*').
 /BFROOT/www/Computing/Offline/DataDist/bbk-localdb.html last last modified on 27th July 2004 by
Tim Adye, <T.J.Adye@rl.ac.uk>
|
