Conditions Snapshots
Warning : on June 2004 new DB interface was introduced
( here),
which takes OO_FD_BOOT from the configuration of farm. Managers do not need to specify it by hand anymore! Old interface still works, but new commands are much more safer. On Jul 2004 not all commands are tested yet (as marked).
If you need to invoke any of these commands - please use new interface and
if successfull - please, remove "not tested" mark.
The conditions need to be managed between the Master conditions federation,
the PR and the ER farms.
The Master conditions federation is not used directly by any of the farms.
It is a repository for all the conditions and rolling calibrations (RC). New constants
that are loaded by hand (by sub-system experts) are loaded here and then transfered
to the PC farms. When the RC are transfered from a PC to ER farm, the same snapshot
must be loaded to the Master to keep it up to date.
More information about snapshot transfers could be found here.
To find out which OO_FD_BOOT is used by farm do
- for PC OO_FD_BOOT :
../bin/$BFARCH/OprCmd.pl -nobjyserv03 -p 1602 -iCdb getboot PCx
- for MASTER OO_FD_BOOT :
../bin/$BFARCH/OprCmd.pl -nobjyserv03 -p 1602 -iCdb getmasterboot PCx
To take a PC snapshot:
- Suspend the PC farm:
../bin/$BFARCH/OprCmd.pl -iFarm,User -noprserv0x suspendfarm PCx
This will tell the farm to stop after the current run is
done being processed and go to sleep.
Lock the PC farm:
../bin/$BFARCH/OprCmd.pl -iFarm,User -noprserv0x lockfarm PCx
this will stop the farm from responding to any commands such as resumefarm
until it is unlocked.
- Take the snapshot(s) from PC:
new: ../bin/$BFARCH/OprCmd.pl -nobjyserv03 -p 1602 -iCdb takePCxSnapshot
old: ../bin/$BFARCH/OprCmd.pl -iCdb -nobjyserv03 -p1602 takePCxSnapshot OO_FD_BOOT:$PCx-bootfile
where $PCx-bootfile is replaced by the actual path of the bootfile. Then
answer yes when asked, if you are sure you are using the correct bootfile
(you can check if uncertain).
It will take quite some time to take the snapshot.
- Unlock the PC farm:
../bin/$BFARCH/OprCmd.pl -iFarm,User -noprserv0x unlockfarm PCx
Resume the PC farm:
../bin/$BFARCH/OprCmd.pl -iFarm,User -noprserv0x resumefarm PCx
To Load PCx snapshot to Master:
- Load the PC snapshot(s) to the Master:
This is a VERY important step since it keeps the Master up to date.
../bin/$BFARCH/OprCmd.pl -iCdb -nobjyserv03 -p1602 importPCtoERSnapshotSlac PCxToMASTER
answer yes when asked, after checking
that the bootfile is the right one.
It will take a couple minutes (these days up to 15!) to import the snapshot.
To Load PC1 snapshot to ER (Not Yet Updated):
- Suspend the ER farm:
../bin/$BFARCH/OprCmd.pl -iFarm,User -noprserv0x suspendfarm ERx
This will tell the farm to stop after the current run is
done being processed and go to sleep.
Lock the ER farm:
../bin/$BFARCH/OprCmd.pl -iFarm,User -noprserv0x lockfarm ERx
this will stop the farm from responding to any commands such as resumefarm
until it is unlocked.
- Load the snapshot(s) from PC1:
../bin/$BFARCH/OprCmd.pl -iCdb -nobjyserv03 -p1602 importPCtoERSnapshotSlac PC1ToERx
- Unlock the ER farm:
../bin/$BFARCH/OprCmd.pl -iFarm,User -noprserv0x unlockfarm ERx
Resume the PC farm:
../bin/$BFARCH/OprCmd.pl -iFarm,User -noprserv0x resumefarm ERx
Master To PC Conditions Transfer
Often when a subsystem needs to load new constants they are
first written to the MASTER and then loaded to the relevant PC farms.
The ER farms will get the updated conditions the next time they import
a PC snapshot.
The conditions federation for the ER farms are meant to be
mirror of the PC conditions. It is important to ensure that a given run
is processed in both the PC and ER farm with the SAME conditions.
- After the expert has loaded the constants, take a snapshot of the Master federation
Take the snapshot of the Master:
new:(NOT TESTED) ../bin/$BFARCH/OprCmd.pl -nobjyserv03 -p 1602 -iCdb takeMasterSnapshot PCx
old:
../bin/$BFARCH/OprCmd.pl -iCdb -nobjyserv03 -p1602 takeMasterSnapshot /nfs/objyserv05/objy/databases/production/conditions/master/BaBar.BOOT
- Suspend the PC farm:
../bin/$BFARCH/OprCmd.pl -iFarm,User -noprserv0x suspendfarm PC1
This will tell the farm to stop after the current run is
done being processed and go to sleep.
Lock the PC farm:
../bin/$BFARCH/OprCmd.pl -iFarm,User -noprserv0x lockfarm PCx
this will stop the farm from responding to any commands such as resumefarm
until it is unlocked.
- Import the Master snapshot to the PC1 farm :
new:(NOT TESTED) ../bin/$BFARCH/OprCmd.pl -nobjyserv03 -p 1602 -iCdb importMasterTo PCx
old: ../bin/$BFARCH/OprCmd.pl -iCdb -nobjyserv03 -p1602 importMasterSnapshot /nfs/objyboot1/objy/databases/production/boot/physics/V4/slac/pc1/con001/BaBar.BOOT PC1
- You then need to update the view for the new constants which were loaded in the PC farm.
The expert which did the loaded should have told you the name of the constants,
e.g. /ifr/IfrDetectorConfigurationKeyP
NOTE: EACH CONSTANT IS IN THE CDB TWICE, WITH A SLIGHTLY DIFFERENT NAME,
once with a lower case and once with an upper case beginning of the sub-system name,
e.g. /ifr/... and /Ifr/... YOU MUST COMPLETE THE FOLLOWING ACTION TWICE (as described below).
For the action below you need to set OO_FD_BOOT path of PC1 farm correctly
(setenv OO_FD_BOOT /nfs/objyboot1/objy/databases/production/boot/physics/V4/slac/pc1/con001/BaBar.BOOT)
and invoke srtpath from RELEASE directory.
- Check the old view:
CdbBrowser config /ifr/IfrDetectorConfigurationKeyP
The REVISION-ID field gives the time (in BdbTime format) that the current
view of the constants was updated.
To convert to a readable time format do:
BdbTimePrint
choose 0
paste in the numerical REVISION-ID
- Update the view to the newly loaded one:
- For SVT local alignment :
CdbManager copy_condition /svt/SvtPWaferAlign -input_view "MASTER::<recent>" /svt/SvtPWaferAlign -force
This command will copy _configurations_ of SVT local alignment conditions from the most recent
view of MASTER origin into the current origin of PC1. The "-force" switch is
needed because the output condition already exists in PC1 CDB. Note also,
that "<recent>" is a keyword. No substitute is required - type it as is.
After the local view of PC1 to get exactly the same configuration as
it's currently set up in MASTER CDB.
- For other conditions :
CdbManager autoconfigure_condition /ifr/IfrDetectorConfigurationKeyP NOW
- Make sure that the update worked:
CdbBrowser config /ifr/IfrDetectorConfigurationKeyP
BdbTimePrint
choose 0
paste in the numerical REVISION-ID
This time should be the time when the expert loaded the new values
to the Master (and be different than the time you got before you updated the view).
- Important: Repeat the same with a capital at the beginning of the sub-system name,
e.g. /Ifr/... instead of /ifr/... (also for SVT local alignment)
CdbBrowser config /Ifr/IfrDetectorConfigurationKeyP
BdbTimePrint (This gives the old view time.)
CdbManager copy_condition /Svt/SvtPWaferAlign -input_view "MASTER::<recent>" /Svt/SvtPWaferAlign -force
( or
CdbManager autoconfigure_condition /Ifr/IfrDetectorConfigurationKeyP NOW)
CdbBrowser config /Ifr/IfrDetectorConfigurationKeyP
BdbTimePrint (This gives the new view time, make sure that it is correct.)
- Unlock the PC farm:
../bin/$BFARCH/OprCmd.pl -iFarm,User -noprserv0x unlockfarm PCx
Resume the PC farm:
../bin/$BFARCH/OprCmd.pl -iFarm,User -noprserv0x resumefarm PCx
- Post to OPRsos-hn that you have loaded these constants to the PC farm,
cc the expert that did the loading and cc the DB expert
involved. Note the first run (with the explicit
rundir path) that used the new constants and ask the expert to check, if
possible, that they are there and working as expected. Also add OO_FD_BOOT settings, in case they can check the contents of the database directly.
How to restart Snapshot Daemon in Padova
The snapshot daemon acts as an interface for the ER farms
to load the most recent conditions snapshot.
- It runs on cls0 from bbrobjy account
- its logs files can be found in /tmp/OprSnapshotDaemon-*
- the release (and workdir) directory is /nfs/opr0/Reprocessing/ER0/snap
to restart the snapshot daemon in Padova
- create a test release
- newrel -t
- ln -s snap
- addpkg OprControlSys with latest tag and workdir
- srtpath; gmake workdir.setup; gmake installdirs;
- gmake OprControlSys.bin
- Execution recipe:
- ssh bbrobjy@cls0
- kill current process if one running
- cd /nfs/opr0/Reprocessing/ER0/snap/workdir
- ../bin/$BFARCH/OprSnapshotDaemonPd.pl -l /tmp &
this tells the daemon to put its logfile (-l) on /tmp.
The log file is found on cls0:/tmp/OprSnapshotDaemon-*
-
It seems that there is a problem to close ssh session in the window where daemon is started.
Just kill the session, and check that daemon is still running (it should)
More on Constant Loading and Views
- The loading of constants to MASTER
by subsystem experts must be synchronised with PR therefore PR managers must
take active part in the whole procedure and must make sure that all reasons and details are
clear and are well coordinated (by PR manager). DB expert (on July 2004 - Igor Gaponenko) must
be involved as well. The intend to load constants is usually initiated in IR2PROD-hn, after which
the request form must be filled. The output of request form appears in Condition DB management hypernews and should be followed up by PR manager.
- The MASTER CDB should have all conditions that exist
(that's why it's so importatn to keep it up to date), and all
should be visible in this CDB.
- PC1 has the option of taking information (such as a calibration of some sort)
but keeping it hidden from processing. In order to make something from a MASTER
snapshot visible, some extra steps are needed, as described above.
- You can see the differences in the visiblity of constants between a snapshot
from MASTER and the current PC1 view like this:
- setenv OO_FD_BOOT <PC-bootfile>
but specify the bootfile, of course. The bootfile can be found in OprProcessingSys/OprConfXXx.xml
- CdbManager diff_view "MASTER::<recent>" -origin MASTER -type REGULAR
This will output a bunch of stuff, but at the end it will list the differences in the views.
- You can also update the view in PC1 to include all new things in the MASTER:
- CdbManager merge_view "MASTER::<recent>" -origin MASTER -type REGULAR -update_config -verbose_mode
- More information on updating PC1 could be found
here
what needs to be done for SP
- SP needs PC1 rolling calibration to generate MC for a given period of time.
For that
- config alias for this period needs to be created by L3 expert in IR2-DB and sweeped to PCx
- PCx needs to be done with analyses of this period and
the RC output should be ok (according to DQG)
- new revision RC should be created by BDB expert (PC1 should be suspended during this time). This part can be done independent on config alias
- PCx to MASTER sweep (take PCx snapshot and load it to MASTER)
needs to be done after previous steps are performed and after IR2 snapshot is loaded to PC1
(this step is done by PR manager)
- MASTER to SP-DB sweep needs to be done (by BDB expert)
- to check that config alias is there, look for proper MonthYear entry in PC1 OO_FD_BOOT with
getTrgConfig Top/ | grep 2004 | sort -k2 -n
(works after srtpath in any directory) If needed period is not there ask L3 expert to make one
Last modified: Sat Jul 3 13:40:48 PDT 2004 by Olya Igonkina