|
face="Times New Roman" size="4">ABAR
Database Makefiles
Version Information
Draft: 19th October 1998
This document is still under development. If you
have any questions or comments, please address them to the author (DRQuarrie@LBL.Gov).
This document describes how developers may
manipulate their own Objectivity Federated Database (FDDB) in the context of the BABARRelease
structure. A FDDB may be created from scratch, or may be cloned from the main Reference
FDDB. If created from scratch, it is initially empty, containing no schema (class
definitions) or databases. If created by cloning from the Reference FDDB, it contains the
reference schema but no data.
A developers FDDB must be associated with its own
FDDB identifier (FID) in order to avoid collisions with other developers. The mechanism by
which this FID must be allocated is described here.
Once a FDDB has been created, it may be
manipulated by a combination of BABAR-specific tools described here, or the standard
Objectivity tools as described in the Objectivity manual "Objectivity/DB
Administration".
Return to Table of
Contents.
The GetFDID command may
be used to determine the base of range of federated database IDs that have been allocated
to each developer. Five such ids have been assigned to each developer and should be used
for all their development. This avoids any potential interference between developers.
THE GetFDID COMMAND CAN ONLY BE RUN
AT SLAC.
Syntax:
> GetFDID -h
usage: GetFDID [options] [<user>]
Prints out the range of five Objectivity Federated Database ID numbers
assigned to <user> (defaults to current user) for use in development.
Options are from among:
-h Prints this message.
-s Prints only the first number of the range.
> GetFDID
The five FDID numbers assigned to quarrie are 16855 - 16859.
> GetFDID -s becla
26200
Return to Table of
Contents.
Three different customization files may be
specified in order to override default configuration settings. These files are:
- SiteConfig/$BFSITE/SoftRelTools/arch_spec_FDDB.mk+.
This file is a site-specific customization file that overrides the global default settings
for a particular site. At the time of writing, such files exist for LBNL & SLAC and
are described in the Section "Site-specific
Customizations".
- $HOME/.bbobjy. This is
an optional user-level customization file that resides in the home directory. If present,
configuration parameter settings defined within it override both the default values and
any site-specific values.
- $PWD/.bbobjy. This is an
optional user-level customization file that resides in the release directory (the
directory from which the gmake commands are issued). If present, configuration parameter
settings defined within it override both the default values, any site-specific values and
any values that might have been specified in a $HOME/.bbobjy file.
The recommended procedure is for the developer to
have a separate .bbobjy file in each of their test releases.
The parameters that can be set in these
customization files, and their default values, are described in the Section "Configuration Paremeters and Default Settings".
The following is an example of a minimal
customization file. It sets the Federated Database ID as determined by prior usage of the GetFDID
command.
> cat .bbobjy
FD_NUMBER = 12345
>
Do not use this number - it is for
illustrative purposes only.
Return to Table of
Contents.
There are several configuration parameters that
determine the characteristics of the FDDB. These parameters have default settings, which
may be overridden by a set of site-specific and user-level customization files discussed
in the Section "Customization Files". The
configuration parameters and their default settings are:
AMS_HOST. This is the
nodename of the FDDB server machine. It defaults to the current machine, although in
general it is site-specific. At SLAC, it fedaults to objyserv1.
BOOT_FILE_DIR. This is
the directory within which the boot file resides. It defaults to $(FDB_DIR)
BOOT_FILE_NAME. This is
the name of the boot file. It defaults to BaBar.BOOT.
FDB_DIR_BASE. This is
the base of the directory tree for the location of the boot file and FDDB file. For a
normal user release, the actual location of the files is $(FDB_DIR_BASE)/<username>
where <username> is the user account name for the current user. For a production
release, the actual location of the files is $(FDB_DIR_BASE)/Releases/$(BFCURRENT).
This parameter has no default and is typically specified in a site-specific customization
file. Consult with your local site administrator if it is not specified for your site.
FDB_DIR_USER. This is
the user name. It should be treated as being read-only, being available for use as part of
an explicit customization.
FDB_DIR. This is the
directory within which the federated database itself resides. For a normal user release,
its $(FDB_DIR_BASE)/<username> where <username> is
the user account name for the current user. For a production release, the actual location
of the files is $(FDB_DIR_BASE)/Releases/$(BFCURRENT). If $(FDB_DIR_BASE)
is not set then this parameter will also not be set.
FDB_NAME. The name of
the federated database. It defaults to BaBar.FDB.
FD_NUMBER. This is the
federated database identifier (FID). It has no default and must be specified within a
site-specific or user-level customization file.
LOCK_HOST. The nodename
of the lock server host. It has no default and is typically specified in a site-specific
customization file. Consult with your local site administrator if it is not specified for
your site. The default lock server at SLAC is objyserv1.
PAGE_SIZE. This is the
Objectivity page size in bytes. It may be in the range 4096 - 65536, and defaults to
16384. The page size is ignored if the FDB is created from the Release or Reference
FDB.
Return to Table of
Contents.
In the BABAR standard release tools
(SoftRelTools) framework, a pseudo-package called database and
corresponding makefile have been created.
All FDDB operations are performed by specifying
one of several make targets for this pseudo-package thus:
gmake database.<target>
Valid targets are:
- backup.
This target creates a backup of the current FDDB and associated database files in the
database subdirectory.
- cleanup.
This target attempts to cleanup a federation after an error, removing any outstanding
transaction locks.
- config.
This target just reports the configuration parameters for the FDDB, but with no further
action.
- deleteboot.
This target deletes the FDDB and all associated databases (if any). This target is
protected by a command line or environment variable flag that must be specified.
- exists.
This target just indicates whether the FDDB exists.
- export.
This target exports a production release database to the export area for later importing
by developers.
- help.
This target displays a list of valid targets.
- import.
This target imports a copy of the production release database to a developers release.
- load.
This targets loads an empty FDDB with an initial configuration.
- makeboot.
This target attempts to create a new empty FDDB based on the current configuration
parameters. This target is protected by a command line or environment variable flag that
must be specified.
- reset.
This target resets the federation to the state it was in immediately following an import,
removing all database files and then reloading the federation with the initial
configuration.
- restore.
This target restores the FDDB from a previous backup.
- schemadump.
This target dumps the schema from the federated database to the export area for later
schemaLoad'ing.
- schemaload.
This target loads the schema into the federated database from a previously schemaDump'ed
version.
- status.
This target displays the status of transactions etc.
- update. This target updates the schema from the
reference schema.
Return to Table of
Contents.
This target creates a backup of the current FDDB
and associated database files in the database subdirectory of
the current release.
Example
> gmake database.backup
backup: Backing up /nfs/objyserv1/u3/ec/stage/objy/quarrie to database/backup.tar
[listing of files being backed up omitted]
backup: Backup complete
This command is limited in functionality to test
or developer federations where all the databases are contained in subdirectories based at
the directory containing the federated database file.
Return to Table of
Contents.
This target attempts to cleanup a federated
database after an error, removing any open locks.
Example
> gmake database.cleanup
cleanup:
Boot file (BOOT_FILE) /nfs/objyserv1/u3/ec/stage/objy/quarrie/BaBar
[remainder of output omitted]
cleanup: Cleanup complete
Return to Table of
Contents.
This target reports the FDDB configuration
parameters but with no further action.
Example
> gmake database.config
config:
Boot file (BOOT_FILE) /nfs/objyserv1/u3/ec/stage/objy/quarrie/BaBar
FDDB (FDB_DIR/FDB_NAME) /nfs/objyserv1/u3/ec/stage/objy/quarrie/BaBar.FDDB
AMS Server (AMS_HOST) objyserv1.slac.stanford.edu
Lock Server (LOCK_HOST) objyserv1.slac.stanford.edu
FDDB ID (FD_NUMBER) 1301
Page Size (PAGE_SIZE) 8192
Import Directory /afs/slac.stanford.edu/g/babar/dist/releases/5.4.1/database/export
Import boot file BaBarRef
Import FDDB BaBarRef.FDDB
Export FID 5799
Return to Table of
Contents.
This target deletes the FDDB and all associated
databases (if any). This target is protected by the DEFINE_BOOTFILE_RULES
flag that must be specified on the command line or as an environment variable. Great care
should be taken when accessing this target and it is designed to be used by experts only.
Example
> gmake database.deleteboot DEFINE_BOOTFILE_RULES=yes
deleteboot: Boot file /nfs/objyserv1/u3/ec/stage/objy/quarrie/BaBar
Federated Database "/nfs/objyserv1/u3/ec/stage/objy/quarrie/BaBar" has been removed.
Possible Errors
deleteboot: Boot file [/nfs/objyserv1/u3/ec/stage/objy/quarrie/BaBar] non-existent
deleteboot: BOOT_FILE not defined
deleteboot: This target is protected by the DEFINE_BOOTFILE_RULES flag
Return to Table of
Contents.
This target attempts to export the production
release FDDB and boot file to a globally-accessible location so that they may later be imported. It is only valid if the current FDDB and boot file
names are identical to the default values (BaBar.FDDB and BaBar respectively). This target
is automatically built on a gmake production or gmake
cold. The EXPORT_NAME flag may be used to override
the default name of the exported FDDB and boot file.
Example
> gmake database.export
export:
Boot file /nfs/objyserv1/u3/ec/stage/objy/quarrie/BaBar
FDDB /nfs/objyserv1/u3/ec/stage/objy/quarrie/BaBar.FDDB
AMS Server objyserv1.slac.stanford.edu
Lock Server objyserv1.slac.stanford.edu
FDDB ID 5700
Page Size 8192
Export Directory /afs/slac.stanford.edu/g/babar/dist/releases/5.4.1/database/export
Export FID 5799
Possible Errors
export: Boot file [/nfs/objyserv1/u3/ec/stage/objy/quarrie/BaBar] non-existent
export: BOOT_FILE not defined
export: Illegal boot file name - must be BaBar
export: Illegal FDDB name - must be BaBar.FDDB
export: Unable to perform export to /afs/slac.stanford.edu/g/babar/dist/releases/5.4.1/database/export - fatal error
Return to Table of
Contents.
This target just displays whether the current
federated database exists:
Example
> gmake database.exists
exists:
Boot file /nfs/objyserv1/u3/ec/stage/objy/quarrie/BaBar
exists: Boot file exists
Return to Table of
Contents.
This target just displays a list of valid
targets:
Example
> gmake database.help
Valid database targets are:
backup : Backup the current federated database & databases
cleanup : Attempt to cleanup locks etc.
config : Display current database configuration
deleteboot: Delete federated database & all databases[*]
exists : Report whether a federated database exists
export : Export the production release federated database
help : Display this help information
import : Import the federated database from the production release
load : Load an empty federation with an initial configuration
makeboot : Create federated database using current configuration[*]
reset : Reset database contents to state following a load
restore : Restore federatred database & databases from backup
status : Displays the current database transaction status
[*] These targets are protected by the DEFINE_BOOTFILE_RULES command-line switch
This target takes an optional flag, VERBOSE,
which extends the display to include the database-related flags.
> gmake database.help VERBOSE=true
Valid database targets are:
[remainder of display omitted]
Return to Table of
Contents.
This target creates a local copy of the FDDB
& bootfile from a previously exported production release
database. The default location of the production release database is:
$BFDIST/releases/$BFCURRENT/database/export
The default FDDB and boot file names are BaBarRef.FDDB
and BaBarRef respectively. This command will issue a warning
message if the destination FDDB already exists and no further action will be performed.
The IMPORT_NAME flag may be used to override the default name of
the exported FDDB and boot file to be imported.
Note that this target automatically removes any
files that have been generated from DDL files in an attempt to ensure that a subsequent
"gmake" will correctly rebuild both the libraries and create the schema in the
newly created federated database. However, it only does this for the current architecture.
Removing these files from other architectures must be performed manually. This limitation
is imposed because of the production build process.
Finally, this target automatically performs a load
action unless the BYPASS_LOAD flag is set.
Example
> gmake database.import
import:
Import Directory /afs/slac.stanford.edu/g/babar/dist/releases/5.4.1/database
Boot file /nfs/objyserv1/u3/ec/stage/objy/quarrie/BaBar
FDDB /nfs/objyserv1/u3/ec/stage/objy/quarrie/BaBar.FDDB
AMS Server objyserv1.slac.stanford.edu
Lock Server objyserv1.slac.stanford.edu
FDDB ID 1300
import: Attempting to copy import FDDB...
import: Attempting to copy import boot file...
Updating Name Service values...
Now updating System Name Space (catalog) values...
Now updating Database File locations...
Federated Database Installation complete.
import: Attempting to setup hosts & paths...
import: /nfs/objyserv1/u3/ec/stage/objy/quarrie/BaBar created ok
import: Removing database DDL generated files for this architecture.
You will need to use "rm tmp/$BFARCH/*/*_ddl.*" manually for others.
Possible Errors
import: Boot file [/nfs/objyserv1/u3/ec/stage/objy/quarrie/BaBar] already exists
import: BOOT_FILE not defined
import: Import boot file [/afs/slac.stanford.edu/g/babar/dist/releases/5.4.1/database/export/BaBarRef] non-existent
import: Import FDDB [/afs/slac.stanford.edu/g/babar/dist/releases/5.4.1/database/export/BaBarRef] non-existent
[this section incomplete]
Return to Table of
Contents.
This target loads an empty federation with an
initial configuration.
Example
> gmake database.load
load:
Boot file (BOOT_FILE) /nfs/objyserv1/u3/ec/stage/objy/quarrie/BaBar
load: Initializing Event Store with default authorizations...
load: Initializing Conditions Database with default authorizations...
load: Loading Conditions Database using /afs/slac.stanford.edu/u/ec/quarrie/release/5.6.1/BdbCondLoaders/BdbCondLoad.tcl
load: Initialization & loading complete
Possible Errors
[this section incomplete]
Return to Table of
Contents.
This target attempts to create a new empty FDDB
based on the current configuration parameters. This target is protected by the DEFINE_BOOTFILE_RULES
flag that must be specified on the command line or as an environment variable. Great care
should be taken when accessing this target. Its use is generally not necessary since the
boot file is automatically created by importing from the reference boot file.
Note that this target automatically removes any
files that have been generated from DDL files in an attempt to ensure that a subsequent
"gmake" will correctly rebuild both the libraries and create the schema in the
newly created federated database. However, it only does this for the current architecture.
Removing these files from other architectures must be performed manually. This limitation
is imposed because of the production build process.
Example
> gmake database.makeboot DEFINE_BOOTFILE_RULES=yes
makeboot:
Boot file /nfs/objyserv1/u3/ec/stage/objy/quarrie/BaBar
FDDB /nfs/objyserv1/u3/ec/stage/objy/quarrie/BaBar.FDDB
AMS Server objyserv1.slac.stanford.edu
Lock Server objyserv1.slac.stanford.edu
FDDB ID 1301
Page Size 8192
Federated Database "/nfs/objyserv1/u3/ec/stage/objy/quarrie/BaBar" created.
makeboot: Removing database DDL generated files for this architecture.
You will need to use "rm tmp/$BFARCH/*/*_ddl.* manually for others
Possible Errors
makeboot: Boot file [/nfs/objyserv1/u3/ec/stage/objy/quarrie/BaBar] already exists
makeboot: BOOT_FILE not defined
makeboot: This target is protected by the DEFINE_BOOTFILE_RULES flag
[this section incomplete]
Return to Table of
Contents.
This target resets the FDDB to the state it was
in immediately following an import. All existing database files are deleted and the
federation is loaded with the initial state information. Note that any schema that might
have been created following the import are retained. The BYPASS_LOAD
flag may be set to present the loading of the initial configuration.
Example
> gmake database.reset
[remainder of example missing]
Possible Errors
[this section incomplete]
Return to Table of
Contents.
This target restores the FDDB from a previous
backup.
Example
> gmake database.restore
[remainder of example missing]
Possible Errors
[this section incomplete]
Return to Table of
Contents.
This target dumps the schema from the current
federation into the export directory.
Example
> gmake database.schemadump
[listing of files being backed up omitted]
backup: Backup complete
Return to Table of
Contents.
This target loads the schema from the inport area
(by default this is the export area of the underlying production release).
Example
> gmake database.schemaload
[listing of files being backed up omitted]
backup: Backup complete
Return to Table of
Contents.
This target displays the status of any
transactions for the current federated daatbase. It is useful in conjunction with the cleanup target to first determine whether any
transaction locks exist before executing the cleanup command.
Example
> gmake database.status
status:
Boot file (BOOT_FILE) /nfs/objyserv1/u3/ec/stage/objy/quarrie/BaBar
[listing transaction status omitted]
Return to Table of Contents.
There are several other database-related makefile
flags that modify the action of the makefiles.
AVOID_COLLISIONS. This
flag is used when simultaneous builds of the same federation are attempted from multiple
machines corresponding to different machine architectures. By default, an informational
message is emitted each time a DDL compiler collision is detected and one of the builds is
forced to back-off and retry. The maxmum back-off period is 60 seconds, after which the
process will attempt to override the collision detection and continue, at which point
another informational message will be emitted. Setting the AVOID_COLLISIONS
flag to no will disable this collision avoidance scheme, and
setting the flag to silent will disable information messages.
This flag is designed for use by the production release coordinator and should not
generally be needed by normal developers.
gmake cold AVOID_COLLISIONS=silent
BYPASS_DB_EXPORT. This
flag is used in conjunction with the production orcold
targets of the main makefile. Normally the final phase of these targets (which are
normally used for production releases) involves exporting the database so that it is
available for later importing by developers. This default may be disabled by specifying
this flag. This flag is designed for use by the production release coordinator and should
not generally be needed by normal developers.
gmake cold BYPASS_DB_EXPORT=yes
BYPASS_DB_IMPORT. This
flag is used in conjunction with the all (or default) target of
the main makefile. These targets normally attempt to import the production release
database into the developer release, but this default action may be disabled by specifying
this flag. Note that this is not a requirement for successful completion of these
makefiles. If the developer database already exists an information message will be
reported and the make will continue. This flag is designed for use by the production
release coordinator and should not generally be needed by normal developers.
gmake BYPASS_DB_IMPORT=yes
BYPASS_DB_MAKEBOOT.
This flag is used in conjunction with the production or cold
targets of the main makefile. These targets are normally used to create a full release. A
full release creates a new reference boot file by default, but this may be disabled by
specifying this flag. This is useful if the build failed for some reason and another
attempt is made from the same release area. This flag is designed for use by the
production release coordinator and should not generally be needed by normal developers.
gmake cold BYPASS_DB_MAKEBOOT=yes
gmake production BYPASS_DB_MAKEBOOT=yes
DEFINE_BOOTFILE_RULES.
This flag must be set in order for the makeboot and deleteboot
targets to be enabled. These targets are meant to be used by experts only and should not
normally be used.
gmake database.makeboot DEFINE_BOOTFILES_RULES=yes
gmake database.deleteboot DEFINE_BOOTFILE_RULES=yes
EVOLVE_SCHEMA. This
flag enables schema evolution during the DDL compilation phase. Schema evolution is
disabled by default. The implications of schema evolution within the BABAR environment are
not yet fully understood so this flag should only be used by experts until further notice.
The syntax requires that the flag be set to the list of classes that are to be evolved.
gmake "EVOLVE_SCHEMA=MyClass1 MyClass2 Myclass3"
EXPORT_NAME. This flag
is used to override the default name of the exported FDDB and boot file (BaBarRef.FDDB and
BaBarRef respectively).
gmake export EXPORT_NAME=BaBarBase
IMPORT_NAME. This flag
is used to override the default name of the exported FDDB and boot file (BaBarRef.FDDB and
BaBarRef respectively) to be imported.
gmake import IMPORT_NAME=BaBarBase
PRODUCTION_RELEASE.
This flag is used internally by the makefiles to identify that the reference boot file
etc. should be created rather than a user-level boot file. It is set automatically by the production
target of the top-level makefile. This flag is designed for use by the production release
coordinator and should not generally be needed by normal developers. The following are
equivalent:
gmake production
gmake cold PRODUCTION_RELEASE=yes
USE_OO_FD_BOOT. If this
flag is set, and the $OO_FD_BOOT environment variable is also
set, it will be used to establish the default values of the BOOT_FILE_DIR
and BOOT_FILE_NAME flags, overriding the global defaults.
Furthermore, if the specified boot file exists, the contents of the file will be used to
override the defaults for AMS_HOST, LOCK_HOST, FD_NUMBER. Note
that these new defaults can themselves be overridden by a site-specific or user-level
custimization file.
gmake database.import USE_OO_FD_BOOT=yes
Return to Table of
Contents.
Site-specific customizations have been setup via SiteConfig/$BFSITE/SoftRelTools/arch_spec_FDDB.mk+
for the following sites:
LBNL
AMS_HOST. This defaults
to online04.lbl.gov
FDB_DIR_BASE. This
defaults to /home/babardata/databases.
LOCK_HOST. This
defaults to online04.lbl.gov.
________________________
SLAC
AMS_HOST. This defaults
to objyserv1.slac.stanford.edu
FDB_DIR_BASE. This
defaults to /nfs/objyserv1/objy/databases/users
LOCK_HOST. This
defaults to objyserv1.slac.stanford.edu.
Return to Table of
Contents.
One possible scenario for developers is for them
to be working on multiple different federated databases, perhaps corresponding to
different base releases, or different modifications to packages under development. In this
case the.bbobjy file within each test release should be modified
to establish the separation between the different federated databases. Normally the
developer will only be working on a single federated database at a time so a single FDDB
identifier can be used.
One possible strategy for locating these
different databases is illustrated by the following fragment taken from a.bbobjy
file:
FDB_DIR = $(FDB_DIR_BASE)/$(FDB_DIR_USER)/$(BFCURRENT)/V1
This will create the federated database files in
theV1 subdirectory of the/<username>/$BFCURRENT
directory under the site-specific database directory tree. Other development federated
databases corresponding to the same release should be directed to e.g. theV2
subdirectory etc. Other releases will be handled by the $BFCURRENT
variable that is derived automatically from the underlying base release.
Note that it is possible that such a directory or
set of directories will need to be created explicitly by the developer, rather than
automatically by the makefiles. In this case the directory access should be set to 775 via
the chmod command, (or chmod g+w).
Return to Table of
Contents.
This information is contained in the Section
"Run-time Environment Variables" of the "Getting
Started with the BABAR Database"
document.
Return to Table of
Contents.
Aborting a running gmake
using ^C when it is in the middle of performing DDL compilations can also cause the
federation to become locked. The recovery procedure is described in the document "Getting
started with the BABAR Database".
In general it is bad idea to manually delete a
federated database (FDDB) file, boot file, or individual database files. Use the provided
makefile tools (e.g. gmake database.deleteboot) or
standard Objectivity tools. If any of these files are deleted manually, it is possible
that the federated database not only becomes unuseable, but that the federated database ID
also becomes unusable. Whilst a database expert might be able to recover the FID, in the
worst possible case a reboot of the Objectivity lock server process might be necessary.
This is an action that carries a large overhead with it since it impacts all other
developers and needs to be scheduled well in advance. Developers should ask for help from
one of the database
developers if they get stuck.
Return to Table of
Contents.
- For some operations the makefiles attempt to
create directories as well as manipulate the federated database files. This means that
they need direct access to the filesystem that the federated database files reside on. A
directory daemon is under development that will remove this restriction and allow the
desired directories to be created remotely. A copy of this daemon will need to be run on
each AMS server machine alongside the AMS server itself. However, until this daemon has
been released, the restriction will remain that the database filesystem will need to be
directly accessible from the machine from where the makefiles are executed.
- Database files should not reside on NFS mounted
file systems unless the NFS server machine also has a resident AMS Objectivity server.
This restriction also applies to the output of a DDL complation (the xxx_ddl.cc &
xxx_ref.hh files). In particular, thetmp/$BFARCH/<package>
directories should not be redirected by a symbolic link to a NFS mounted volume unless
there is a corresponding AMS server.
- Whilst the makefiles will create databases in AFS
space, they will only be fully accessible from the machine from which the makefiles were
executed. This is due to an incompatibility between Objectivity and AFS.
- Refer to the " BDBExperience with
Objectivity/DB" document for more problems.
Return to Table of
Contents.
DB
Home | BaBar Home | Computing | Online | Reconstruction
| Simulation
| Search
 DRQuarrie@LBL.Gov
|
