Printer friendly version of the document
Differences from the old Condition/DB test installations
Setting up a test CDB
Updating an existing condition
Creating a new condition
Modifying Persistent Schema
Updating a federation with more recent snapshot
This document will help those who need to establish and manage his/her own private test CDB. The document complements "Case 3" of a basic installation procedure described in BaBar HOWTO software package. To get some ideas on this procedure please read first any the following documents:
A procedure description from the HEAD OF CVS can be found here.
A procedure description specific to your current test release can be found by printing the following plain text file: $BFDIST/release/$BFCURRENT/HOWTO/HOWTO-database-importing".
The rest of the current document is essentially a step-by-step guide covering the topics missing in the HOWTO package. A reader is also encouraged to look into the [OVERVIEW] of the CDB to familiarize him/herself with main concepts and vocabulary used in CDB.
The management commands presented in this document require at least the following development release:
13.5.0 and up
At the time this document was being written none of the production release contained the "take over" procedure required in section [UPDATE] below. However with the exception of the [UPDATE] scenario any production release beginning from 12.5.0 and up should be okay.
If you're not familiar with the old Condition/DB then just skip this section. If you're - then reading through the rest of this section will help you to avoid possible confusions.
The new CDB differs in many ways from the old one. This includes the conceptual model, general design, API, API implementations, to count just a few. Perhaps it would be easier to say what are the similarities rather than differences. However in the current context (of installing a test CDB) the major differences are found in a part of the new CDB design which provides a foundation for distributed CDB installations. Unlike the old Condition/DB the new design from the very beginning defines CDB as a distributed database. In this paradigm parts of the distributed CDB are originating at various specially installed and initialized federations. These parts in a CDB jargon are called origins:
Please don't mix origins with federations! Each federation may have artifacts from more than one origin!
Each CDB federation participating in a distributed CDB is always associated with some origin. This origin becomes a native origin of the federation. In contrast, other origins are said to be the foreign ones to that federation.
There is another distinction between origins based on their types. CDB supports tree different types of origins: MASTER, SLAVE and TEST. Depending on a type of its origin a federation may play different roles in a distributed CDB. Some origins allow a federation to contribute its persistent artifacts into a distributed CDB (other CDB federations associated with other origins). Others may only receive artifacts from other origins. The artifacts include conditions, persistent configurations, etc. See [OVERVIEW] for a complete list and descriptions of those.
Eventually due to data exchange process going on between federations partitcipating in a distributed CDB each federation will have its own collection of artifacts associated with its own (native) origin and a (not necessarily complete or up-to-date) collection of artifacts from other (foreign) origins. The foreign artifacts when they are brought into a particular federation automatically become read-only in this federation. In CDB only a federation associated with a native origin of a particular artifact can be used to modify the artifact. For example, if a condition named /emc/EmcFooClassP were created in the IR2 CDB (which is of SLAVE type) then it could only be updated in IR2 CDB. Other CDB-s will prohibit (at the level of CDB API and management tools) any local modifications to this condition.
In order to glue persistent artifacts from various origins into a particular CDB federation CDB design has special provisions, namely: local namespaces at the level of each origin and a global registry of origins in MASTER CDB (only one MASTER type federation is allowed in a distributed CDB installation).
Here is a list of differences following from the new design:
Difference #1: installing and initializing a federation
To participate in a distributed CDB a federation has to be properly installed, loaded and initialized. Now it's not sufficient enough just to load a federation with some snapshot. One also needs to establish a proper CDB infrastructure in this federation to associate it with the corresponding origin. Here we can see the first operational difference of the new CDB from the old Condition/DB. The next [SETUP] section will explain how to perform the initialization in case of a test CDB federation.
Difference #2: updating existing conditions loaded with a snapshot
Another difference is related to the above mentioned feature of the artifacts ownership. Since conditions loaded into a test CDB with an initialization snapshot do not belong to the test CDB then they can't be modified in this federation either. The solution described later in this document is to create a local copy of a condition in question in a local collection of the test CDB. The corresponding [UPDATE] section below will describe it in detail.
Difference #3: creating a new condition non-existing in a loaded snapshot
Unlike the old Condition/DB a new condition is not being automatically created when a loader application is trying to load condition objects into this non existing condition. The condition creation is an explicit management procedure in CDB. By separating this procedure from the loading of new condition objects into a condition CDB provides database managers (you, as an owner of you test CDB, are not an exception) with greater flexibility on how to plan the database. See the description of the conditions creation procedure at [CREATE] below.
Other differences are less essential in the current context. Please read the conceptual documents on CDB to understand all possible implication of using new CDB in test installations.
Although at a first glance all the mentioned differences may seem to be just inconvenient that's not quite true in reality. The main benefit of this design is that it preserves the contents of the original snapshot. It also preserves conditions from accidental overwriting. Later on a test CDB can easily be reset into its original state w/o bringing a new snapshot. Another advantage works in an opposite direction. It comes with a concept of incremental snapshots which are possible in new CDB design. In case of a test CDB it means that the foreign part of this federation can be updated with a smaller increment to bring new constants w/o affecting the native part of the federation. In effect this looks like a concept of the test "sandbox". The rest of the document is devoted to describing how to prepare and how to manage this sandbox to solve specific problems.
The next diagram is illustrating what will be in your test CDB after you finish setting it up:
This is actually an interactive diagram if displayed in a Web browser. Each box in this picture represents the corresponding origin filled with its local artifacts. Just click onto a box to get the origin's description. Blue boxes represent foreign origins. The only red (the bottom one if you've printed this document on a gray scale printer) box is the native origin of your test CDB. This is where your locally created persistent objects will go into.
Pleas read the [BABAR SETUP] document to get more information on the BaBar CDB installation.
As it has been mentioned in the [INTRODUCTION] section, a test federation has to be installed as it's described by the "Case 3" scenario of the the basic installation procedure. The only important part on the CDB side of this procedure is the following installation parameter:
This parameter defines a range of Objectivity/DB database identifiers which are going to be used to store data locally created in the federation. Technically this parameter can be specified either in the .bbobjy file or in a command line when creating the federation. The default value of the parameter is "default". This value should NOT change because CDB relies on it when initializing the test CDB infrastructure. This is just a warning. Normally you don't have to do anything if you don't have this parameter in the .bbobjy file and not passing it in the command line when creating the database.
If you're not sure what's the actual value of this parameter then after you federation had been created you may check the parameter's value using the following command (which will print "default" in this example):
BdbCondDataDistr dbidrange NAME default
If you see something else then recreate your test federation. Otherwise it won't be able to function as a test CDB and the currently presented recipe will be not applicable.
Once a federation has been installed and loaded with an appropriate snapshot then it has to be initialized as a test CDB. This is done by the following command (to be run just once):
CdbManager initialize_test "This is my test CDB installation."
This command should not print error or warning messages if the federation was properly installed and loaded. The last parameter of the above shown initialization command is a description of this CDB installation. You're free to put any arbitrary string into this description. The value of the description won't affect the functionality of the CDB.
The next step is to create a local view, which will be used by your applications to access conditions in the database. Without this view they (the applications) won't be able to access any existing conditions (brought in with a snapshot at the initialization step). The view is also needed to register locally created conditions or copies of those from other origins of the loaded snapshot. Once created, this view will become the default view of your test CDB. Usually a typical user requirement when creating a test CDB is to establish a database which would look (respond) like a MASTER CDB. If this is a case (this document does not discuss other scenarios) then the local view is created by copying the most recent view found in the MASTER origin of the loaded snapshot. The following command will do this:
CdbManager -v "MASTER::<recent>" clone_view "first" "This is my first local view."
And again this command has to be run just once. The output view will be called "first" in the scope of the test CDB. It will get an arbitrary description specified through the last parameter of the command.
To check if the view has been successfully created the next command can be used:
CdbBrowser views [-long] ... NAME="TEST::first" ID=2::1 STATUS=NOT-FROZEN,DEFAULT NAME="TEST::main" ID=2::0 STATUS=NOT-FROZEN ...
The names of views owned by your test CDB are prefixed with the 'TEST" keyword. Each view also has an identifier, made of two fields: an origin and a local view identifier in the scope of its origin. One of these views will be marked by the special "DEFAULT" keyword. You may also notice that there are actually two views in the namespace of this CDB. The one called "main" is a special view created and maintained by the CDB API itself. See the corresponding reference documentation on the CDB API implementations for details. An optional "-long" switch can be used to print more information on views. By the way, this is where you'll see the description of the above created view.
At this point your applications would be able to read (Note, just read!) conditions from the test CDB as if they were running against the MASTER CDB. The next sample command would illustrate how to get a list of all known conditions in your test CDB:
CdbBrowser conditions / [-long] ...
This will produce a long list of conditions. The list will be even longer (and of course, more interesting) when an optional "-long' switch is specified.
To check the quality of the installation you may pick your favorite condition and print its intervals by:
CdbBrowser objects /emc/EmcFooClassP ...
The last remaining step in the setting up of the test CDB is to create a cluster of (for) local conditions. This is where the actual conditions (either newly created or copies from other origins) will be stored. If the above used "CdbManager initialize_test" command were used to initialize the local namespace of the test CDB then a cluster creation command will create the actual "sandbox" for your local experiments with conditions.
Here is the corresponding command which has to be run just once:
CdbManager create_cluster "regular" "This is my local cluster for regular conditions." REGULAR
The above shown command has three parameters:
The name of the cluster. It's an arbitrary string identifying the cluster. I'd recommend to use "regular" unless you're planning to establish a very complex test CDB.
Then goes an arbitrary description of the cluster. Put here whatever you want. This information will appear in the output produced by related information query commands.
And finally, you have a predefined keyword indicating a type of the cluster. In case of test CDB installations it always has to be REGULAR. Other types are meant to be used for production CDB installations.
At this point the basic setup procedure is finished. Now depending on your task you may proceed to any of the subsequent sections.
NOTE: examples presented in this section are based on an assumption that we're going to update the following condition: /emc/EmcFooClassP.
The new CDB design requires that in order to update a condition in a test CDB the condition has to meet both of the following criteria:
it has to exist (be registered in a default view of the CDB)
and it has to belong to the test CDB (owned by its native origin called TEST)
A simplest check on the existence of the condition would be to try printing its intervals as it's shown in this example:
CdbBrowser objects /emc/EmcFooClassP ...
This command will produce quite obvious output, from which it would be clear whether the specified condition exists or not. If the condition does not exist then it has to be created following a recipe described in the next [CREATE] section. Just proceed to that section if this is going to be the case. Note that the creation procedure will by definition put new condition into the right (native) origin thus making the new condition be fully consistent with both requirements.
To check if an existing condition satisfies the second requirement we should obtain its origin using the following command:
CdbBrowser conditions /emc -long ...
and get to the detailed description of the condition in question. If this condition already belongs to the native origin of the test CDB then we're going to see something like this:
/emc/ | . +- EmcFooClassP | | # TYPE : REGULAR | # 'PHYSICAL' ID : 2::1 | # 'PHYSICAL' NAME : "TEST::EmcFooClassP" | # CREATED : 3210964641.0171287560 : Tue Oct 01 15:37:21 2002 (local time) 171287560 ns | # REGISTERED IN VIEW : 3223935491.0180720000 : Fri Feb 28 17:38:11 2003 (local time) 180720000 ns | # LAST TIME MODIFIED : 3210964646.0295509560 : Tue Oct 01 15:37:26 2002 (local time) 295509560 ns | # DESCRIPTION : "created automatically by the conversion procedure" .
A relevant part of the above shown output is highlighted by red color. It tells us which origin owns the actual ("physical") condition. In this example it's the TEST origin (which is the native origin for test CDB-s). Therefore both of the above mentioned criteria are met and the condition can be updated in this CDB. At this point just stop reading the current document and proceed to your loader (or whatever application you have).
If we see something else, like:
/emc/ | . +- EmcFooClassP | | # TYPE : REGULAR | # 'PHYSICAL' ID : 0::145 | # 'PHYSICAL' NAME : "MASTER::EmcFooClassP" | # CREATED : 3210964641.0171287560 : Tue Oct 01 15:37:21 2002 (local time) 171287560 ns | # REGISTERED IN VIEW : 3223935491.0180720000 : Fri Feb 28 17:38:11 2003 (local time) 180720000 ns | # LAST TIME MODIFIED : 3210964646.0295509560 : Tue Oct 01 15:37:26 2002 (local time) 295509560 ns | # DESCRIPTION : "created automatically by the conversion procedure" .
then it means that we can't update the condition. A solution in this case is to make a copy of the original physical condition, bring this copy into a local "sandbox" of the test CDB and re-point the current default view to associate the condition path as it appears in the view with the local copy. The following script will do this (it has to be run just once):
CdbTakeOverCondition.pl /emc/EmcFooClassP ...
At this point the condition is ready to be updated in the test CDB. Now you may want to repeat the "CdbBrowser conditions /emc -long" command to make sure that the origin of this sample condition has changed from MASTER to TEST.
Finally a few words about the limitations of this method. It's important to note (you can check it yourself) that in the current implementation of the "take over" procedure the resulting copy won't have all revisions of the old condition. But this should not be a problem for the majority of use cases for which test CDB federations are established. If you still think that preserving revisions information is essential for your particular application then you can get in touch with CDB developers following instructions explained in the [SUPPORT] section.
NOTE: examples presented in this section are based on an assumption that we're going to create the following condition: /emc/EmcFooClassP.
In order to be able create condition objects and put them into a certain condition the condition itself has to exist in a CDB federation. Otherwise a code attempting to write into the condition will fail (with self explaining diagnostics). In addition this condition must belong to the native origin of the local (test - in the current document) CDB. A simplest way to check if a condition in question is known to CDB is described at the [UPDATE] section presented above. Please check this section first to make sure that you really need to create a condition.
If you finally discovered that there is no such condition in the CDB then it can be created using the following three-steps algorithm:
creating a physical entity of the condition
tuning up this condition's configuration in the current default view
creating a parallel condition's configuration for legacy applications
Here is the command meant to create the physical condition:
CdbManager create_condition "/emc/EmcFooClassP" "This is my test condition." REGULAR CLUSTER "regular"
Please note that new condition will be created in the cluster called "regular". This cluster itself is supposed to be prepared at the CDB initialization stage presented at the [SETUP] section. In addition to the physical condition the above shown command will also create the corresponding entry in the current default view of your test CDB. Then this entry has to get a proper configuration as follows:
CdbManager autoconfigure_condition "/emc/EmcFooClassP" TOPMOST
The final step is to create and configure a parallel entry for this condition in the default view. The following two commands will do this:
CdbManager include_condition "/Emc/EmcFooClassP" "This is my test condition." "TEST::EmcFooClassP" CdbManager autoconfigure_condition "/Emc/EmcFooClassP" TOPMOST
The only difference of the parallel entry from the one initially created is that the detector name in the condition path begins with a capital letter. The problem is that in CDB a condition path is treated as a case sensitive string. Meanwhile some legacy BaBar applications are still relying on the old Condition/DB model in which detector name was always case insensitive. As a result these application will simply fail to find a condition. If you know for sure that you don't have this problem in your code then you may skip this step. Note that the parallel entry will still refer onto the same physical instance of the condition, so that any new objects stored through one (any) of these entries will be immediately seen through the other one.
Now you may check if both entries of the condition can be found in the current default view:
CdbBrowser conditions /emc -long /emc/ | . +- EmcFooClassP | | # TYPE : REGULAR | # 'PHYSICAL' ID : 2::1 | # 'PHYSICAL' NAME : "TEST::EmcFooClassP" | # CREATED : 3210964641.0171287560 : Tue Oct 01 15:37:21 2002 (local time) 171287560 ns | # REGISTERED IN VIEW : 3223935491.0180720000 : Fri Feb 28 17:38:11 2003 (local time) 180720000 ns | # LAST TIME MODIFIED : 3210964646.0295509560 : Tue Oct 01 15:37:26 2002 (local time) 295509560 ns | # DESCRIPTION : "created automatically by the conversion procedure" . CdbBrowser conditions /Emc -long /Emc/ | . +- EmcFooClassP | | # TYPE : REGULAR | # 'PHYSICAL' ID : 2::1 | # 'PHYSICAL' NAME : "TEST::EmcFooClassP" | # CREATED : 3210964641.0171287560 : Tue Oct 01 15:37:21 2002 (local time) 171287560 ns | # REGISTERED IN VIEW : 3223935491.0180720000 : Fri Feb 28 17:38:11 2003 (local time) 180720000 ns | # LAST TIME MODIFIED : 3210964646.0295509560 : Tue Oct 01 15:37:26 2002 (local time) 295509560 ns | # DESCRIPTION : "created automatically by the conversion procedure" .
To check what's in the actual configuration you may do:
CdbBrowser config /emc/EmcFooClassP --------------------------------------------- --------- -------------- --------------------- VALIDITY INTERVAL PARTITION USING REVISION REVISION-ID --------------------------------------------- --------- -------------- --------------------- -Infinity : +Infinity 0 Yes +Infinity --------------------------------------------- ---------- -------------- ---------------------
At this point the condition will be ready for use.
There are a few issues related to modification done to a persistent schema of a test CDB federation. The first one is about binary applications built against the test federation:
- Never ever run these applications against any production federation to load new objects into these federations! It's going to cause imminent object-class incompatibility problems resulting in (at best) crashes of production applications or (in a worse scenario) incorrect results produced by these (production) applications if they will try to use incompatible objects. This is in fact a general problem related to the persistent schema development in Objectivity/DB. The current document uses an extra opportunity to remind about this problem.
The correct algorithm of loading new objects of a new persistent class into production federations is described below:
You're free to make any schema modifications (in BaBar we only allow adding new persistent classes) to your test federation. You're also free to use any applications against your test federation.
When you finish developing and testing your new persistent class and the corresponding code (to create new objects of this class or read these objects from a databases) then make sure that your new code (including new class) gets into an official software release. When this release will be built then your new class will be added into an production schema.
Ideally your loader application has to be built as a part of this release. Then it can be run from the release to load new objects into production federations. That's the best way to deal with the new class.
If so some reasons the loader's binary is not a part of the official release then it can be built in a test release using the following precaution measures:
- When preparing to built the loader make a cleanup to your test release to remove any binaries, libraries and temporary directories out of the release (the best way is: "rm -r bin lib tmp; gmake installdirs"). Ideally a fresh test release has to be established.
- To avoid a chance of hitting the traces of the wrong development schema left after previous development activities never reuse an existing test federation. Just wipe out this federation using “gmake database.delete” and create a new one. Do not attempt to upgrade its schema.
Another problem is related to so called “Persistent Schema Evolution”. Briefly it appears when you're not satisfying with an existing persistent class (it does not have sufficient data members to store new information as it's need due to changed requirements, etc.). And at the same time you want to preserve the original name of a persistent class. The solution to this problem is described here.
Sometimes you may want to update your federation while preserving whatever you've created in the TEST are of the CDB. The current CDB implementation lets you to do so.