The Design of the BABAR
Event Store

David R. Quarrie

Version Information

Draft: 29th June 1997

This document is still under development. If you have any questions or comments, please address them to the author.

Table of Contents

• Introduction
• Authorization Levels
• Processing Stages
• Events
• Collections
• Dictionaries
• Registry
• Applications
• Clustering Strategy
• Directory & File Organization
• Federated Database Management
• Access to data from Reconstruction Applications
• Data Distribution Strategy
• Data Logging Strategy
• Useful Quantities
• References

Introduction

This document describes the global design of the BABAR Event Store, which is a component of the BABAR Database Management System (BDMS). It introduces the concepts and terminology that describe the components of the system, together with some design guidelines.

The BABAR Event Store is a domain of the BDMS, having the domain name events.

Return to Table of Contents.

Authorization Levels

Authorization levels are discussed in detail in the context of the BABAR Database Management System. An application can be executed in one of several authorization levels. The level determines certain access rights to the contents of the database and may restrict the ability of the application to update the contents. Available authorization levels are:

• System. An application operating at this level has full control over the event store, allowing it read, update, create and delete privileges.
• Group. An application operating at this level has read-only access to the event store apart from databases and contents corresponding to a specified group. Groups are intended to correspond to physics analysis work-groups and similar. Information from the other authorization levels and other groups may be replicated within the context of the current group.
• User. When operating at this level, the application has read-only access to the event store apart from databases and contents corresponding to a specified user. Users correspond to UNIX accounts [SLAC AFS Accounts?]. Information from the other authorization levels and other users may be replicated within the context of the current user.

Authorization levels are domain-specific, and an application may execute at one authorization level for one domain (e.g. Conditions database access) and at another level within another domain (e.g. Event Store access).

By default an application operates at User authorization level for this domain, granting it restricted access. It may request a higher level of authorization for this domain, but this will only be successful if previously granted by the database administrator. Note however, that it is always granted read access to all the event store information.

Return to Table of Contents.

Processing Stages

Experimental data (events) are assumed to flow through a set of processing stages that may either make selections, or add information to the event, or both. Major processing stages are:

• Simulated event generation. This process results in the generation of fake raw data and truth information for each event. The raw data corresponds to the information that would have been generated by the online system, the truth information to the underlying physics, absolute positions and energies etc.
• Experiment event generation. This process results in the generation of true raw data and corresponds to the output of the data acquisition and online system [ignoring for now Prompt Reconstruction].
• Event Reconstruction. This process attempts to derive physical quantities such as tracks, energy clusters and vertex information from the raw data.
• Data Compression. This process, which might be part of the reconstruction or a separate stage, selects events based on certain criteria and produces a condensed set of information for them. This is assumed to comprise information in enough detail to do event display, generate analysis objects and redo most of the reconstruction.
• Physics Analysis. This performs further selections and detailed analysis based on the underlying physics data (particle types etc.).

Further processing stages might well be necessary, but these are the basic ones that form the concept.

Although the above discussion introduces the various processing stages, it does not present a complete picture of the processing than an event may undergo. In particular, it is highly likely that an event will be reconstructed several times, typically with different configuration parameters reflecting the better understanding of the detector and physics goals. Data compression and physics analysis are also likely to occur multiple times. Each of these processing stages can optionally overwrite the earlier output data, or preferably, generate a new version of the output data. Each version must be identified by both the input and configuration data that was used.

Data versioning will be used to manage the different versions of output data, with back-pointers connecting the output data to the corresponding input data. This allows a default version to be established and accessed. Other versions may be accessed by explicit reference, or may optionally be deleted.

The connection between the output data and the configuration parameters has not yet been designed. Two options exist:

1. The output data is identified by its creation date & time and the configuration parameters are identified by a validity date and time range. The correlation is therefore based on matching the creation date & time with the appropriate time interval, as is already done by the conditions database.
2. A direct object reference is created from the output data to the corresponding configuration data. This has the advantage of being faster than the index retrieval of 1.
3. A combination of 1. & 2. This has the advantages of consistency with the conditions database whilst also offering the optimal access.

I suggest that we tentatively adopt 3.

Return to Table of Contents.

Events

Events are represented as a hierarchy of objects based upon an event header. This header contains references to multiple child objects, each corresponding to a particular processing stage as described earlier. The information for each processing stage will itself typically be organized as a stage header with references to further child objects. In addition, summary or tag objects are designed to allow for rapid but simple tests to be made on a small subset of the attributes of the children objects without having to access the latter directly. A single event tag is proposed, together with stage tag objects at each of the stages. There is a trade-off between performance and complexity of such queries. Extremely high performance queries can make selections based on the event tag, while less performance but more complexity can result from accessing the stage tags and finally the data from each stage. This strategy can result in dramatic performance gains if appropriate tags are created.

The types of information that can be associated with each event are the following:

• Simulated data (sim). This is specific to events that have been generated as a result of Monte Carlo simulations, and contains the equivalent of the raw data and the underlying physics, absolute position and energy information (truth).
• Raw data (raw). This is the information that is available at the output of the online system, having been processed within the front-end electronics and possibly within the Online Event Processing (OEP) farm.
• Reconstructed Data (rec). This is the output of reconstruction. It includes tracks and calorimeter clusters etc.
• Event Summary Data (esd). This is a compressed form of the reconstructed data. The terminology is taken from Atlas, and corresponds approximately to DST data in a conventional file-based organization. It is assumed to comprise information in enough detail to do event display, generate analysis objects and redo most of the reconstruction.
• Analysis Object Data (aod). This is end-user physicist data (particle types etc.) that is used for final physics analysis. The terminology is again taken from Atlas.
• Tag Data (tag). This is a highly compressed summary that classifies the event and allows simple selection queries to be performed without accessing other information.
• Event Header (hdr). The event header contains references to the other information in the event. It is designed to be as compact as possible.

Note that this is not expected to be a complete set of event components. The structure should be designed to be extensible, allowing for new components to be added to the event in a manner that minimizes schema evolution.

A simplified view of the event object hierarchy is shown below.

Note that the event header, the stage headers & the various tag objects must be located separately from the bulk of the data that makes up the event in order to take advantage of clustering prefetching etc.In order to allow for separate migration to the mass store, these should be in separate databases files rather than separate containers within the same databases.

A very rough estimate of the sizes of this information is:

My working assumption is that the stage headers are as small as possible apart from containing references to their data objects, and that the stage tags are typically 1-10% of the size of the corresponding stage information.

Return to Table of Contents.

Collections

Event collections contain references to events within the event store. Multiple types of collections are supported, although they all exhibit the same abstract interface. The multiple collections are designed to accommodate both small (limited to less than 10^6 events) and large (allowing up to 10^12 events) collections of events, as well as collections that organize events according to the run number as derived from the online system.

As noted above, all event collections obey the same abstract interface, which is the mechanism by which new events can be added to an existing collections, or by which all the events within the collection may be accessed. One restriction is that these collections do not yet support the concept of iterators and that therefore only a single iteration within a process for a particular collection is supported. This limitation might be removed following evaluation of the prototype.

Event collections are persistent-capable and care should be taken in creating them. A standard mechanism is provided for creating transient collections that can later be migrated to being persistent. However, such collections are restricted in their size. The prototype does not enforce a size restriction.

Return to Table of Contents.

Dictionaries

Dictionaries are a mechanism by which a persistent event collection may be assigned a name and may then later be accessed by that name. Multiple dictionaries can exist, corresponding to the different context levels described above, and to the various groups and users. Thus there is a single System level dictionary that can be updated only by applications running at that context level, but may be accessed by any other database application. A single dictionary may further be defined for each defined group and user, that can only be updated by that group or user.

In order to facilitate database management activities, all persistent event collections will automatically be assigned an internally generated name and will be associated with the corresponding dictionary depending on the context of the application. The application-specified man replaces this automatically generated one. This approach ensures that all persistent event collections can always be located by scanning through all available dictionaries.

Each dictionary has a separate name space and thus collection name clashes can only occur within the same dictionary. An event collection may appear in several different dictionaries under the same or different names.

Typical processing modes for an event store application are the following:

• Locate an input event collection using an appropriate dictionary. Iterate over the events in this collection, building another output event collection of events that satisfy the desired selection criteria. This new collection may itself be used as the input collection for a further selection process, or may discarded and the selection process repeated with different selection criteria. Eventually a new output collection will be created that contains all the events of interest. This collection may be registered within the same or another dictionary for later access.
• Locate an input event collection using the appropriate dictionary. Iterate over the events in the collection, processing them and adding more information to each event. This new information might comprise a high level summary allowing later rapid event selection in addition to the detailed information that has been added. In order to add additional information to an event, the application must have the appropriate authority at the current context level. This implies that the event headers must be updatable at the current context level.

Return to Table of Contents.

Registry

A registry of all dictionaries is maintained for management purposes. This enables the current set of dictionaries to be located, and thence all the persistent collections to be located.

Return to Table of Contents.

Applications

All applications using the BABAR event store access information via a single BdbEventStore class. This provides access to the registry and dictionaries that themselves provide access to named collections of events. In addition it performs transaction and error reporting management as well as statistics gathering. The BdbEventStore class is implemented as a singleton and thus provides a single instance through which all management must be performed.

Applications can simultaneously access any of the BDMS domains using the appropriate domain-specific application classes.

Return to Table of Contents.

Clustering Strategy

The goal of the clustering strategy is to optimize access to event data. This is particularly important in a situation where much of the data resides on tape rather than disk. This will be true for BABAR, although the use of a migrating file-system (as provided by HPSS) will automatically bring accessed files into the disk cache if they are not already present.

There are several components to the strategy:

• The number of database file opens should be minimized. This results both from the intrinsic overhead of opening a database file, but also because of the possibility that the database file resides on tape.
• Once a database file has been opened, as much data as possible should be accessed from it.
• The OODBMS is page-based, allowing the possibility of pre-fetching small objects into the memory cache.
• Data that is more likely to be accessed should reside in a separate database file than data that is less likely to be accessed.

The implications of this strategy are:

• The data for each stage (raw, sim, etc.) should be kept in different databases. It is unlikely that a single application will wish to access all the event information and much more likely that an application will wish to access only a subset.
• The data for each stage (raw, sim, etc.) should be located in separate databases from the header & tag. The header is more likely to be accessed than either the tag or data since it acts as the portal to them. Since most accesses to the header will probably result in accesses to the tag, it is less clear that these should be separated.
• The event header & tag should be located in different databases from the stage headers & tags. This provides optimal access for hierarchical queries where most information is located in the tag.

Since the goal is to allow simple queries to operate purely on the event tag, it might make sense to give the tag an interface that allows access to the remainder of the event and bypass use of the event header as a portal to the tag. This interface would be implemented by delegation to the event header, there being a di-directional association between them. The advantage of this is that a collection of event tags would be the most compressed representation of events and avoid the space and time overhead of the event header.

Return to Table of Contents.

Directory & File Organization

Directory Tree

$BDBROOT/ -+- events/ -+- system/ -+- dic/ -+- dictionary.bdb
                       |           |
                       |           +- raw/ -+- raw000000/ -+- raw000000.bdb
                       |           |        |              |
                       |           |        |              +- [...]
                       |           |        |              |
                       |           |        |              +- raw0000FF.bdb
                       |           |        |              +
                       |           |        |              +- rawhdr000001.bdb
                       |           |        |              |
                       |           |        |              +- [...]
                       |           |        |
                       |           |        +- raw000100/ -+- raw000100.bdb
                       |           |        |              |
                       |           |        |              +- [...]
                       |           |        |
                       |           |        +- [...]
                       |           |
                       |           +- rec/ -+- rec000000/ -+- rec000000.bdb
                       |           |        |              |
                       |           |        |              +- [...]
                       |           |        |
                       |           |        +- rec000100/ -+- rec000100.bdb
                       |           |        |              |
                       |           |        |              +- [...]
                       |           |        |
                       |           |        +- [...]
                       |           |
                       |           +- sim/ -+- sim000000/ -+- sim000000.bdb
                       |           |        |              |
                       |           |        |              +- [...]
                       |           |        |
                       |           |        +- [...]
                       |           |
                       |           +- esd/ -+- esd000000/ -+- esd000000.bdb
                       |           |        |              |
                       |           |        |              +- [...]
                       |           |        |
                       |           |        +- [...]
                       |           |
                       |           +- aod/ -+- aod000000/ -+- aod000000.bdb
                       |           |        |              |
                       |           |        |              +- [...]
                       |           |        |
                       |           |        +- [...]
                       |           |
                       |           +- tag/ -+- tag000000/ -+- tag000000.bdb
                       |           |        |              |
                       |           |        |              +- [...]
                       |           |        |
                       |           |        +- [...]
                       |           |
                       |           +- hdr/ -+- hdr000000/ -+- hdr000000.bdb
                       |           |        |              |
                       |           |        |              +- [...]
                       |           |        |
                       |           |        +- [...]
                       |           |
                       |           +- col/ -+- col000000/ -+- col000000.bdb
                       |                    |              |
                       |                    |              +- [...]
                       |                    |
                       |                    +- [...]
                       |
                       +- groups/ -+- group[i]/ -+- dic/ -+- dictionary.bdb
                       |           |             |
                       |           |             +- raw/ -+- raw000000/ -+- raw000000.bdb
                       |           |             |        |              |
                       |           |             |        |              +- [...]
                       |           |             |        |
                       |           |             |        +- [...]
                       |           |             |
                       |           |             +- [...]
                       |           |
                       |           +- group[j]/ -+- dic/ -+- dictionary.bdb
                       |           |             |
                       |           |             +- [...]
                       |           |
                       |           +- [...]
                       |
                       +- users/ --+- user[i]/ -+- dic/ -+- dictionary.bdb
                                   |            |
                                   |            +- raw/ -+- raw000000/ -+- rawoo0000.bdb
                                   |            |        |              |
                                   |            |        |              +- [...]
                                   |            |        |
                                   |            |        +- [...]
                                   |            |
                                   |            +- [...]
                                   |
                                   +- user[j]/ -+- dic/ -+- dictionary.bdb
                                   |            |
                                   |            +- [...]
                                   |
                                   +- [...]

Database File Details

Subdirectories
    raw Raw data
        Name: rawNNNNNN.bdb
        Length: Fixed Length [~20GB]
            Broken into fixed length containers of ~1GB
        Raw Headers & Tags
        Name: rawhdrNNNNNN.bdb
              rawtagNNNNNN.bdb


    rec Reconstructed data
        Name: recNNNNNN.bdb
        Length: Fixed Length [~20GB]
            Broken into fixed length containers of ~1GB
        Rec Headers & Tags
        Name: rechdrNNNNNN.bdb
              rectagNNNNNN.bdb


    sim Simulated Truth data
        Name: simNNNNNN.bdb
        Length: Fixed Length [~20GB]
            Broken into fixed length containers of ~1GB
        Sim Headers & Tags
        Name: simhdrNNNNNN.bdb
              simtagNNNNNN.bdb


    esd Event Summary Data
        Name: esdNNNNNN.bdb
        Length: Fixed Length [~1GB]
            Broken into fixed length containers of ~1GB
        What's the data organisation - by event or by class?
        ESD Headers & Tags
        Name: esdhdrNNNNNN.bdb
              esdtagNNNNNN.bdb

    aod Analysis Object Data
        Name: aodNNNNNN.bdb
        Length: Not yet specified
        Data organisation by class (first guess)

    tag Tag data
        Name: tagNNNNNN.bdb
        Length: Not yet specified
        Data organisation by event (single object per event tag)

    dic Dictionary
        Contains appropriate dictionaries and any other book-keeping
        databases
        Name: dictionary.bdb
              [others?]

    col Event Collections
        Contains event collections
        Name: colNNNNNN.bdb
        Length: Not yet specified

    hdr Event Headers
        The Event Header is the object that has references to the other
        processed objects - tag, aod, esd, rec, raw, sim (if appropriate).
        Name: hdrNNNNNN.bdb
        Length: Not yet specified

In the above, NNNNNN denotes the Sequence Number in Hex.

Notes/Issues

• Goal is to spread database files across subdirectories in a manner that allows a human being to identify approximately what's where, to accommodate access controls and to allow self-consistent partial clones of the database to be created & distributed. - Files should have meaningful names (or at least some meaningful component of the name) so that a human being can infer the contents & correct location.
• Directory & file names - all lowercase apart from Group directories (determined by actual group name) and tlaIndex.bdb files in conditions database tree. - Unambiguous
• All database files are of type .bdb
• Sequence Number. These are just incremented by one each time a new fixed length file of the same basic name is created. Sequence Numbers are local to a processing stage & authorization level and start at zero (0000). How the sequence numbers are maintained is yet to be determined (another database?).
• I've assumed the use of fixed length files for much of the data. The file sizes need to be optimized for each data type. For example, raw, sim & rec need to be large files in order to accommodate the 60-100TB per year requirement without exceeding the Objectivity 64k file total limitation. Taking 20GB as an example, this results in 640TB for 50% of the total file number space, approximately 10 years at our quoted data requirements. Note that this implies a filesystem supporting 20GB file sizes. DEC does, what about IBM/Solaris/HP? See section below. Data files for other quantities can generally be much smaller than the above. What is optimal?
• Replicated/compressed data are stored either in the same directory with another file name, or the appropriate group/user directory of the corresponding context level. i.e. If user "bloggs" replicates the aod data for a subset of events, then they appear in a file in the bloggs/aod directory.

Deriving the creation location

The following derivation tree is implied for the creation clustering hint

                     BasicHint
                                       |
            +--------------------------+---------------------+
            |                          |                     |
       ConditionsHint             EventsHint            OnlineHint
            |                          |
          [tla]                    [Context]
            |                          |
     +------+------+        +----------+------------+
     |      |      |        |          |            |
   tla[i] tla[j] tla[k]  System      Group         User
            |               |
        [Sequence]       [Family]
                            |
              +------+------+------+------+------+------+-----+-----+
              |      |      |      |      |      |      |     |     |
             raw    rec    sim    esd    aod    tag    col   hdr   dic
              |      |      |      |      |      |      |     |     |
         [Sequence]     [Sequence]   [Sequence]     [Sequence]      |
              |                                                     |
     +--------+                                              dictionary.bdb
     |        |
raw000000 raw000100
              |
         [Sequence]

Return to Table of Contents.

Federated Database Management

The FDDB contains both the schema (class definitions) and the database catalog (locations etc.). It's imperative that this be well managed and protected against accidental corruption or deletion.

In general, once a significant amount of data is loaded into a FDDB, it should be "hard" to change the schema. Objectivity supports several strategies for schema migration and conversion:

• Treat attempted changes to the schema (as identified by the DDL compiler) as errors and cause the compilation to fail. This is the default.
• Allow changes to the schema to be made. Existing objects then have to be converted to "fit" their new shape. This can be done in one of several ways:

  Deferred: Objects are converted one at a time as they are accessed and will only be converted if accessed in update mode.

  On-demand: Objects are converted automatically at one of several granularities (container, database, FDDB).

  Immediate: Depending on the schema changes (e.g. replacing a base class), a stand-alone upgrade application might need to be run before other application can access the new schema.

Further considerations are that two developers attempting to run the DDL compiler against the same federated database may come into conflict such that one of their compilations fails because it can't lock the FDDB for exclusive use during the course of the compilation.

Another factor is the concept of multiple partitions where one site is designated as being the primary partition and contains the primary FDDB and data, and other sites (secondary partitions) may have local copies of only some of the databases within the FDDB, and may make extensions to the primary schema without affecting the primary FDDB. The secondary partitions act as write-back caches such that if requested data is available locally it is directly accessed, otherwise it is fetched from the primary partition. Similarly, if data is updated then it will automatically be propagated back to the primary partition (and to other secondary partitions) unless it is purely local to the secondary partition.

The present (very preliminary) strategy for managing the FDDB and allowing developer access is the following:

• Maintain (at least) two copies of the primary FDDB. One (the Reference FDDB) contains just the reference schema; the other (the Production FDDB) contains both schema and the production data.
• New releases of the BABAR software are compiled against the Reference FDDB, normally disallowing any schema migration.
• After some sort of reconstruction/analysis review, modified schema are accepted and accommodated in a new release by appropriate DDL compile switches, and a object conversion strategy is put in place. It's possible that the mechanics of this might involve building against both the Reference & Production FDDBs.
• When developers create new test releases (via newrel -t) they get their own copy of the Reference FDDB (Developer FDDB). Subsequent development is performed against this developer-specific FDDB and this bypasses any lock conflicts with other developers. Database files may be imported from the Production FDDB (a management activity that Objectivity/C++ supports). Any new data will be entirely local to this Developer FDDB. Any schema migration is entirely local and does not impact the Production FDDB. One disadvatage of this scheme is that it is difficult to propagate data back to the Production FDDB.
• An alternative developer scheme puts them each in a separate secondary partition. This would allow them to propagate back compatible data to the primary FDDB. This needs to be thought through in more detail. This use of secondary partitions is not entirely compatible with our use of it as a mechanism to distribute data at the Institution level.

The present design supports within the Production FDDB both work-group and user-based databases and event collections in adition to the global system ones. This concept is not yet integrated with the developer view outlined above.

Return to Table of Contents.

Access to data from Reconstruction Applications

The basic access model is that there will be named collections of events which are accessed through one of several dictionaries. There is one System-wide dictionary, several Group wide dictionaries and many User-specific dictionaries. Thus there can be named collections at the System level representing particular physics data samples (and obviously the complete event sample), similar collections at the group-level (e.g. for each physics working groups) and at the User level. In general these collections need not contain any data, they just contain references to the event headers of the selected events. However, it may be advantageous to duplicate part of the information for the events in order to improve repeated access to that data.

Components of the strategy

• Database Input Module (BdbEventInput). Allows selection of input event collection.
• Database Output Module (BdbEventOutput). Allows additional information to be added to the events.
• Tools to allow selected parts of the selected events to be duplicated in order to improve repeated access. [these don't yet exist]
• etc.

Return to Table of Contents.

Data Distribution Strategy

[This section missing]

Return to Table of Contents.

Data Logging Strategy

[This section missing]

Return to Table of Contents.

Useful Quantities

Federated database capacity vs. database file size

A maximum of 65535 databases files is allowed in an Objectivity/DB Federated Database. The following table assumes that 50% of these database files are small and therefore make little contribution to the total capacity. Thus the following table corresponds to 32767 database files of the specified size.

Number of files for a nominal year's worth of data (10^7 secs)

Number of files for a weeks worth of data (604800 secs)

Time taken to fill a fixed-length database file at various writing rates

0.25 MB/sec writing rate
     1GB =   4000secs =   66mins 40secs =  1hrs  6mins 40secs
     2GB =   8000secs =  133mins 20secs =  2hrs 12mins 20secs
    10GB =  40000secs =  666mins 40secs = 11hrs  6mins 40secs
    20GB =  80000secs = 1333mins 20secs = 22hrs 13mins 20secs
    50GB = 200000secs = 3333mins 20secs = 55hrs 33mins 20secs

    1 MB/sec
     1GB =  1000secs =   16mins 40secs
     2GB =  2000secs =   33mins 20secs
    10GB = 10000secs =  166mins 40secs =  2hrs 46mins 40secs
    20GB = 20000secs =  333mins 20secs =  5hrs 33mins 20secs
    50GB = 50000secs =  833mins 20secs = 13hrs 53mins 20secs

    2.5 MB/sec
     1GB =   400secs =   6mins 20secs
     2GB =   800secs =  12mins 20secs
    10GB =  4000secs =  66mins 40secs = 1hrs  6mins 40secs
    20GB =  8000secs = 133mins 20secs = 2hrs 13mins 20secs
    50GB = 20000secs = 333mins 20secs = 5hrs 33mins 20secs

    10 MB/sec
     1GB =  100secs =   1mins 40secs
     2GB =  200secs =   3mins 20secs
    10GB = 1000secs =  16mins 40secs
    20GB = 2000secs =  33mins 20secs
    50GB = 5000secs =  83mins 20secs = 1hrs 23mins 20secs

Return to Table of Contents.

References

1. Object Database Management Group, The Object Database Management Group Standard, 1993
2. ATLAS Collaboration, ATLAS Computing Technical Proposal, CERN/LHCC 96-43, Dec1996.

DB Home | BaBar Home | Computing | Reconstruction | Simulation | Search

DRQuarrie@LBL.Gov