Geant4-based simulation in BaBar
Requirements for the BaBar simulation
The long term BaBar simulation design must meet a number of requirements, including
- The capability to model the detector geometry, materials, magnetic field, particle interactions and decays, and detector response with a high degree of accuracy commensurate with the precision physics goals of BaBar.
- Physics of particle interactions in matter that is trusted, proven and sufficently complete in the domains relevant for BaBar.
- Efficient utilization of CPU such that the high statistics required can be attained with the available resources without compromising required levels of simulation detail.
- Flexible architecture with selectable levels of simulation detail at the subsystem level such that the modelling detail matches the requirements of the analysis, and doesn't waste resources and unnecessarily increase the turnaround time to deliver the
required statistics.
- Architectural compatibility with the rest of the BaBar software infrastructure. Concretely, this means an object-oriented architecture based on C++.
- Modular design compatible with highly distributed development and subsystem development efforts with a high degree of autonomy.
- Support for a migration path from our present Geant3.21-based simulation that utilizes limited manpower efficiently, avoids redundant effort by supporting dual-use code, and is based on a single development effort rather than two largely
decoupled efforts. Central to this migration plan is keeping the Geant3.21 based simulation the fully-featured production simulation until physics performance of the new simulation is fully verified.
The current detailed simulation, BBSIM, based on Geant 3.21, fails on a number of these requirements. While Geant3's modelling capabilities and physics of particle interactions are for the most part trusted and of high quality, there are areas of weakness
important to BaBar, such as hadronic interactions; because Geant 3.21 is frozen no improvements can be expected other than those we might implement ourselves. It is very difficult to incorporate variable levels of detail to use CPU efficiently while
benefiting from the flexibility, simplicity, easy cross-checking and many other advantages of supporting selectable detail levels in a single simulation framework. As a Fortran program it is not architecturally compatible with with the rest of the BaBar
software infrastructure. While fairly modular, the absence of the control over data access and interfaces imposed by C++ and OO is keenly felt.
For all these reasons and no doubt more, it has been widely (if not universally) accepted within BaBar that in the long term we should move beyond our present simulation.
Geant4: Background and motivations for BaBar adoption
An early decision to be addressed was what underlying simulation package (if any!) should be used for our final simulation. At the Sept '95 Collaboration Meeting the BaBar computing group proposed, to no objection, that Geant4 be pursued as the basis for
the final production simulation for BaBar. Geant4 is a C++ object oriented simulation toolkit under development by an international collaboration centered at CERN, led by the team that developed and now maintains Geant3. It is a complete redesign, rewrite
and extension of Geant3, designed to meet the simulation needs of next-generation experiments including LHC, the B factories, heavy ion experiments, and others. The project was initiated in the fall of 1994 and will produce a full public release at the
end of 1998. A version that reproduces the functionality and physics performance of Geant3 is due in early 1997. That version will not be a wide release, but will be offered only to a short list of collaborators and friends of the project well-positioned
to critically evaluate the package and participate in its improvement. Following its public release Geant4 will be maintained and developed by a maintenance team centered at CERN, as was the case for Geant3 for many years. Geant3 has been frozen (at
version 3.21) since the initiation of the Geant4 project; all development effort is going into Geant4. A key Geant4 design precept is continuous comparison of performance and physics with Geant3 throughout Geant4's development, and the comparisons made to
date will be discussed.
An introduction to the Geant4 project and a survey of the design and prototyping activity in the project's first year, including the important results of the Geant3-Geant4 benchmarks conducted with the prototype in 10/95, is found in the project's status report for 1995 and, more briefly, in a CHEP95 conference paper. Activity in
1996 and the project's status with respect to its next milestones is discussed below, and further recent news can be found at the web site of the Aug '96 Geant4 Workshop at TRIUMF.
The original motivation for the adoption of Geant4 as presented to the collaboration in Sept '95 is given on this slide presented at that meeting. Geant4 was adopted
because it is the only OO simulation toolkit likely to emerge as a full-featured, mature, proven, well-supported physics tool on a timescale relevant to BaBar. A simulation program based on OO and C++ is preferred for consistency with the rest of the
BaBar software base which is object oriented and written in C++, driven by the Collaboration's belief that OO technology and the C++ language are the best foundation on which to build a consistent, robust and easy to maintain software environment. These
choices also enable leveraging the popularity of C++ in industry, and the consequent familiarity of students with the language (students today have no interest in learning Fortran, and forcing it upon them is a disservice). There is simply no other OO
simulation toolkit that could present a viable alternative to Geant4 without major investment on the part of BaBar in development of the toolkit itself, (eg. developing Gismo or starting from scratch) whereas adopting Geant4 allows us to leverage a major
international project led by the team with the accrued experience of Geant3's development. Simply remaining with Geant3 is not a viable option because the program is frozen, and real improvements in Geant valuable to BaBar are going into Geant4. We don't
want to find ourselves locked into a five year old, obsolete simulation package at startup. To summarize some other strengths of Geant4:
- It is committed to including the full physics and feature set of Geant3
- Development of the physics is led by some of the same experts responsible for Geant3 electromagnetic and hadronics (Michel Maire and Laszlo Urban in electromagnetics; Hans Fesefeldt in hadronics), and seeks to improve upon the Geant3 implementation
based upon accrued experience in the development and application of Geant3
- Very important for us, a group at TRIUMF is focusing on low energy hadronic interactions (below 3 GeV) and improving upon Geant3's Gheisha
- The schedule is compatible with availability of a full public release by BaBar startup
- A smooth migration path from Geant3 is provided
Geant4 independently adopted many of the tools and technologies since adopted by BaBar, and making Geant4 very compatible with the BaBar software base. The project shares BaBar's approach of following (de facto) standards to focus manpower on core
development, not peripheral tools, and leverage developments in industry. Key tools in use are the Rational Rose CASE tool, CVS for code management among distributed developers, support for both native C++ and g++ compilers on all of BaBar's platforms,
gmake for automated code building, the Tools.h++ class library, Objectivity-based OODB event storage (from the sister CERN project RD45), OpenGL/Open Inventor based graphics, the LHC++ toolkit (successor to CERNLIB), and (not considered by BaBar to date)
the PVM parallel library.
Concerns with Geant4
Geant4's timescale is the principal concern. The scheduled date for a full public release, end '88, essentially coincides with BaBar startup.
Some have made the point that it took several years for the original Geant to mature to the point that it became widely accepted as HEP's standard simulation tool. What if the same happens to Geant4? An important consideration here is that Geant4
development is led by the CERN group that developed and has supported Geant for many years. Group membership and leadership has of course changed over time, but CERN's attention to continuous support and improvement of Geant over the years has ensured
that the group (including many non-CERN Geant3 developers involved in Geant4) has retained and accumulated a deep knowledge and experience base that is being applied in Geant4's development. Also, continuous comparison of performance and physics with
Geant 3.21 throughout the development of Geant4 is one of the foremost requirements of the project, and is being followed. What is released at end '88, if the project is minimally successful, will be an object oriented toolkit with the capability and
performance of Geant 3.21. If the project is a true success, reaching the goals of its developers, it will improve substantially upon Geant 3.21 in areas including tracking performance, quality of the electromagnetic and hadronic interaction physics in
energy ranges including those of interest to BaBar, greater flexibility in defining geometries and materials, fast simulation capability, powerful yet carefully controlled provision for incorporation of user code, modularity and transparency of operation
and physics implementation, etc.
The timeline of the project combines an aggressive schedule to reach the full functionality of Geant 3.21 and a conservatively long period of almost two years following that milestone to refine the design, address problems, extend Geant4's capability
and arrive at a robust releasable product by end '88. The early '97 milestone to match Geant 3.21 functionality will provide a good basis for BaBar to assess the progress of the project and the likelihood of its success. At the same time it will enable
BaBar to reproduce the BBSIM simulation in Geant4 and begin detailed performance and output comparisons, to be continued and extended over the subsequent two years such that by the time of the formal public release BaBar has an advanced and well tested
Geant4 implementation, with design and performance problems having been addressed with the Geant4 developers prior to release.
This implies that BaBar be an 'insider' on Geant4 development and testing prior to release of the final product, and indeed this is a central tenet of BaBar's approach to Geant4. Our adoption of Geant4 would be inconceivable without substantial design
and implementation level involvement on the part of BaBar during development. BaBar members have been active collaborators on Geant4 since the early days of the project, and we hold the US seat on the collaboration's management board. BaBar involvement is
largely focused in areas of particular concern to us such as design and implementation of the basic Geant4 analogues of Geant3 geometry and materials, early implementation of Geant3 to Geant4 geometry conversion, and design and implementation of fast
simulation capability (parameterisation and fast tracking). A BaBar collaborator (John Allison) also leads visualization development. The number of BaBar collaborators involved has been too few, however; only John and Torre until recently. The Geant4
prototype has recently progressed far enough that a concerted BaBar effort in implementing prototype fast and full simulations has begun, and our activity in the project is broadening. A group at Ecole Polytechnique Paris (Gerard Bonneaud, Marc Verderi,
Christophe Thiebault, Paulo Mora de Freitas) has joined Geant4 to participate in the development of BaBar's Geant4 simulation and in the design and implementation of BaBar-critical Geant4 components such as fast simulation capability. Bill Lockman, a
principal BaBar simulation developer, has joined to participate initially in Geant3 to Geant4 geometry conversion. Bob Jacobsen and other members of the reconstruction team have also profitably injected design suggestions from a reconstruction perspective
(and bug fixes).
An important concern is whether the Geant4 project is sensitive to the requirements of BaBar and other pre-LHC potential users and collaborators. This concern is addressed further in the discussion of the 1996-97 milestones below. Conversations with
Geant4 management and, more importantly, their actions in terms of responsiveness to our design inputs, requirements and timeline, and the inclusion of BaBar on the management board, suggest that they are sensitive to our requirements. It is in the
interest of the project, CERN, and the LHC experiments for Geant4 to be widely used and exercised early, and they seem to recognise this.
However confident we are that Geant4 can meet our needs on our required timescale, it would be very imprudent to rely on Geant4 coming through, and we have no intention of doing so. The Geant3-based detailed simulation BBSIM will continue to
be developed and will remain the production simulation until the physics performance of the Geant4 simulation is verified in comparison with Geant 3.21 and beam test data, and speed, robustness and quality assessment are adequate for production. Dual
development of Geant3 and Geant4 based simulations implies careful attention to how to achieve this without duplication of work, bifurcation of the simulation effort, or slippage of either effort in an environment of perpetually low manpower. This is
addressed at length in the discussions on C++ migration and code reuse.
Geant4 design highlights relevant to BaBar
- CAD compatibility. It will be possible to read the detector geometry directly from STEP-compliant CAD programs into Geant4 and cross-check the simulation of the CAD model (with reduced performance, of course) against the simulation model (eg. material
profile)
- Automatic Geant3 geometry importing. This is an essential capability discussed elsewhere that permits detailed BBSIM geometries to be imported into Geant4 without recoding.
- Volume clash detection. Geant4 will automatically detect overlapping volumes and so can flag clashes signalling placement and dimension errors.
- User definable 'physics' processes. In quotes because the very flexible provisions for user definition of processes to be invoked during stepping extend beyond physics processes proper; eg. the fast simulation capability is implemented as a
'generalized process' and our new approach to DC simulation with a non-Geant analytic model of cells and wires could be cleanly implemented as a generalized process. Geant4's design requirements of user accessibility to the physics and transparency of its
implementation will be important, for example, if we come to implement our own treatment of pion interaction in the ECAL and IFR.
- Event level (and sub-event level) parallelism. Needed to exploit large parallel resources available in the collaboration.
- Visualization of detector and events with high-level graphics (eg. Open Inventor).
Unified (fast to detailed) simulation
A principal design goal of the final BaBar simulation is to support in a single unified simulation environment a selectable range of detail for the detector subsystems, from fast through fully detailed, with the possibility to tune the detail at the
subsystem level to the needs of a particular analysis. A study focusing on vertexing, for example, may select fully detailed treatment of the tracking systems and parameterised outer detectors. There are many advantages to such an approach:
- efficient utilization of CPU. Use of coarser parameterised detectors or fast tracking where appropriate saves the full simulation time and reconstruction time for those components.
- Easy cross-checking of the consistency of results at varying detail levels; a study can be repeated swapping one representation for another with no other changes to the simulation environment.
- Using the Geant4 framework for fast simulation exploits Geant4 strengths in inter-volume tracking: years of experience have gone into the development of very refined optimizations implemented in Geant4 that can be exploited to efficiently propagate
between parameterised detector envelopes. The overheads of the framework must be low; last year's benchmark for rectilinear propagation of geantinos in a 2500-element calorimeter (many more volumes than a fast simulation) yielding timing of about 2msec
per event; adequate, and to be revisited in the current prototype and once magnetic field tracking is implemented.
- Implementation of parameterisations is straightforward, using the detailed treatment of the detector in the same environment, and the same detector envelope, to generate the parameterisation.
- A detector group interested in implementing an extremely refined simulation for detector studies is not constrained by the speed impact on the simulation; the default can be a faster, less refined version.
Key capabilities of Geant4 (non-existent in Geant3) support such a unified simulation:
- Support for multiple detector representations. Different representations -- some detailed with hits generated as detector response objects, and some parameterised with reconstruction/analysis objects (eg. tracks, particle ID probabilities)
generated for the detector response -- coexist, represented as different logical volume structures, with the one actually placed in the physical detector being used during simulation. Representations can be changed even within an event, at the track
level; for example to invoke a parameterised treatment of the calorimeter everywhere except for tracks near the edges, for which the detailed model is used to model the lack of containment correctly.
- Support for the association of a parameterised response object with an active (or inactive) detector as an alternative to the conventional sensitive detector plus hits/digis.
- Support for detecting the existence of a parameterised response object associated with a volume upon entry to that volume, and passing control for track handling within that volume to the parameterisation.
- Support for the parameterisation code returning any number of particles (eg. a relocated primary or a number of uncontained shower secondaries emerging from the back of a calorimeter) to Geant4 at the exit of the parameterised system for propagation
to the next system
BaBar is leading the development of this capability and is the principal driver for new and amended designs of the affected Geant4 components. The Ecole Polytechnique group is active in this as will be reported in the discussion of the prototype.
The present focus is to implement and prototype the capability for parameterised simulation and selectable detail levels. Then will follow the implementation of an Aslund-level fast simulation with the objective of developing an Aslund replacement; as
for the full simulation, Aslund must continue to be maintained until the transition is possible. Further gradations of detail will be motivated by the requirements of the physics and subdetector groups. A number of people have expressed interest in this
work -- in contrast to Geant3 based work it has the appeal of working with a new capability using modern software and a package of key future importance in HEP, and working close to the physics in that fast simulation will continue to be an important
physics tool. So, if people follow through, the manpower should be there.
The prototype BaBar Geant4 simulation: Bogus
A prototype Geant4 based simulation is under development, called Bogus (BaBar Object-oriented Geant-based Unified Simulation) and based on the current Geant4 prototype. Its initial objective is to implement and exercise the key design elements of Geant4
required to support a unified simulation and to exercise the BaBar-developed tools for migration from Geant3 to Geant4, dual-use code, and coexistence of BBSIM and an OO simulation with as much commonality and as little redundance as possible.
The commonality starts at the inputs to the simulation, the geometry database and the event generator. The present BaBar geometry database uses an in-house package dbio (and a precursor, dbin) which supports the definition of a language-independent
structured parameter database (based on ascii files) and generates all the code required to read and access the database from either Fortran or C++. Thus the fully detailed databases developed over the last three years for BBSIM are automatically
available to a C++ simulation with no recoding, and with no additional coding overhead as the database is modified. dbio is discussed further elsewhere. Also because a single data repository, the ascii database (.db) files, serves both Fortran and C++
simulations, consistency is assured. The Bogus prototype reads the geometry database via dbio and makes use of C++ member functions generated by dbio (discussed elsewhere), which provide implementation-independent access to the geometry parameters, as the
basis for building C++ geometry classes that provide the geometry representations needed to build Geant4 models of the detectors. At present only a few of the detectors (IFR, EMC, DCH) are represented by simple geometry classes, and they are represented
crudely by cylinders (until recently the only geometry, apart from boxes, available in Geant4).
The other input to the simulation is the event generator. Bogus presently uses internally generated particles, either non-interacting or real, via a simple particle gun. The capability to input STDHEP format events via the /HEPEVT/ common has just been
implemented in Geant4 which will permit us to interface our BEGET generator used by both BBSIM and Aslund to Bogus.
Bogus is now being used as the testing ground for the implementation of parameterised detector representations and multiple detector representations. We are iterating with the Geant4 developers on the design modifications to Geant4 needed for these.
Discussions over the last three months with the Geant4 team have led to a very clean low-overhead design utilising the 'generalised process' approach to stepping actions, but detailed study is uncovering further needed modifications.
We are operating very much at the 'leading edge' of Geant4 development. The full Geant3 geometries just became available and we have integrated them in the Geant3 to Geant4 converter and are working on bringing up a full BBSIM Geant4 geometry in Bogus
and the Framework. Tracking/stepping is nearing the end of an intensive design cycle that with our input will include the parameterisation capability. This close and early involvement is valuable both to BaBar, to exercise and assist the design and make
sure it meets our requirements early, and to the project, to have the design implications of requirements like parameterisation addressed now and so avoid a design iteration later.
Geant4 project highlights and status
Geant3-Geant4 benchmark
Following a period of defining requirements and iterating on analysis and design of the principal project components, development of a Geant4 prototype began in earnest in summer '95. The first year milestone in October '95 was benchmarking this prototype
against Geant3, and was very successful. Performance was compared on pure geometry calculations, 'geantino' tracking (no interactions), and muon tracking (1 TeV muons with E loss, bremsstrahlung, delta rays, and pair production but no tracked secondaries
or magnetic field). The tested geometry was a hex array of 50x50 cylinders making up a calorimeter. Benchmarks were run on the CERN IBM SP-2 and cited gain factors are Geant3/Geant4 timing ratios.
- Comparing pure propagation through the geometry with minimal overheads, Geant4 outperformed Geant3 by factors of 2.2, 4.1, 3.8 (for three orthogonal track directions through the geometry). Factoring out algorithm improvements in Geant4 the gain factor
was 2.1.
- Comparing geantino propagation (event and track processing overheads and user action invokations added in), gain factors were 1.7, 3.1, 3.0 with an average gain of 2.4 factoring out the geometry algorithms.
- Comparing 1 TeV muons Geant4 again outperformed Geant3 by a factor 1.6 (direction independent, since physics dominates the timing). Factoring out the geometry, the gain factor comparing physics algorithms only was 1.45.
Thus by every measure, Geant4 prototype performance substantially exceeded Geant3. This was verified on another platform (HP) with architecture (good integer performance) and compilers (crummy C++ compiler) favoring Geant3 over Geant4. A detailed study of
the origins of the speedup uncovered culprits that lend confidence to the conclusions. The Zebra memory management overhead was found to be substantial in Geant3, and user-defined types in Geant4 built-in to the C++ language allowed much greater scope for
compiler optimization than is available to Fortran dealing with data structures defined via Zebra's huge equivalenced arrays.
The benchmark result alleviated the concerns of some that OO/C++ may be inherently handicapped in performance relative to Fortran. It increased confidence in the project (among participants as well as outsiders!), led to excellent financial support in
year two, and was a factor in an expansion of involvement and manpower from circa 16 FTEs in year one to circa 25 FTEs in year two.
Activities and objectives in year two
The year two milestones are:
- Release a first working version with geometry, tracking, electromagnetic physics and high energy hadronic physics components at least equivalent to Geant 3.21 for the simulation of events in LHC detectors.
- Incorporate some I/O capabilities from RD45 (Objectivity-based OODB event storage).
- Limit the release to physicists working closely with the Collaboration in evaluation and testing.
- Users should be able to determine easily and, when necessary, modify the data and assumptions used in the simulation of physics processes (user accessibility and transparency requirement).
The second item clearly reflects an LHC bias; fortunately developments since the milestones were laid down make the situation look a lot better. A large group at TRIUMF has undertaken to focus on low energy (from about 3 GeV down) hadronic physics, and
their internal milestone is to complete Geant 3.21-equivalent Gheisha functionality in Geant4 at the same time.
Survey of principal '96 activities and status:
- Design and implementation of particles, materials and hits/digitizations is nearly complete. A first implementation of persistent hits/digis (with RD45) is scheduled for end October.
- Design and implementation of the physics processes is complete, and the first implementations of electromagnetic and hadronic physics processes using the design are in place. Comparisons between Geant3 and Geant4 show good agreement.
- Analysis and design of the visualization and user interface is complete; a rough prototype of the former and a working version of the latter are in place.
- Implementation of the Geant3 geometry solids is complete (all those used by BaBar, at least), but MANY is not yet implemented.
- Physical volume classes needed to implement Geant3 positioning techniques are in place
- G4PVReplica for GSDVNx divisions
- G4PVParametrised for parametrised divisions (negative PAR() parameters
- GSPOSP implemented using conventional physical volume placement G4PVPlacement
- positioning using full displacement/rotation transformation available
- Interface to event generators (via the /HEPEVT/ common) is done
- The G3toG4 converter supporting all implemented Geant3 geometries is in place and under test
- The implementation of user action and parameterisation capability is being finalised with BaBar's participation.
- After a major improvement in the design (more flexible 'generalized processes'), the stepping design is almost final; implementation work remains.
- The major project for the remainder of the year will be implementation of magnetic field tracking, discussed further below.
- Testing procedures and quality assurance are an ongoing project.
After some months of serious concern over progress in the geometry, which fell behind schedule when the principal implementer left the project, the schedule is back on track (demonstrated by the completion of the Geant3 geometries, urgently needed by
BaBar) following a final productive visit by the expert and the work of a new person.
The principal concern now is magnetic field tracking. Several full C++ implementations of Runge-Kutta propagation in a magnetic field are in place, but the big problem to solve is how to efficiently determine the distance over which to propagate in a
given step based on geometry limitations, and how to perform the stepping both efficiently and without missing any volumes. Several approaches have been identified and the plan is to implement a number of them by the end of the year (manpower is in place)
and hopefully find a clear winner. By this schedule, we will have some version of working magnetic field tracking by the end of the year.
Development timeline for BaBar's Geant4 simulation
- Bogus prototype initiated 7/96
- Mechanism for parameterisation support established (TRIUMF workshop) 8/96
- Ecole Polytechnique Paris joins development effort 9/96
- Detailed BaBar/Geant4 design discussions at CERN decide implementation approach to parameterisations 9/96
- Iterations with Geant4 on design requirements 10/96
- Fast simulation 'design complete' milestone 12/31/96
- Full simulation prototype functionally complete 2/3/97
- Developers release of full Geant 3.21 functionality in Geant4 3/1/97
- Fast simulation functionally complete; general design complete 9/30/97
- Fast simulation deployment 12/1/97
- Production OO simulation functionally complete 4/30/98
- Deployment 9/30/98
- Public release of Geant4 ~11/98
|