The report of the Gilchriese panel
----------------------------------

TECHNICAL REVIEW OF BABAR IMPROVEMENT PLANS

OCTOBER 11 - 12, 2000

INTRODUCTION
A technical review of the plans of the BaBar collaboration for detector
and computing improvements was held at SLAC on October 11 - 12, 2000.
The Charge to the Review Committee is given in Appendix A and the
composition of the committee in Appendix B.

The operation of PEPII has been extraordinarily successful and there is
every reason to believe that the performance of PEP II will steadily
improve in the next three years. The current luminosity is about 2 x
1033 cm-2sec-1. A peak luminosity of about 1034 cm-2sec-1 is expected to
be reached by 2003.

The BaBar Collaboration initiated a planning process in 1999 to
determine the impact on the performance of the BaBar detector and the
related computing of this rapid increase in luminosity. Possible
improvements to the BaBar detector were extensively studied and
documented for this Committee. The computing enhancements required to
cope with much larger data sets were also presented to the Committee.

The Review Committee was generally very impressed with the comprehensive
and thorough planning presented by the BaBar Collaboration. The detector
improvements proposed by the Collaboration are sound, although the
Committee has some specific concerns that are presented below.
Similarly, the enhancements to computing are also judged to be generally
well justified but the Committee has some specific concerns in this area
as well.

A general concern expressed by almost all subgroups in both the detector
and computing areas was the lack of manpower to simultaneously maintain
the detector, operate the computing/software infrastructure and do
physics analysis. The addition of responsibilities for significant
improvements will result in additional manpower requirements. The
Collaboration management has recently started a process to formalize
institutional and regional responsibilities via Memoranda of
Agreement(MOA). The Committee strongly endorses this process and notes
that the BaBar Collaboration is large (relative to comparable collider
experiments) and that effective utilization of existing manpower should
be possible.

OVERVIEW OF DETECTOR IMPROVEMENTS
The production of additional modules to replace the few non-working
elements of the SVT is well organized and on schedule to be completed by
2002. BaBar should weigh carefully the benefits of a major intervention
to replace the few dead SVT modules against the considerable risks of
such a major operation. The Committee recommends that
the current detector should be left in place for as long as possible,
subject to its continued satisfactory operation and the overall schedule
of major BaBar interventions.

The improvement plans presented for the DCH, EMC, DIRC, trigger, data
acquisition and online computing are well founded - see the detailed
sections later in this report. The Committee recommends proceeding with
the improvement plans as presented.

The IFR was identified by BaBar as an area of major concern. There is
substantial degradation of the efficiency of the RPCs. If this
continues, it is likely that muon identification will be fatally
impaired by the end of 2002. The degradation of the RPCs, which is
related to the high temperature of operation in the summer of 1999, is
not yet fully understood. Recent tests at elevated temperatures in the
US and Europe to attempt to understand the cause of the degradation have
given different results. Tests using small BaBar RPCs show evidence of
linseed-oil blobs that could be the cause of the reduced efficiency.
However, similar tests in Italy on newer chambers under similar
conditions do not yet show the same effect.  Endcap RPCs will be removed
in November and examined. New RPCs will be ready to install at the same
time. The Collaboration has established a working group to evaluate the
RPC status and future possibilities and to consider an alternative
scintillator option for the barrel region. The goal is to complete an
initial evaluation by mid-January 2001, leading to a decision to either
continue with new RPCs or replace the barrel RPCs with scintillator. It
is vital to maintain the existing RPC system as long as possible and to
take all reasonable steps to minimize further deterioration and maximize
the longevity of the system.

The Committee has the following recommendations for the next few months

1. The effort to understand the cause of the degradation of performance
of the RPCs should be strengthened immediately.
2. A detailed test plan, taking advantage of resources in the US and
Italy, should be developed now, well in advance of when chambers will be
removed in November.
3. Work on the scintillator option should move forward to present a
realistic plan by mid-January without diverting effort to understand the
barrel RPCs. Additional manpower may be needed.

The Committee also has recommendations that apply to the replacement of
the RPCs by new RPCs or scintillator, should this be necessary

1. The technical and management team providing the IFR detectors should
be strengthened. Quality control and careful schedule planning must be
key elements in an expanded team.
2. BaBar must make every effort to minimize the down time and risk that
will result from such a major intervention. This will require
considerable engineering coordination between the IFR-detector team and
the BaBar technical management.

OVERVIEW OF COMPUTING IMPROVEMENTS

The proposed improvements to computing were a major focus of the review.
BaBar has made great progress in developing the computing and software
infrastructure necessary to reconstruct and analyze data. Problems
existing at the start up of the experiment have been solved and physics
analysis has been completed very successfully. The general approach to
computing improvements proposed by BaBar appears sensible. A
comprehensive summary of the requirements to cope with the increased
luminosity was presented. However, an implementation plan to meet the
requirements does not yet exist.  Such a plan is under preparation in
the next two months. The costs associated with the yet-to-exist
implementation plan are very preliminary - too preliminary for the
Committee to review. The Committee recommends that a careful, bottoms-up
approach to estimating costs be completed along with the implementation
plan rather than scaling from immediate past experience. Such an
approach would allow an effective evaluation of the inevitable tradeoffs
between meeting requirements and cost to be made.

The Collaboration has proposed a model in which there are Tier A sites
(SLAC and IN2P3), Tier B sites (RAL and INFN(Rome)) and Tier C sites.
Close coordination among these different sites is essential. In
particular, there must be joint implementation of computing solutions
between the two Tier A sites, and these must be compatible with the
operation of the Tier B and Tier C sites.

BaBar should articulate a strategies to manage and optimize data access
for the whole spectrum of physics analysis activities with their
differing priorities and requirements. This should be presented to the
collaboration for feedback and the final strategy and implementation
approach agreed upon.

DETECTOR IMPROVEMENTS

The detailed reports of the Committee for each element of the detector
are given below.

SVT
Performance: The Committee was impressed by the successful overall
performance of the SVT and its contribution to the early BaBar physics
results.  They noted the significant long-term software alignment effort
that will be required to achieve the full tracking potential of the
experiment.

Spare Module Construction:  It is fully justified to have started the
construction of spare modules now while expertise,  facilities and
components are still available.  The program in place for completing the
work by 2002 seems realistic, provided no serious problems are
encountered with the rerun of the ATOM chip.

SVT Repair: The removal, re-configuration and re-insertion of the SVT is
a high risk operation and should be carefully considered.  Based on
present evidence there is no compelling reason to modify the SVT until
2004, or possibly later.  The Committee recommends that the current
detector should be left in place for as long as possible, subject to its
continued satisfactory operation and the overall schedule requirements
of major BaBar interventions.

Radiation Damage Monitoring: The radiation seen by the SVT is
non-uniform in phi and possibly also in z.  Radiation damage is a
potential concern for only a few of the layer 1 modules in the highest
radiation zones, and even these modules are likely to show small
degradation through to the end of 2004.  Data on radiation damage
presently come primarily from monitoring diodes.  The Committee
recommends that additional monitoring be implemented:
? Adding suitable current monitoring to some of the bias voltage
supplies to measure the time evolution of the detector leakage current.
? Performing a hit efficiency versus bias voltage scan for inner layer
modules at least annually.

Further Tests and Checks: The Committee noted the following points to
check in connection with performance after irradiation:
? The properties of irradiated detectors from both companies after
type-inversion should be measured as a high priority when connected to
readout electronics (CCE, strip quality, noise).
? Detectors should be irradiated under bias with alternate metal strips
grounded and floating (as in the inner layers) and their readout
performance confirmed.
? Bench test should be made of ATOM chip operation beyond 2Mrad ionizing
radiation.
? The radiation hardness of all materials and glues used should be
confirmed.
? The thermal management of the irradiated silicon in the SVT should be
checked.

Occupancy: As with other detector systems, the biggest impact of the
higher luminosity in the SVT may be through occupancy rather than
radiation damage.  The effects of added occupancy should be carefully
modelled, including realistic phi and z distributions.  The impact on
the DAQ and offline processing should be fully taken into account, as
well as vertex resolution and tracking efficiency.

DCH
The drift chamber is operating well and only minor improvements are
anticipated to ensure operation through 2004. Wire aging studies will be
performed over the next years in case mitigating action is needed,
although none is foreseen to be required before 2005 and possibly later.


DIRC
The DIRC is performing well although some additional work is planned in
the area of software improvements to obtain the particle identification
expected in the Technical Design Report. Electronics upgrades are
required to cope with the increased luminosity. Additional shielding
will also be added to reduce backgrounds. The Committee supports the
proposed electronics improvements(DFB/DCC clock upgrade and new TDCs)
and the proposed plan, cost and schedule appear to be in good shape.

Degradation in the response of the phototubes has been seen and traced
to corrosion of the phototube faceplates. There appears to be no
short-term or long-term risk of catastrophic failure. But it does appear
that all of the phototubes will suffer slow aging and reduction in
response. The impact of this on particle identification has been
simulated and is tolerable if the degradation rate does not increase. A
major intervention to replace phototubes is not foreseen and, in fact,
is believed to be more risky than allowing for a slow degradation of
performance. The Committee concurs with this but expects the
Collaboration will closely monitor the situation and continue to perform
accelerated and other aging tests to minimize the risk of additional
surprises.

Insufficient information was presented on future read-out R&D for the
Committee to provide advice at this time.

EMC
The EMC ?0 mass resolution is presently limited by electronics and
background noise. The design of the readout was always intended to use
digital filtering in the ROM. This has not been implemented due to a
lack of the appropriate personnel and the pressure to maintain stable
operating conditions during the collection of the first large data
sample. Unprocessed waveforms are being accumulated during data taking
to determine the optimal filter coefficients, a tradeoff between reduced
electronics noise and backgrounds.  Radioactive source calibration
results indicate that the noise can be reduced from 450keV to 220keV.
The goal is to implement the digital filtering by the end of the
upcoming down period if personnel can be found.  The committee agrees
that digital filtering should be commissioned as soon as possible. Since
the digital filter hardware is already in place, no hardware
improvements are thought to be needed for operation up to 1.5x1034 cm-2
sec-1, but we saw no predictions of the performance with the attendant
higher backgrounds, with or without digital filtering. It would be
useful to have simulations of ?0  mass resolution vs. luminosity using
available background extrapolations.

IFR
    The present situation of the IFR RPCs shows a serious degradation of
the system, which is losing efficiency at an average rate of about 1.5%
per month.   Although the cause of this deterioration is not firmly
established there is evidence that the problems are related to the
effects of temperature.  In particular, an analysis of the spatial
distribution of the low-efficiency chambers shows a correlation between
the operating temperature in the summer of 1999 in the immediate
neighborhood of the chambers and the degree of degradation.   This is
confirmed by dedicated tests, which show that similar failures can be
induced by operating chambers at elevated temperatures and currents.
Autopsies performed on the test-stand chambers revealed clear physical
damage to the electrodes in the form of linseed oil “droplets.’’

    In both cases the chambers were operated at voltages higher than
those needed to sustain single-streamer operation.  It is likely that,
coupled with the high temperatures,  this overvoltage also contributed
to the deterioration.

However, similar tests of newly-manufactured RPCs (built to simulate the
previous BaBar construction) have started in Italy.  The deterioration
in efficiency as seen in the tests at SLAC is not yet observed. This is
currently not understood.

    A small but dedicated team of physicists has been working very hard
to study and understand these problems and to develop a strategy to
mitigate them.  They have acquired a considerable body of data, which
has already provided some understanding of the situation.  However, the
situation is not under control and it is not clear that the existing
team is large enough to effectively address the problems on the
timescale required.   Moreover, it appears that the current problems are
the result of past oversights symptomatic of an understaffed effort.

  Barring a halt in the deterioration of the RPCs and/or the
identification of a way to mitigate the problems in situ  replacement of
the RPCs will be needed.   The group has already taken steps in this
direction by ordering a set of new RPCs of a similar design.  The new
RPCs, which incorporate design improvements developed for ALICE and
ATLAS, differ from the current BaBar RPCs in small, but,  important
ways.  Specifically, they are made with bakelite plates having smoother
surfaces, employ a much thinner linseed oil coating, and use
polycarbonate edge spacers  having the same long-conduction-path profile
as is used for the inner spacers. In parallel, the group is also
studying the possibility of replacing the barrel part of the system with
a new technology, notably scintillating strips with wavelength-shifting
fiber readout.   Any replacement operation(RPCs or other technologies)
will be a major undertaking, requiring of order six months detector
downtime.


Recommendations

Near Term

   Independent of the long term resolution of these difficulties, the
existing chambers will be in operation until the summer of 2002.  It is
therefore important that this system be maintained with care and that
all reasonable steps are taken to avoid further damage and to maximize
the longevity of the system.  In particular, close attention must be
paid to the operating environment of the RPCs, especially the
temperature.    The operating high voltage should be kept as low as
possible.  To this end, the electronics thresholds should be set to a
level low enough to support single-streamer operation, if at all
possible. Moreover, the operating voltage must be compensated for
temperature and atmospheric pressure variations.   Careful monitoring of
efficiencies and currents should continue.

   The group should move forward with their plan to replace 24 of the
endcap chambers with new RPCs.    Autopsy data on the removed chambers
will likely be valuable in developing an understanding of the problems.
The replacement chambers should be regarded as a pilot project for the
possible eventual replacement of the full system.  To that end,  it is
important to establish and follow the procedures that would be used in
such an undertaking at this stage.   Specifically, this involves careful
monitoring of the construction,  systematic preinstallation inspections
and tests, close control of the operating environment and careful
monitoring and recording of chamber performance at all stages
(efficiency, current, gas tightness, mechanical integrity, etc.).   For
the replacement chambers,  priority should be given to acquiring the
knowledge needed to understand how best to go forward, as opposed to
enhancing the performance of the RPCs.

The study of alternate technologies is also prudent to the extent that
it does not significantly distract from the RPC effort.

Long Term:

Assuming that it will prove necessary to replace the RPCs, the
collaboration faces a decision, which will involve weighing the
uncertainty associated with implementing another RPC system against the
expense and effort that would be required to implement a new technology.

In view of the problems currently being experienced, the risks
associated with the RPC replacement option appear to be substantial.
However, there are at least two large systems (L3 and Belle) in
successful operation and other systems are under construction.
Moreover, there has been a considerable world-wide RPC R&D effort.  In
view of this,  an informed decision should be possible.    Given the
need to come to a timely conclusion, it is important to gather as much
data as possible, both from within the BaBar system and from other RPC
efforts,  in an efficient and organized way.

 A properly operating single-gap system should provide adequate
efficiency to meet BaBar’s requirements.  However, if a replacement RPC
system is pursued, then one might explore the possibility of modifying
the design to incorporate a double gap.  Although this approach will not
compensate for a fundamentally flawed implementation, it does provide a
measure of redundancy and will modestly improve the module efficiency
even for properly functioning RPCs.   There was some discussion of
implementing a double-gap design, but a carefully thought out scheme
was not presented.  If the group decides to go in this direction, they
should give serious thought to implementing an existing concept rather
than developing a new scheme.   It is also not clear from the
information provided that the gaps in the IFR iron are large enough to
accommodate a viable design.

Management:

     As noted, the problems that have been experienced can be attributed
in part to inadequate staffing.  Although BaBar has taken steps to
remedy this situation, it appears that further improvements are in
order.  In particular, there is a critical shortage of senior people
onsite.   This situation would be risky even for a well functioning
subsystem,  but is particularly serious given the present
circumstances.  BaBar management is strongly encouraged to further
strengthen the IFR subgroup and its management.


TRIGGER AND DATA ACQUISITION

Expanded DAQ/L3 capacity is vital for the physics program of BaBar.
Scaling of present data flow and L3 trigger to increased luminosity
requires a dataflow rate capacity of 2.8, 3.0, 4.0 kHz and a L3 output
rate capacity of 130, 220, 310 Hz (present, ’02, ’03).  We note that the
dataflow numbers do not (and cannot at this time) include the possible
benefits of the accelerator vacuum work during the upcoming shutdown nor
do the L3 output rates include potential improvements to the elimination
of the QED leakage events. Offsetting this, only linear current
extrapolations have been used for estimating backgrounds. Bottlenecks
have been identified and one or more solutions are being considered. The
collaboration will have to choose among them based on available
personnel and funds and how these offset each other.

The existing L3 can log data at 500 Hz but a factor of two CPU increase
for FY02 will be required to process the hadronic and QED “leakage”
events. When to implement this improvement is complicated by the desires
to explore the use of new platforms for better price-performance and to
delay purchases to the latest possible date, and the possible
introduction of Gigabit Ethernet. Thirty-two CPUs now execute the L3
code, a limit hardcoded in the dataflow.  Manpower has to be identified
for this restriction to be eliminated.  The alternative approach is to
increase the power of each existing node. It was pointed out that
extending the present system from 16 to 32 nodes was not simple and that
new system issues could arise during another node count extension.
Further algorithm improvements are required to control and reduce the
fraction of background events(QED leakage) relative to physics and
calibration events.

When the increased data volume due to backgrounds is included, the
existing dataflow can only handle 2.8, 2.5, 2.0 kHz (present, ’02, ’03).
The limitations occur in several places. ROM processing power for the
DCH, DIRC, EMC, and EMT can be addressed with software tweaks in some
cases but probably require CPU upgrades in others. More ROMs each
handling fewer input links for the DCH and DIRC could be considered if
spares are available. Slot-0 CPU performance will be first addressed by
event batching to reduce data handling overheads. The links to the
switch-based event builder will require the addition of more links where
needed by either doubling up the number of links per SLOT-0 CPU,
splitting the crate back planes to accommodate more Slot-0 CPU’s, or
conversion to a higher bandwidth link such as Gigabit Ethernet.  The
latter solution seems preferable since it provides greater long term
headroom for increased data volumes, but there are consequences at all
other levels of the dataflow; the event building switch and the 32 L3
nodes would need to be replaced to handle the higher performance links.
We did not hear discussion of possible mixed Gigabit and 100 Mbit
evolutionary solutions.

The L1 trigger of course determines the rate requirements of the
dataflow and L3. The only improvement being considered is the
introduction of a z-vertex discriminator based on the stereo wires of
the DCH. Under present operating conditions, an effective z trigger
would reduce the total trigger rate by a factor of two and have
potentially greater impact in the future as the backgrounds rise. A
conceptual design for such a trigger will be generated over the next six
months. It is crucial that this design not only be simulated with
existing data but also under conditions of the increased backgrounds
predicted for the DCH up to 1.5x1034 cm-2sec-1. For the immediate
future, dataflow and L3 planning should continue with no assumptions
about the use of a z trigger.


COMPUTING IMPROVEMENTS

1. Analysis Model
Babar’s computing model working group has presented a report defining
the experiment’s requirements for computing and providing a broad-brush
description of a proposed model to meet these requirements.  Highlights
of this approach include:
? Development by the experiment of an agreed-to set of physics
priorities and a trigger strategy consistent with these priorities
? Division of the reconstructed data into on order 20 self contained
streams, each of which will contain a number of overlapping and/or
related skims
? Hierarchical organization of the data into 5 levels (tag, micro, mini,
reco, raw).  The introduction of a new mini of size ~20 kB/event is
intended to allow more detailed analysis without reverting to the much
larger reco
? Recognition of the need to reduce the fraction of raw, reco and mini
that is disk resident as the integrated luminosity increases.
? Hierarchical organization of the collaboration’s computing resources
into Tier A (SLAC, IN2P3), Tier B (RAL, INFN) and Tier C (Universities),
with performant import and export mechanisms for transferring data
between sites.

The committee commends the collaboration on the analysis model
requirements document and endorses the overall direction it sets.  It
should be noted, however, that the technical plan to implement this
model is still under development.

Recommendation :
The implications of this model are that BaBar requires:

? a robust mechanism for staging data from disk to tape
? a resource management system that encourages use patterns that access
data in an organized fashion (by stream and skim)
? tools that encourage the use of desktop and remote resources by making
export simple and straightforward
? coordination with IN2P3 to fully exploit  their strong commitment  to
their role as a Tier A center.  Because technical decisions about Babar
computing will affect both Tier A centers, both centers must have input
to these decisions. Tier B centers must also be involved in these
discussions.

2. Staging
The committee was asked to comment on whether the model of how data is
staged from tape to disk is realistic. In the materials distributed and
presented we do not feel we have found a clear model for how staging is
to be managed over the next few years. We feel that a strategy for data
access management at the physics analysis level still has to be
articulated. The term 'staging' perhaps implies too narrow a view of the
needed strategy; a data access management strategy will couple staging
mechanisms and policies tightly to automated mechanisms for the location
and the automated, optimized retrieval of data required by analysis
jobs.
With computing costs tightly constrained and disk costs a principal
driver of the overall costs, it is vital that there exist a strategy to
manage and optimize data access in as automatic a way as possible. This
strategy should be a central component of the overall plan for
optimizing ease and speed of physics analysis vs. hardware cost. An
effective data access strategy will contribute to a computing model that
is adaptable to accommodate uncertainties in luminosity and budget. As
is recognized, it should include mechanisms to enforce collaboration
defined policies in the prioritization of analyses and optimization of
throughput, driven by physics priorities.
Streams are viewed as the fundamental units of data transfer and
deletion, and the basis for managing resource constraints and
particularly disk space. This approach is sound, and data access
management should flexibly support the full spectrum of disk resource
assignments from a fully tape resident to a fully disk resident stream,
with the operation and behavior of the system as seen by the
analysis-user differing only in the job latencies across this spectrum.
In addition to the 'horizontal' management of access to the 20 or so
streams foreseen, data access management could also support the
'drilling down' to event components from earlier processing passes (e.g.
from micro to mini or rec) in an efficient way, in an environment in
which the components to be retrieved are often to be found on tape. In
the context of data distribution tools, BaBar is already addressing the
need to efficiently index and retrieve information on the file locality
of event components. When coupled to an access management system, this
information can form the basis for mapping a 'request set' of required
event components to the containing files, and retrieving the files
either in the context of the current analysis job or (more efficiently
and realistically) in a second processing pass that can allow the data
access manager to martial the full request set for the job and optimize
the retrieval (together with those of other concurrent jobs).
The Grand Challenge (GC) software developed at LBNL and elsewhere is an
example of a tool that may map well onto BaBar's needs for data access
management. It is in production use in one experiment, STAR, and while
it is not used in an Objectivity context there it was originally
developed for use with Objectivity. The GC software also supports fast
multidimensional indexing of physics tags and can serve up event
components matching a query on these tags. While it was designed to
support HSMs other than HPSS, it has to date only been used with HPSS
and its use at non-HPSS sites would require a porting effort.
Recommendation
BaBar should articulate a strategy, or a few strategy alternatives to
manage and optimize data access for the whole spectrum of physics
analysis activities with their differing priorities and requirements.
This should be presented to the collaboration for feedback and the final
strategy and implementation approach agreed upon.

3. Scalability of Production Facilities
We have been shown projections for how the Online Prompt Reconstruction
(OPR) farm is required to grow in order to cope with increases in
luminosity. These indicate that a doubling of capacity will be required
for 2001 with further comparable increases in subsequent years (3 times
by 2002, 4 times by 2003). These projections assume that performance
must scale with the peak luminosity in order to be sure that OPR can
keep up with data-taking.
Overheads need to be understood and potential bottlenecks identified and
dealt with to be confident that the operational performance scales
linearly with increasing capacity. Significant progress has already been
made in bringing the performance of the OPR facility to its present
level and a number of additional factors have already been identified
that can impact on the ability of the current OPR facility to scale
further. Scaling requires minimizing the startup/closedown transients,
minimizing outages, and maximizing the processing rate. Operational
aspects, in particular the rolling calibration scheme, determine the
dataflow and processing steps and therefore play a key role.
The main factors affecting the scalability of performance appear to be
the bandwidth to the filesystem and the lock server traffic.  A
significant effort is being made to understand the overheads at startup
and closedown. For example tests have shown that database overheads can
be reduced by the use of CORBA servers, which can be used to speed up
the reading of conditions information during the run initialization
phase (by ~30% on a 50 node system). Tests are underway to study the
impact of introducing a number of potential improvements e.g. reuse of
containers, reducing the number of databases, improving load balancing.
The possibility of such improvements gives some confidence that a modest
scaling of the current implementation ( e.g. to ~200 nodes) is possible.
However it is not clear to us at which point the system will saturate.
To scale the system by larger factors a number of different measures may
be needed e.g.
? To augment servers and filesystems.
? To replace existing cpus with more powerful machines in order to
minimize the number of nodes and therefore overheads
? To split the OPR farm into multiple farms
? To modify operational aspects of the rolling calibration

Upgrades of the OPR farm are urgent since it has to be in place by Feb
2001 to be ready for the planned doubling of luminosity in 2001. An
implementation plan is needed describing how this facility will grow for
2001 running and beyond. Software upgrades and changes should be tested
to check for reliability as well as performance. The possibility to test
an enlarged facility before it is employed would be invaluable and the
possibility of scheduling such a test using the combined OPR and
Re-processing farms during the next shutdown investigated.
The collaboration has decided to focus its future efforts on an
Objectivity-based analysis facility. Here there are also issues that
effect scalability of the analysis system to cope with expected data
loads. The limitation on the number of database ids (DBID) has serious
consequences for the analysis system. Extended DBID support is essential
for reducing the size of databases from 10GB to ~250 MB. This is needed
to optimize the efficiency of staging and data distribution. Although
this feature is included in the latest release of Objectivity (v6) tests
are needed to understand whether it can be used in production in time
for next year’s running.
Lock traffic is expected to be significantly reduced in Objectivity V6
which should allow for  optimization of container and database creation
and allow a large potential gain in physics analysis federations using
~read-only databases. This will help to improve scalability of the
analysis system.
Recommendations
Produce a detailed implementation plan describing how the production
farms should evolve, specifying how the number of cpu nodes, number of
fileservers, network bandwidth and storage capacity should grow to
handle the expected increase in event rate. The effectiveness of the
changes should be assessed in terms of the cost of providing the
required computing resources, the risks and the contingencies.
Review the operational strategy of the rolling calibration scheme to see
whether the overall efficiency of the OPR farm operation can be
significantly improved.
An attempt should be made to understand and minimize the overheads due
to the packaging of data at all levels.

4. Data Distribution

We commend the significant progress made on data distribution issues.
TAG and micro data are being exported to IN2P3 in  Objectivity format
and many sites are receiving data in  KANGA format. In addition Monte
Carlo data are being produced at several remote sites and are imported
to SLAC over the network.
Managing the production and distribution of data is manpower intensive.
Tools for automating this are recognized as being essential and are
under development. An efficient metadata catalogue is one such tool that
is an important ingredient for data distribution.
Experience has shown that significant effort is needed for managing the
installation and operation of databases. The ability to use databases
remotely without local expert support is essential if the collaboration
is to eventually stop its support for Kanga.

5. Manpower
As the size of the datasets and sophistication of BaBar analyses
increase there will be a need for continuing improvements to the
software. These efforts must be widely recognized by the collaboration
as an important contribution to the experiment as a whole. We endorse
the use of MOAs to help ensure the adequate resourcing of manpower for
all computing activities.
Moreover experience has shown that managing the development and
operation of the Objectivity databases is a complex and difficult task
requiring people with great experience and technical skill. Since the
data processing systems will need to evolve to meet ever increasing
demands placed upon them, the need to monitor the performance of the
database and implement changes will continue indefinitely.

Recommendation
A careful watch needs to be maintained to ensure that adequate staffing
is maintained and consolidated when appropriate and that key positions
are filled. MOAs should be used to aid in this process.

6. Cost for Offline Computing

BaBar is the first big HEP experiment using a single ODBMS for managing
all kinds of data as a coherent and scalable solution. The technology is
only maturing now and the first wide deployment in an HEP experiment was
not without problems and performance limitations. The experiment is
learning the new methodology for analysis and has mastered many of these
limitations successfully. It has put in place software and hardware
resources which allow reasonably fast and successful data analysis.
During the initial problem solving the focus had to be on fixing
software problems. Limitations of the underlying hardware resources were
avoided by putting in place larger storage and CPU resources than those
calculated from first principles. This strategy was successful and was
probably the right balance given that many of the startup problems
showed up only when data taking had started and the manpower and
computing professional resources were obviously limited.
BaBar is facing a promising increase in luminosity with a corresponding
increase in data volume now. Since many of the startup problems related
to the ODBMS approach for data storage and the particular choice of
Objectivity have been overcome the experiment must return to a
bottoms-up calculation of resource needs for data storage, processing
and analysis. The requirements presented to the committee were based on
scaling the current resource usage. They do not question the assumptions
that went into making resource estimates. Areas where there may be
potential savings include :
? The large storage space allocated for miscellaneous data storage and
ntuples on expensive RAID disks
? Maintenance of two complete versions of the entire micro dataset on
disk
? Large buffer space allocated for the OPR and reprocessing farms
A more detailed study is needed to understand and optimize the use of
resources and to match budgetary constraints.

The costing of the resources for the next three fiscal years is based on
previous experience in BaBar and at SCS. The presentations were not
detailed enough to judge whether they are complete or whether the right
implementation choices will be made. For this a more detailed
implementation plan is required which spells out the resources and the
required infrastructure items.

Recommendation
As part of the implementation plan BaBar should develop a bottoms-up
cost estimate that sets priorities and attempts to optimize the
combination of cost and performance. This should include fall-back
solutions to deal with shortfalls in funding.