Version of: Sun Jun 22 23:26:11 PDT 2003 + -> Should be covered in some workshop session - -> Can be covered after workshop or outside sessions ? -> Not sure New Micro --------- o Required for July test: + we need to make a standard labeled figure to describe "reduced micro", mini, etc. to people (fix the terminology) + finalize ROOT mini implementation [Mon2a] + validation of root mini [Mon2a, Tue1A] + reduced mini root implementation [Tue1A] + composite candidates mini persistence [Mon3a] + user data mini persistence [Mon3a] + writeMini script extensions to include user data and root persistence + skim content policy ('micro' + requested cands or just requested cands) [***] + "Configuring mini based skims"?? [***] o Longer term issues: + Load-on-demand design [Mon4a] + post-july event metadata [Mon4a] + composite object reference persistence optimization + tru persistence optimization [Tue1a] - detailed physics comparison of mini vs micro - optimization of root mini (size + speed) Eventstore ---------- o Required for July test (or soon): + Site-specific data access configuration files for Framework jobs (file vs xrootd access by portion of eventstore, backup primary xrootd servers or DNS mechanism, what is configurable in sitewide versus user configurations) [Mon3b] + What should the user-visible tcl interface look like (including borrowing and clustering components) + Integration or evolution of the "writeMini" script to make it work with other applications + How does the registry handle multiple output streams, while borrowing components from an input collection? [Mon2a] - Graceful handling of 2GB filesize limit - relocatable LFN in headers - merge application o Longer term issues: - StdHep needed? What is migration plan? (Are we writing the tru now in the Kanga/ROOT eventstore) + Support for collection "run database" [Tue3a] - Support for extended collections /users/elmer/mergedevents@10-20 + Support for multiple "usr" components [Mon3a] - package cleanup: when do we delete the *R classes, when do we move things out of RooUtils? Do we want to change core names to decouple from 12.x.x Kanga production? - need to gracefully handle opening a file while already exists Non-Framework access -------------------- + What changes are needed to make this work with the "reduced" mini? [Tue2a] + What infrastructure do we need to provide to make this work with the new eventstore (with event headers, taking collections rather than files from uers, integration with site-specific data access methods, xrootd, etc.) [Tue2a] + Should users be able to write output from this which is then readable as standard eventstore code? If so, how do we insure that the event header is setup correctly? [Tue2a] Skimming -------- + Adding modules which produce user data + Tcl configuration for user data, reduced output + Switching from "micro" to "cache" mode [Tue3a] + Issues related to all mini read modes (cache, refit, "repair"?, raw) [Tue3a] + Configuring SkimMiniApp to use proper event output modules and streams (as opposed to the "honest_this_is_a_normal_module" configuration), this is connected to the registry and tcl configuration for borrowing components issues in the eventstore section (not needed for July test?) [Tue3a] + Is it possible for AWG's to combine their skims into a "stream"? Do they need pointer collections for this or is it sufficient to run on the tag of the combined skims? [Tue3a] + How do we manage the size of custom output data from the AWG's? [Tue3a] + SkimTools extensions for multiple files (until new bookkeeping arrives), needed in particular for PR/SP - Adding hooks for running reco code in the SkimApp - PC pass reading from mini Data distribution/management ---------------------------- o Required soon: + Which Bdb/Objy data management use cases are relevant for Kanga/ROOT eventstore? What new use cases arise? What are the resulting requirements on xrootd/dlb? [Mon3b] + Presentation from FabrizioF on Xrootd/XTNetFile, does it fit with data management use cases? [Mon3b] + How is the XTNetFile configured? (What configuration parameters are needed from the point of view of data management?) [Mon3b] ? No "existence check" command in xrootd: how to prevent long timeouts? [Mon3b] + Initial tests of xrootd, testing plans (who is going to do testing and where?, which datasets?) [Mon3b] + Tier A <-> Tier A data distribution, how will this work? What pieces still need to be developed? (Who/when?) [Mon2b] + Tier A <-> Tier C data distribution, how will this work? What pieces still need to be developed? (Who/when?) [Mon2b] + Presentation from Jean-Yves and Liliana on SRB, what is role of SRB in new system? [Mon2b] + What are our plans for BdbServer++ in the new model? [Mon2b] + What are the requirements for collection/file integrity-checking tools? + When/where to do checksum? Which one to use? Where is the checksum useful? + Bookkeeping for UUID's: we start producing data files in Sep with UUID's, how do we track this with skimData? (New system should also do it.) o Longer term issues: + What collection names do we want for PR, SP, skimevents? + What is the standard naming scheme for merged output? + How does the xrootd get information about which files are where on mass storage? [Mon3b] - Distribution of file sizes expected from analysis? - Current system reads/writes files via NFS. Can we run bbftp/bbcp on the disk server? - How do we handle exports from/to MSS? Do we go via the XRootd? - If we use tar archives for MSS, should we transfer files as tar? PR -- + How will data distribution from Padova to Tier A work? [Mon4b] + How does the Padova-local file catalog work with the new eventstore? [Mon4b] + What do we need to do to export continuously and deliver data from PR with minimum latency? What is the minimum latency? [Mon4b] + How do we recover from and recognize failures in PR? [Tue1b] + What "QA" should be done? (read checks as well as real "QA") [Tue1b] + What needs to be done (by when and by whom) to start PR on Sep. 8 with Kanga/ROOT output ane export that data to SLAC? (Core development, local farm changes, data distribution, bookkeeping, etc.) [Tue1b] - Padova-local skimming of PR output? (How would this be exported?) + Do we need the 4 streams from PR? Or are AllEvents, BackgroundStream, TriggerStream enough? (i.e. can we turn off BrecoStream given the subsequent deep-copy skim.) Should be coherent with the Fall conversion [Tue3a] + Should we add any additional data (BReco composites, etc.) in PR/SP AllEvents micro? [Tue3a] + Validation plan [Tue1b] SP -- - Integration of the merge step + Should we do the merge at the production site or Tier A? Baseline is doing it at production site using 'sprite'. What are the issues with having people using 'sprite' for this? How should this work in the SPGrid system? [Mon4b] + How does the data import at the Tier A work with the new eventstore? [Mon4b] + What is the naming scheme for merged runs? [Tue1b] + How will data distribution from production site to Tier A be done? [Mon4b] + What will a typical SP farm (Data servers, NFS, I/O, etc.) to handle the new eventstore (and merge)? [Tue1b] - Conversion and management of the background triggers + SPGrid issues, making test rpms for the latest releases, integration of rpms/dump mechanism into release [Tue1b] + How do SPGrid plans fit with plans for deployment of new eventstore in SP? How will merge be done in SPGrid? [Tue1b] + What needs to be done (by when and by whom) to start SP6 with Kanga/ROOT output at the beginning of Oct2003? [Tue1b] + Validation plan (including input raw data in Kanga/ROOT eventstore) [Tue1b] Fall full data conversion ------------------------- + Can SkimTools be modified to run the converter application? Do we have the bookkeeping for this? [Tue3a] + Should we convert AllEvents or AllEvents+TriggerStream+BRecoStream? + What do we do about the pointer collections? [Tue3a] + Validation plan [Tue3a] Infrastructure -------------- - root target versus rootlib target in SRT, do we need both? - reference schema (how do we let users know that they have modified existing persistent classes?) - Framework additions for input/output summary - Writing different components from different streams (e.g. raw for Background stream in Elf) Bookkeeping ----------- + Can extensions to the existing bookkeeping support what we want to do until the new bookkeeping is available? How/when do we transition?