Requirements from the Computing Model ===================================== 10.1.1 Improvements in tools for tracking data sets should be pursued. These should provide a very simple interface to their ordinary users, allowing the use of short, mnemonic names for concepts like "the full set of good runs of off-resonance data approved for use in the current conference cycle, processed with the latest official software releases". These tools should be well-integrated, using common user interfaces and naming conventions, with the others envisioned in this section. 10.1.2 Tools should be developed for automating the application of defined analysis and production (e.g., skims) tasks to data sets. These tools should, in particular, relieve users of the presently common task developing their own personal tools for managing large numbers of TCL files, submitting large numbers of corresponding jobs, tracking their progress, re-running failed jobs, and collecting and cataloging their outputs. It should allow users, as much as possible, to treat these tasks and their outputs as units, and free them from the need to explicitly juggle hundreds or thousands of individual "things" (jobs, files, etc.) Computing requirements ====================== 1) tcl files will be site independent 2) data sets information will be the same at different sites Extraction from the user cases reported below ============================================= Data sets selection and management ================================== The user will have to deal with collections rather than files. The selection tools should be the same for MC and data. Output from the selection of MC and data should be comparable. Selection must allow exclusion. Selection must be done by data sets according to collection attributes: 1. identifier: collection name 2. identifier: dataset name 3. identifier: decay.dec 4. identifier: condalias 5. identifier: detector conditions 6. user set of runs 7. set of runs declared good/bad/flawed 8. production release 9. skim release 10. processing version of release 11. more generic like "Run1 january" (a bit like cond alias) 12. ................ Mode names are strongly rejected as identifier for MC selection Stability of RQM data sets ========================== A set of runs declared good/bad/flawed can change the information The system has to maintain memory of the old sets. Differences among different data sets must be computable. It must be possible to select different data sets without overlapping (i.e. if a bug is discovered there must be a distinction between before and after. Are we sure of this as a requirement? shouldn't this go in the documentation) Bookkeping output ================= 1. tcl files containing collection lists + other informations 2. ascii files with information related to the data set selected 1. type of information can be: luminosity bcount mupaircount estimated luminosity (for montecarlo) offpeak/onpeak run block number of events in input number of events in output production release skim release ? Bookkeeping output organisation =============================== It should be possible to organise the output by. 1. a specified number of events 2. a specified selection of detector conditions 3. a specified selection of condition alias for MC Output should be subdivided in a way that allows the user to match their own set of ntuples with the set of processed tcl files. COMMENT: this is part of the task management each tcl file is an input for a job and a production system keeps track of the the inputs associated with a produced collection whatever is the input. Task Management =============== Take SkimTools as a starting point. ============================================================================== USER CASE (1) This message is a summary of our experience for production of reduced collections for BReco events, currently used by more than 10 analyses, as well as prodution of ntuples for private and general use. Therefore I think it is important to identify two areas where bookkeeping tools are needed and it's not clear to me whether they can be unified or not: 1) accessing the existing data and MC events in a relatively easy and transparent way (to be defined below) 2) Bookkeeping of O(10^4) analysis jobs to produce ntuples/collections data access =========== One of the innovations introduced by us last year in the BReco/sin2beta analysis and now used officially by the RQM was the GoodRuns package. This allowed to keep tracks of runs being added, but specially those being _removed_, on a weekly basis. a simple diff wrt previous week's tag would allow to process a handful of runs and mark the already processed runs as bad. We used SkimTools for the bookkeeping of the production, very similar to the way it is used for skim and kanga production. we decided to process each run separately because: 1) it's easier to combine small blocks of data and isolating it a larger dat set 2) easier to remove runs decalred bad after our processing 3) better success rate: Since db problems such as lock collisions, lockserver crashes, ams crashes, sever saturation etc. are a real part of the production it is better to have a single run crash rather than a job with 50 runs crash after 43 runs. 4) the speed of the analysis job (2-4 Hz) does not allow very long jobs anyway In addition, in order top deal with the multiple federations we had to implement a new package tool (AnalProdTools) on top of SkimTools. Since this is hopefully not a problem anymore I'll not go into the details of this part. One of the main problems in this skim was keeping track of whether a reduced collection is a good run or not at some point in time. This is one place where a unified tool for users both to access the dataset and to process it can be very useful. An extended/new version of SkimTools could keep track of the runs declared good by the RQM. Rather than relying on the particular version of GoodRuns used, one can query this db at anytime and be sure to have the most up to date and CORRECT set of good runs. Again, this was done by maintaing a flat file of good runs inthe GoodRuns package, and therefore exposed to otential human errors at differnt stages. improvemets for the future =========================== I think the natural next step is to be able to access the official datsets from within framework with something like talkto GoodRunsInput { dataSet set 2002_good_b1-s0-phys12 } I think the implementation of GoodRunsInput can then insure that one looks at the GoodRuns in the user test release and then the official location. Even better, it can use the db of goodruns. It can also decide whether to read in objy/kanga/future_data_format based on the job configuration. However given the current and probably future processing rate for analysis jobs I doubt one can process an entire set. Therefore one should also implement the "first" parameter and allow the user to run many jobs on different chunks of the same data set. MC access ========== The problem with the access to the SP MC was brought up by Gianluca, Owen, and myself at various HN and meetings, including the PAC meetings, and IMHO it was simply ignored. The main problem was: The SPx inventory pages are basicaly useless. This is why there are many home grown recipes/scripts that parse the output of bfreport. bfreport is by far the best and only reliable source of information. Using its output one can split the data in config periods, releases, etc. Adding just the SkimTools-like capabilities to bfreport would be an enourmous success. SkimTools has been useful to access MC once one had the magic string for each mode from the bfreport output. However the parsing of the string needs some work to make it more robust against additional/missing empty spaces and the possibility of searching on partial strings. Rather than using the magic strings that are basically decided by the fantasy/creativity of whoever first created a mode, it is better to query SkimTools and bfreport by decay files in ProdDecayFiles which are unique and easily looked at by the users. Bookkeeping for analysis jobs ============================= Given the large data and MC sample available today, most of the analyses require running a few thousands jobs to make ntuples/kanga files/rootfiles or collections. Even in absence of the known db problems, it is reasonable to assume that there can be a failure rate of few % due to users mistakes, power outages, disk failures etc. Finding out which jobs crashed and needs to be resubmitted is best done by using a bookkeping tool. This improves the usage of the resources and reduces the users' fraustration. Recently I used SkimTools for production of a few thousand ntuples. Since SkimTools was designed mostly with collections in mind as input, a few considerations were necessary when processing tcl files with a few runs each. After these modifications running jobs and fnding out the list of sucessful ntuples was extremely easy and saved a lot of time. Currently running on all the available generic MC requires more than 5000 jobs (assuming a rate of 5 Hz or so). Unless you implement a bookkeeping tool it can be a real nightmare to figure out what failed and what to resubmit. The bookkeeping tool can benefit from communicating with the central DB for data and MC runs. However one should be able to use such a tool in a standalone mode, by defining the list of input collections/files, for example user made tcl files. This would allow the system to be used also on smaller sites which simply copy over the data and do not necessarily want to setup a central book keeping tools and DB's. MySql has proved to be very easy to setup and use together with SkimTools for analysis purposes. Overall, it would be good to have ONE tool to access list of goos runs, data sets, luminosity information etc. Right now this is done by writing sevral scripts that combine the information from bfreport, skimTools, GoodRuns, and lumi scripts. =========================================================================== USER CASE (2) I wanted to split my tcl files (eg, for TauTau SP4 MC) according to month/year mapping. I found this information at http://www.slac.stanford.edu/babar-internal/spreq/showruns.html?cycle= SP4&modenum=1071[1] which gives the set of run numbers for a given INREL version for a given CONDALIAS. However, searching for options in skimData, I did not find such capability. Would it be considered worthwhile to add this CONDALIAS in the optionlists for skimData, so that for a given monte-carlo mode, one can get the set of run numbers or tcl files for a given CONDALIAS. I wrote to skimData authors 2 months ago, but ... I ended up writting a personal shellscript-and-fortran combination to split the list of run-numbers per CONDALIAS, but everytime the page changes, it is quite tedious to recreate the split list. However, it would be much useful to add this capability in skimdata command itself. =========================================================================== USER CASE (3) First, bookkeeping, from my perspective, starts with logbooks, Hypernews, etc. I am consistently frustrated by the lack of an accurate search tool for both Hypernews and the e-logbook. I try to use the available HN Inktomi search often, but usually end up just giving up on it and searching through links by hand. OTOH, when I search the web with Google, I'm consistently amazed with the relevance of what I find. If anything, this in my mind is a proof of principle that better search technology exists. As far as physics analysis bookkeeping (data and MC), I agree with Owen. In particular, there should be one general-purpose script for both data and MC. In my ideal world, it would accept either a list of runs, or a specifier (e.g. decay.dec or skim name) + maybe a processing version, and output a list or file(s) of collections or tcl files to run on, with an option to specify the maximum number of events per tcl file + other options skimData currently provides (e.g. list of bad runs etc.) This ideal script would also provide information on both number of events and luminosity (or an estimate of luminosity equivalent, in the case of MC, from the decay.dec)... Finally, I thoght maybe I was behind the times when I saw Owen's reference to bfreport. This tool isn't listed on the analysis scripts page (linked off of the Physics page) at: http://www.slac.stanford.edu/BFROOT/www/Physics/Tools/Scripts/intro.html A friendly BaBar web search for 'bfreport' gave me as its first hit this page: http://www.slac.stanford.edu/BFROOT/www/Computing/Offline/Simulation/ web/production.html Aside from the tiny fonts, this page claims the tool 'bfreport' was valid for MDC-I, MDC-II, PRV0 and SP1, and maintained by Paul Raines. ============================================================================== USER CASE (4) I have a few observations on the use of skimData: o some of the SP modenames don't work when you cut an past from the inventory page [most are fine] - so an option to feed the modenumber in its stead would be great [does this already exist? I don't think so] o folk law [that I have come across recently] disputes the claim in the workbook that skimData does everything - for SP some people are still using getdata [doesn't have the modename problem as it uses the number]. I guess this is now psycological. o I have two scripts - one to split up the good runs lists from the GoodRuns package into more manageable sizes: ~bevan/scripts/production/makeDataTcls and one to facilitate making SP tcl files: ~bevan/scripts/production/makeTclFiles2 The reason being that I wanted to make a structured tree of tcl files for the onres2001, 2002, 2000, 1999 and off res data in addition to many different possible backgrounds so that we can run over these data - but without the need to do anything more tedious than write a config file .... I found them useful for production purposes recently .... Carlos (cced) is preparing a script to fit the job description of matching up data with MC of a given condition set - i.e. one should be able to specify runs of either D/MC and get a return of the appropriate MC/D to use back. If you like I can make a more formal posting of these comments/observations if you feel it appropriate to do so. =============================================================================== USER CASE (5) I wound up writing a whole mess of scripts to manage the bookeeping. I needed them for the following purposes: a) to manage the different types of Monte Carlo and data I was processing b) to integrate certain information into my output ntuples c) to help match data and Monte Carlo sets d) to manage job submission and ntuple validation. The first bookkeeping script I use is a simple shell script which creates a directory tree to hold the tcls for different Monte Carlo and data sources (e.g. generic B+B-, generic B0B0bar, inclusive hard lepton samples, B->Xulnu, signal Monte Carlo of various types, etc, on resonance and off resonance data). It uses skimData to seed each directory with two sets of text files, one for objectivity, one for kanga, which contain lists of selected runs with some additional iformation. I did this just because I needed a systematic way to keep track of the large number of different Monte Carlo sources, + data, that I had to process. I use a matching directory tree in a different location for the ntuples I produce with these tcls. I also wrote another script which takes these text files and produces tcls: it wraps skimData and the lumi script, but integrates the output information. The final result is the usual set of tcl files (a roughly fixed # events per tcl, one tcl file per job, one job = one output ROOT file), but these tcl files contain the following additional information (those quantities that are only relevant for data are 0 for Monte Carlo): module talk EcsJobInfo skimName set (e.g. AllEventsKanga, Stream7) skimEvents set <# skimmed events in tcl> inputEvents set <# events in those collections before skimming> lumiCount set BCount set muPairCount set exit There is a very lightweight Beta module, EcsJobInfo_Module, that I've added to my sequence; all it does is read in the tcl parameters specified here, and write out a single-entry ntuple/tree containing branches for each of these entries. It also records the number of events processed by the module. That means that for any job I run, the output file contains all of this information in addition to the analysis ntuples. The reasons I wrote these scripts were as follows: with both event-based and luminosity-based normalizations, I frequently had difficulties because of failed jobs of one kind or another. For instance, using the lumi script on my tcl files afterwards meant that I had to resubmit crashed jobs until they all (or almost all) succeeded, or I couldn't get a meaningful estimate of the luminosity I had stored in my ntuples. Likewise, I sometimes had the problem that the number of events processed in a job didn't always match the number of events that had been in the input tcl (either because I'd accidentally left an environment variable set, or because a kanga collection wasn't there, in which case the job continued without crashing. I'd wind up with perfectly good looking ROOT files that were missing events). I wanted a way to integrate all this information into the output ntuples themselves, so that I could immediately check output ntuples for self-consistency and catch these subtle job failures, and so that I could work with a subset of my data while I resubmitted crashed jobs without driving myself crazy over what the total luminosity was supposed to be. My final set of scripts is a set of perl scripts that "chunk" Monte Carlo and data according to three blocks---Run1, Run2 2001, and Run 2 2002. They also split the SP4 Monte Carlo according to whether it was produced before or after the Ks->pi0pi0 bug was fixed. I would have liked to be able to also subdivide things more finely---according to month and conditions alias---but I didn't really need it and it was kind of a headache. As it is, it is a nuisance to keep following the posted run ranges and keep updating the scripts to match. I then use these "chunks" as a basis to scan the output ntuples and validate them. This last script writes out two text files, one of which contains valid ntuples that I use as the basis for further analysis, the other of which contains a list of failed jobs that need to be resubmitted. This goes back as input to the set of scripts I wrote to loop over my directory tree of Monte Carlo and data sources and submit jobs as needed. All of my scripts are really a crude approximation of the functionality I'd really like to have. Ideally, it would be wonderful if we could have the following standard bookkeeping tools. First, something with the current functionality of skimData, which could produce lists of runs or .tcl files, but which could handle information about collections available at any site, and which would also let me integrate luminosity, event count information, etc. into my .tcl files so that I could keep adding that information to my output files. It would also be nice if dividing up the data and Monte Carlo in certain ways was a little bit easier---i.e. it would be really nice to specify "Run 1, January" as a command-line argument for either data or Monte Carlo and have the run numbers be limited appropriately. It would also be nice to have a more standard set of management scripts both for submitting batch jobs, and for managing the output of those batch jobs. It would be especially nice if they came along with some watchdog utilities: utilities which could notify the user of the status of pending, running, and completed and exited jobs, and perhaps one that monitors the disks to which the user has directed output. If a utility ran as a standard part of batch submission that a) tells the user how much space they are using on that disk or disks b) as jobs are running, tells the user how fast they are eating disk space and c) warns the user if any of the disks they are writing to are close to being full, perhaps some of the disk space collision problems we've all run into could be avoided. (Okay, so people would still have to be good citizens. But if we gave the monitoring problem an easy and standard solution, it might help). ============================================================================= USER CASE (6) - It should be possible to use the book-keeping tool to manage any production in any site. - It should be possible to keep track of any output collection produced in any sites. - It should be possible to find any collection produced in any site from anywhere. "Any", includes the user's private collections. ============================================================================= USER CASE (7) I am running a skim production at IN2P3 but wish the bookkeeping to be stored in the main Oracle DB at SLAC. The transatlantic DB queries are handled by a DBi proxy which was added to SkimTools by Alessandra and Douglas last year. The main changes that I have made have been to take into account the fact that IN2P3 uses BQS batch system rather than LSF which is employed used at SLAC. The crucial differences are: o Every job submitted to batch must have a different name with *at most* 15 characters (specified by a -N option). - I have chosen to give each job the name 'run$Run' where $Run is the collection number; e.g 'run00025646'. o Arguments for the executable jobWrapper cannot be passed at the submission command line like with LSF. - I solve this by passing the arguments jobWrapper requires (like the request ID and the name of the SQL table) via environment variables. - Care must be taken to ensure the job 'sees' the environment variables you wish. The following sequence seems to happen: (1) The environment variables from the current session are transfered to the batch machine. (2) The user's .cshrc is called which in turn calls the global group_cshrc. (3) The jobWrapper script is started. Therefore, environment variables which are pre-set in the group_cshrc file (e.g. SQL_HOST) will be re-set to the default values before the job starts. The solution is simply to set them [back] to the desired value at the *end* of the user's .cshrc. o The jobs require to be submitted with the option afs_cell=slac.stanford.edu. Therefore a valid afs token for slac.stanford.edu is required. Furthermore BQS checks that the amount of time remaining on your token is *greater* that the maximum time of your job. If it is not, the job will be put on HOLD. - This eventuality is resolved by: (1) Obtaining a new token for slac (2) Use 'qlog' to pass the new token to the batch jobs (3) Use 'qrls ...' to release the held jobs. - This restriction limits the length of jobs that could be run at IN2P3. The time limit of long queue is longer than a afs token's duration! o The PATH varialble may still be incorrect. I have had reocurring problems in getting jobWrapper to find oocleanup. - So I add 'oocleanup -local' to my job script (prodRun in the example below) o One other minor point is that the skimmed collections will be in Objectivity at IN2P3 until they are manually copied by privileged person. Therefore there will temporarily be a difference between the bookkeeping record at SLAC and the contents of the DB. A tip: the skim requests can be done whilst logged on at SLAC rather than at IN2P3. This only thing to remember is to specify the your login name at IN2P3 as the name of the requestor. This has sconsiderable time advantage rather than doing thousands of transatlantic queries. e.g. skimReq --requestor john --import mySkimJobName myFileWithListOfCollections.lst ----------------------------------------------------------- An example skimJob definition for IN2P3 is: (skimming SP4 for the Breco AWG) skimJob --maintainer john \ --inpath @/groups/SP/ \ --inname /allevents \ --outpath /groups/Breco/ \ --logpath /sps/babar/j/john/Breco \ --wrap "RELEASE/bin/Linux24/jobWrapper" \ --submit "qsub -eo -V -l t=30000,M=512MB,platform=LINUX,scr=500MB,u_bb_simuboot,afs_cell=slac.stanford.edu -q G -o -N " \ --create mySkimJobName mySkimReleaseName /afs/in2p3.fr/home/j/john/rel12.3.4/workdir/prodRun B0ToDLight,BchToDLight where my executable script 'prodRun' is #!/bin/csh echo "== prodRun (lyon) ====================" cd /afs/in2p3.fr/home/j/john/rel12.3.4/workdir srtpath 12.3.4 Linux24 simuboot bin/Linux24/mySkimApp my.tcl oocleanup -local >> /dev/null exit ==============================================================================