Manual Leak Checking

• Measure the baseline leak rate. Run the job for, say, 100 events and record the memory used. If you use the LSF batch system, it is the "Max Swap" number in the job report, 319 MB in this sample output.

  Sender: LSF System "<lsf@palomino12>"
  Subject: Job 116699: "<061740>" Done
  
  Job "<061740>" was submitted from host "<percheron>" by user "<young>".
  Job was executed on host(s) "<palomino12>", in queue "<extralong>", as user "<young>".
  "</u/ea/young>" was used as the home directory.
  "</u/ea/young/BaBar/QA7103/workSimApp>" was used as the working directory.
  Started at Wed Feb 10 18:58:41 1999
  Results reported at Wed Feb 10 21:50:01 1999
  
  Your job looked like:
  
  ------------------------------------------------------------
  # LSBATCH: User input
  RELEASE/bin/SunOS5/SimAppApp ./SimAppProduction.tcl
  ------------------------------------------------------------
  
  Successfully completed.
  
  Resource usage summary:
  
      CPU time   :   8349.35 sec.
      Max Memory :       153 MB
      Max Swap   :       319 MB
  
  

  Using the same set of input, repeat for progressively more events. Determine the slope of memory used vs number of events. This is your baseline leak rate. The number of events needed depends on the size of leak you are after. You should avoid using very few events. Some memory allocation associated with initialization is done when a piece of code is first called.
• Find the ordered list of sequences and modules used by your job. When the job starts up, issue the tcl command path list. If more than one path is displayed, look for the one you are using. It will be something like this.

    Everything                   (enabled) 
          CommonInitAppSequence        (enabled) 
              EvtCounter                   (enabled)      print event number
              GenBuildEnv                  (enabled)      Build General Environment
              PdtInit                      (enabled)      initialize Particle Data Table
          BdbSequence                  (enabled) 
              BdbSetTime                   (enabled)      Set event time
              BdbCreateCM                  (enabled)      Data Model Creates BdbConverterManager
              PepBdbLoad                   (enabled)      Pep BaBar Database Load
              DchBdbLoad                   (enabled)      Dch BaBar Database Load
              DrcBdbLoad                   (enabled)      Drc BaBar Database Load
              EmcBdbLoad                   (enabled)      Emc BaBar Database Load
              G3BdbLoad                    (enabled)      G3 BaBar Database Load
              IfrBdbLoad                   (enabled)      Ifr BaBar Database Load
              L1DctBdbLoad                 (enabled)      L1Dct BaBar Database Load
              L1EmtBdbLoad                 (enabled)      L1Emt BaBar Database Load
              L1FctBdbLoad                 (enabled)      L1Fct BaBar Database Load
              L1GltBdbLoad                 (enabled)      L1Glt BaBar Database Load
              L3TBdbLoad                   (enabled)      L3T BaBar Database Load
              StdHepBdbLoad                (enabled)      StdHep BaBar Database Load
              SvtBdbLoad                   (enabled)      Svt BaBar Database Load
              TrkBdbLoad                   (enabled)      Trk BaBar Database Load
              TagBdbLoad                   (enabled)      Tag Loader Module
              BtaBdbLoad                   (enabled)      Beta BaBar Database Load
              BdbEventUpdate               (enabled)      Data Model Update Module
          DchInitSequence              (enabled) 
              DchBuildEnv                  (enabled)      Build Dch environment
          IfrInitSequence              (enabled) 
              BBGeomRead                   (enabled)      read runtime.db geometry
              IfrBuildEnv                  (enabled)      IFR - build IfrEnv
              IfrGeomInit                  (enabled)      IFR - detector model init
              IfrRndmInit                  (enabled)      IFR - Geant4 HepRand init at each event
              IfrRestoreMCMap              (enabled)      IFR - restore MC info (if reading from Bdb)
              IfrVstModule                 (enabled)      IFR - visitor manager
              IfrPidPersister              (enabled)      IFR - create Pid RW Persisters
              IfrMuCalib                   (enabled)      IFR - mu/pi calibration procedure
          SimAppProductionSequence     (enabled) 
              StdHepLoad                   (enabled)      Load StdHep data
              StdHepPrint                  (enabled)      Print StdHep data
              FcsInputSequence             (enabled) 
                  RandomCtl                    (enabled)  Random number controller
                  FcsClockCtl                  (enabled)  Fcs Clock controller
              GfiSetCollision              (enabled)      Load PepCollision
              MixFrameInputSequence        (enabled) 
                  MixCTL                       (enabled)  Hit Mix Controller
                  Mixer1                       (disabled) Background 1
                  MixEnd                       (enabled)  End of Mix Controller
              MixedEvtCounter              (disabled)     Mixed Evt Counter/Skipper
              MakeXReference               (enabled)      Cross reference pointers GTracks
              SimAppPreTrigSequence        (enabled) 
                  SimAppPreTrigHist            (enabled)  SimApp: Hists before trig phase
                  SimAppPreTrigStat            (enabled)  SimApp: Stats before trig phase
              EmcWaveformSequence          (enabled) 
                  EmcBuildEnv                  (enabled)  Build Emc Environment
                  EmcGHitsToWaveform           (enabled)  Turns GHits into waveforms
              L1SimL1TrigSequence          (enabled) 
                  L1EmtSimModule               (enabled)  Emc level 1 trigger simulation
                  L1EmtSimDigiToTrgEM          (enabled)  Cludge interface to trgEM
                  trgDC                        (enabled)  DCH Trigger
              L1FctFastCtlSequence         (enabled) 
                  L1FctMakeDigis               (enabled)  FCT Digi maker
                  L1FAcceptFilter              (enabled)  Filter on L1Accept
              L1SimDigiSequence            (enabled) 
                  L1EmtMakeDigis               (enabled)  Truncates L1EmtSimDigis to Digis
                  L1DTsfMakeDigis              (enabled)  TSF Digi Maker
                  L1DPtdMakeDigis              (enabled)  PTD Digi Maker
                  L1DBltMakeDigis              (enabled)  BLT Digi Maker
                  L1GltMakeDigis               (enabled)  GLT Digi Maker
              L1FAcceptEvtCounter          (disabled)     L1FAccept Evt Counter/Skipper
              SimAppPreDigiSequence        (enabled) 
                  SimAppPreDigiHist            (enabled)  SimApp: Hists before subsystem digi phase
                  SimAppPreDigiStat            (enabled)  SimApp: Stats before subsystem digi phase
              SvtSimSequence               (enabled) 
                  SvtBuildEnv                  (enabled)  initialize Geometry for the Svt
                  SvtMakeDigis                 (enabled)  makes Digis from Hits
              DchSimSequence               (enabled) 
                  DchInputFilter               (enabled)  filtering GHits
                  DchMakeDigi                  (enabled)  Testing DchDigi
              EmcDigitiseSequence          (enabled) 
                  EmcSparsify                  (enabled)  Reduces waveform list via energy cut
                  EmcWaveformsToDigis          (enabled)  Turns waveforms into Digis
              EmcSimSequence               (disabled)
                  EmcFastGHitsToDigis          (disabled) Fast Ghits to Digis MC
              IfrSimSequence               (enabled) 
                  IfrInputFilter               (enabled)  IFR - GHit pruning
                  IfrGHitToStrip               (enabled)  IFR - strip digitization
                  IfrMakeMCMap                 (enabled)  IFR - create digi <-> MC info map
                  IfrNoise                     (enabled)  IFR - noise generator
                  IfrMakeDigiTime              (enabled)  IFR - time simulation
                  IfrDigiTest                  (enabled)  IFR - Test digitization output
              DrcSimSequence               (enabled) 
                  DrcEnvModuleSequence         (enabled) 
                      DrcBuildEnv                  (enabled)      DIRC - Setup environement for the DIRC
                      DrcBuildGeom                 (enabled)      DIRC - Create the DrcDetector
                  DrcSimModuleSequence         (enabled) 
                      DrcMakeHits                  (enabled)      DIRC - Makes simulated PMT hits from a list of GTrkHits
                      DrcGenerRndBkg               (disabled)     DIRC - Generates random background
                      DrcMakeDigisFromSimHits      (enabled)      DIRC - Makes a list of Digis from a list of SimHits
              SimAppPostDigiSequence       (enabled) 
                  SimAppPostDigiHist           (enabled)  SimApp: Hists after digi phase
                  SimAppPostDigiStat           (enabled)  SimApp: Stats after digi phase
          CommonUtilitySequence        (enabled) 
              PrintModule                  (enabled)      print events
              CpuCheck                     (enabled)      Check CPU time remaining
              Signal                       (enabled)      intercept ^C
              PrintParms                   (disabled)     print all parameters
  
  

• Measure the leak rate after disabling modules/sequences. Since the output of a module may influence the operation of a subsequent module, the disabling must proceed from the bottom of the path list output to the top, i.e. start from PrintParms and proceed to EvtCounter. The change in leak rate can be attributed to the most recently disabled unit. You may find a binary search approach more efficient.