SLAC PEP-II
BABAR
SLAC<->RAL
Babar logo
HEPIC E,S & H Databases PDG HEP preprints
Organization Detector Computing Physics Documentation
Personnel Glossary Sitemap Search Hypernews
Unwrap page!
Det. Search
Who's who?
Meetings
FAQ
Images
Archive
Systems
Performance
Intern. region
Vertex Tracker
Drift chamber
DIRC
Calorimeter
IFR
LST
Magnet
Electronics
Trigger
Operations
Run Coordination
Contact Experts
Shift Takers Info
Operations Manual
Electronic Logbook
Ops Hypernews
Shift Signup
Check this page for HTML 4.01 Transitional compliance with the
W3C Validator
(More checks...)

EmcMonTools How-To Page

This document will attempt to explain how to keep the EmcMonTools package up-to-date and the other responsibilities of the EmcMonTools package coordinator. This package contains the code for the EMC pages that are monitored by the shifters in IR2. It also contains ASCII files used for the automated monitoring of the online EMC histograms.

The EmcMonTools Package

The EmcMonTools package is where the EMC's Java Analysis Studio pages are maintained. Java Analysis Studio or JAS is a data analysis tool developed at SLAC mainly for the online monitoring of BaBar data but also now used outside of SLAC. The HTML and XML pages used by JAS for the monitoring of the EMC are kept in EmcMonTools. Online histograms can also be automatically monitored using a number of other BaBar packages. The automatic monitoring of the EMC is controlled using the CRI (ASCII) files in EmcMonTools.

The files that are currently being used in IR2 are kept in the following self-service area:

/nfs/bbr-srv02/bfdist/MonitoringData/EmcMonTools
		
This area is the production release of the EmcMonTools package, which has the following four subdirectories:

(The two OPR (Online Prompt Reconstruction - detailed monitoring) subdirectories are not really the responsibility of the EmcMonTools package coordinator and have only been used once or twice by the EMC OPR experts. The EmcMonTools package coordinator is the EMC Live Fast Monitoring OEP (Online Event Processing) 'expert'!)

FastMonHTML/
    HTML and XML files for presentation of OEP histograms in JAS
FastMonAuto/
    CRI files to provide automatic validations of OEP histograms
OprMonHTML/
    HTML and XML files for presentation of OPR histograms in JAS
OprMonAuto/
    CRI files to provide automatic validations of OPR histograms
		

Warning: If you edit files directly in the production area and don't commit the changes to CVS then you are in danger of loosing all your work. To avoid this happening set up a test release, as is done for the editing of all BaBar software, and tag any changes in CVS before copying them to this self-service area. The CVS history of EmcMonTools can be found here.

Reference Histograms

The most regular responsibility of the EmcMonTools package coordinator is to update the reference histograms from a recent representative run. These are used in the Java Analysis Studio pages monitored by the shifters in IR2 and for the automated histogram comparisons. The EMC reference histograms often need to be updated as new dead channels crop up or are masked out. If you are not aware when this has happened someone will usually send a mail informing you of new dead channels or any other reasons for the references to be updated.

Useful (netscape) back to the top link!?

Updating the reference histograms is pretty easy and only really requires care to choose a suitable run to take the references from. This is usually a recent run that has collected a large amount of data. To check which recent runs have collected the largest amount of data and did not occur during adverse conditions see the electronic logbook. Once you have chosen a run go to the EMC monitoring pages and check the postscript file of the run (under EmcMoni) to see if the histograms represent a typical run. (You can do this by comparing the histograms from a couple different, largish, runs.)

Now you know the run from which to extract the references you need to setup a link to the appropriate hbook file. You can pick up recent LiveFastMon hbook files from:

/nfs/bbr-nfs102/pubfastmon/Monitoring/OutputArchive/YYYY/MM/LiveFastMon
		
The hbook files are not named by their run number but instead by the start time of the run, which is similar to that in the electronic logbook (although not always exactly the same). Note that there is often more than one hbook file for large runs and so it is preferable to choose the largest of these. So the link to the hbook file, which must be called inputlivefastmon.hbook, is setup with
ln -s /nfs/bbr-nfs102/pubfastmon/Monitoring/OutputArchive/YYYY/MM/LiveFastMon/LiveFastMon.YYYYMMDD-XXXXXX.hbook 
  inputlivefastmon.hbook
		
where YYYY is the year, MM is the month, DD is the day, and XXXXXX is the time at which the hbook file was created. Now just execute reduce.kumac, which is in EmcMonTools/FastMonHTML/ , on this link (reduce.kumac needs EmcMonTools/FastMonHTML/myhcopy.kumac to be in the same directory)
paw -b reduce
		
This is a bit of a hack that strips out and renumbers the EMC histograms that are needed, producing an hbook file called outputlivefastmon.hbook (ignore the error messages that appear in paw when you execute reduce.kumac). This has to be done as the LiveFastMon hbook files are large and the histograms are randomly numbered.

Now you are ready to install these new reference histograms. Write access to the output directory is restricted to very few machines. So ssh to bbr-reflin, possibly as user babaremc. First copy the outputlivefastmon.hbook file to the EMC livefastmon directory (give it an descriptive name, including the date or the run number), and make a symbolic link to it in the same directory called LiveFastMon.hbook. Then run a script to merge this hbook file with those from other subsystems. (editing of any online code must now be done from a special IR2 machine, bbr-reflin.):

cp outputlivefastmon.hbook /nfs/bbr-srv02/u4/Monitoring/References/Emc/LiveFastMon/emc_ref_run[RUNNUM].hbook
cd /nfs/bbr-srv02/u4/Monitoring/References/Emc/LiveFastMon/
ln -s emc_ref_run[RUNNUM].hbook LiveFastMon.hbook

Accoring to the updated instructions from Jim Hamilton all you need to do now is to type

/nfs/bbr-srv02/bfdist/MonitoringData/LiveFastMon-hbook-install
LiveFastMon-hbook-install merges the EMC histograms with those of the other detector subsystems, currently being used, into one hbook file. This hbook file is used by the JAS reference server (LiveFastMon-ref) and in automated histogram comparisons. You will be prompted for a one-line description of your change.

After you have updated the merged reference hbook file, the new reference histograms will not show up on the JAS pages until the LiveFastMon-ref server is restarted. This can be done by the DQM shifter via the FastMonControl gui on bfo-con05. Select the appropriate green button labeled "Reference" and choose "Restart" from the pulldown menu. Now you can view the results in the JAS display.

The following is probably obsolete:

cd /nfs/bbr-srv02/u4/Monitoring/MonitoringData
srtpath 
LiveFastMon-hbook-install . LiveFastMon 
    (you will be prompted to write a short explanation of the reason for the 
     update, which will be mailed to Edward Chen and Gregory Dubois-Felsmann)
tcsh (if not already using the tc shell)
chmod -R g+w References/merged/LiveFastMon.hbook 
    (write permission is now set by LiveFastMon-hbook-install)
		
This will be done by the OEP coordinator (who is automatically mailed by LiveFastMon-hbook-install) when they have a chance. If you want to make sure it is done immediately you can ask the shifter by mailing them the instructions below. You can do this yourself but it freezes the JAS system for a couple of minutes surprising the shifter and causing them to page the OEP coordinator! The server automatically restarts within a couple of minutes after it is shut down. When it restarts it will use the new references. So if it is not already running when you update the references there is no need to restart it. To check whether the server is currently running type
TaoNSDumper | grep LiveFastMon
		
A list should appear, which will contain "LiveFastMon-ref" if the server is running. For the shifter to restart the server they need to do the following as user babar from bbr-con18
cd /static/logfiles/Oep/tmp/refServerTest
srtpath
DotAdmin -n LiveFastMon-ref -s
hbsRun >& /dev/null & 
		

If you want the automated monitoring to start using the new references immediately the system needs to be reset. You can do this yourself or ask the shifter. The following should work for user babar on bbr-con03 (more details here)

cd /nfs/bbr-srv02/bfdist/FastMonAuto/
srtpath
cd workdir
DhpMonAdmin -n LiveFastMon-mon -s  (or FastMonAutoAppKill)
		

This will stop the automated monitoring process which will then be automatically restarted within five minutes. If the reason for updating the references effects the automated monitoring but is not an immedite problem and you'd prefer to let the OEP coordinator restart the server you must make a note of this in the mail you submit with LiveFastMon-hbook-install script. Otherwise the OEP coordinator will probably only restart the JAS reference server. Some of the references, such as the one that gives the current dead channels, are used for the automated monitoring. So if the server is not restarted EMC error messages will continuously be generated on the shifters CMLOG screen from the old references. (More on automated monitoring below.)

(Note: You can also pick up orderly numbered histograms in the EMC hbook files which are created by the EMC OEP fast monitoring cronjob and deposited in: $BFROOT/detector/emc/hist/ but you should not use the hbook files in this area to make reference histograms since a different trigger is currently being used. For the EmcLive job, it skips 1000 events and for the next 4000 events it takes 1 out of every 5 events.)

Using JAS to View Live Histograms

To connect to the JAS server in IR2 and view live data, type:

/nfs/bbr-srv02/bfdist/LiveFastMon/bin/SunOS5/RemoteBabarJas
		
or if that fails, the old alternative was:

BabarJas -c bbr-srv05.slac.stanford.edu/Histograms 
    -s file:$BFROOT/www/Detector/Operations/shifts/LiveFastMon-html/EmcFastMon/index.html
		
It is probably better now to mail the online GUIs HN or Victor Serbo if the first command doesn't work.

Using JAS to View Histograms from HBook Files

To view the histograms from an hbook file in a JAS browser you need to run three processes from three different terminals: a histogram reference server; a JAS interface server; and a JAS browser connected to the JAS interface server. The fuller, more general details on how to do this are here, but the instructions below are enough to do this for the EMC. For now this only works on SUN machines.

  1. Firstly, open up 3 terminals on a SUN machine (bbr-dev20 was recommended to me). In the first terminal create the release (if not already created), cd into the release directory and do (at the time of writing the only release that could be used for this was 8.6.3-online-1, you could instead try the latest online release)

    srtpath
    		    
    To be on the safe side cd into this release and issue the srtpath command in the other 2 terminals. Now, back in the first terminal (if not already done) add the latest version of the EmcMonTools package so you can develop it locally (using the HEAD tag ensures that the latest version is used). Now choose a hbook file (see previous section) and start the hbook reference server:
    HbkServer -n emcserver -f myfilename.hbook
    		    

    where emcserver is the name of your server (you can call it anything) and myfilename.hbook is the name of your hbook file which has to be in lower case ("HbkServer -h" explains the options). When "Server will not connect to a requestor" appears in the terminal you can proceed to the next step.

  2. In the second terminal (logged onto the same machine to make things easy) create a JAS connection to your hbook server from the same release directory (and after doing srtpath) with
    JasInterfaceServer -n LiveFastMon@Dot/emcserver -u 5 -V
    		    

    where emcserver is the name of your hbook server and LiveFastMon is the top level folder for the histograms seen by the DQM in IR-2 of which the EMC histograms are a subset ("JasInterfaceServer -h" explains the options).

  3. Then from the third terminal start JAS and connect it to your JasInterfaceServer and the JAS page that you want to view using:

    BabarJas --connect myhost.slac.stanford.edu/Histograms 
        --startPage file:pathToYourRelease/EmcMonTools/FastMonHTML/index.html
    		    

    where myhost is the machine on which you are running the JasInterfaceServer and pathToYourRelease is the full directory name of your test release - for example /u/ec/huw/EMC/OnLine863 (again "BabarJas -h" explains the options). If you want to check out the version being used in ir2 you should be able to replace pathToYourRelease by /nfs/bbr-srv02/bfdist/Builds/MonitoringData although there is a single script for viewing exactly the same JAS pages as in IR2 as given above.

    The EMC's histograms are in the JAS folder (or "path") Histograms/LiveFastMon/EmcOepDiag and the reference histograms are in the path Histograms/LiveFastMon-ref/EmcOepDiag.

When you are done with JAS, you have to close down the three processes in the following order:

  1. Exit from the JAS browser itself and selecting the "no" option when asked if you want to save any histograms.
  2. Press return in the window running the JasInterfaceServer
  3. Type "^C" once in the window running the HbkServer

Updating JAS HTML and XML Pages

The HTML and XML files which are used by JAS in the Live Fast Monitoring are stored in EmcMonTools/FastMonHTML . You can edit the HTML or XML files in your test release and then see the results by reloading the HTML page in the JAS browser.

The BaBar JAS page has some useful links including instructions for creating XML files. Below is essentially a summary of these instructions. If you have problems with JAS functionality then mail questions to Victor Serbo and/or the Online GUIs HyperNews.

If you wish to make a new XML file you can either copy and edit an existing one or you can load an individual histogram from the JAS tree in the frame on the left of the JAS browser, right-click on the histogram to change the way in which the histogram is displayed and then, by left-clicking on the "Save Plot As.." option, save it as an XML file with the live data option set. You will then have to edit the XML file itself to include the path to the histogram you want using the LiveFastMon server for live histograms or the LiveFastMon-ref server for reference histograms. (You may also want to edit the XML file to improve on the standard formatting.) In EmcMonTools the XML files for reference histograms have filenames ended by "_ref.xml" but otherwise have the same naming convention as the live versions. Here is an example of path to a live histogram:

<class name="jas.bean.TreeDataSourceProvider" 
    param="Histograms/LiveFastMon/EmcOepDiag/diag(302): Hitmap"/>
		
The path to the corresponding reference histogram would then be:
<class name="jas.bean.TreeDataSourceProvider" 
    param="Histograms/LiveFastMon-ref/EmcOepDiag/diag(302): Hitmap"/>
		

If you want to make a fixed reference histogram which does not change when the histogram reference server is updated then make an XML file in the same way as described above but with the "save current snapshot of data" option set so that the bin contents of the on-screen histogram are stored within the saved file.

If you just want to update the contents of a customised, nicely formatted, XML file which has stored bin data in it, then use JAS to make a new fixed reference XML file (as just described), delete all the badly formatted lines around the bin contents, copy the nicely formatted lines from your customised XML file, and, finally, overwrite your customised XML file with the new, updated version you have made.

Updating EmcMonTools in the Production Area

To install a new, official version of EmcMonTools in the production area, just do (editing of any online code must now be done from an IR2 machine, such as bbr-dev100)

cd /nfs/bbr-srv02/bfdist/MonitoringData/
srtpath
addpkg EmcMonTools VXX-XX-XX
		
where VXX-XX-XX is the version number. Then, if you have modified the HTML files, type:
LiveFastMon-html-install
    (you will be prompted to write a short explanation of the reason for the 
     update, which will be mailed to Edward Chen and Gregory Dubois-Felsmann)
tcsh (if not already using the tc shell)
chmod -R g+w .
		

Automated Monitoring

The automated monitoring is performed by comparing reference histograms with live ones (the references can be different from the ones used for the JAS pages). Histogram comparisons are determined by Comparison Record Info files. The main CRI file for automated monitoring of EMC histograms is FastMonAuto/main.cri which simply refers to other CRI files being used, one for each histogram comparison. If your comparison finds an error (or warning) with the data currently being collected the messages are sent to the CMLOG screen, which is constantly monitored by the shifters. If the shifter decides the problem warrants it they will page the EMC weekly shifter. You can provide the shifter with additional information on possible problems causing the messages and their severity on the FastMonHTML/automain.html page, which is linked in the main navigation bar (this is a bit out-of-date at the moment). You can check all the messages sent to the CMLOG from the file:

/nfs/bbr-srv02/u4/Monitoring/Outputs/LiveFastMon/FastMonAuto/EmcMonTools.log
		

The most recent documentation on CRI files can be found from the OPR home page. The QA OPR group have done a lot of work on this recently and have much improved the methods that can be used in automated monitoring since it was first started in 1999 (which was essentially the system that I used). Since I have not done any work on this for over a year I have not had a chance to try the new system or read the documentation. The two BaBar notes by Frederic Brochu seem a huge improvement on the original documentation. (Although Frederic Brochu has now left the collaboration and there is currently no package coordinator for DhpToken.)

The original documentation is DhpToken/ExampleTokenFile.cri and DhpToken/README. Also a useful example of a CRI file that performs bin-by-bin histogram comparisons was DhpMonitorApps/DhpMonAppTest2.cri. More details on bin-by-bin comparisons can be found here but the rest of the web pages are quite out-of-date now. Although the original web pages by Paul Bright-Thomas (who has also left) - writing CRI files - and Ed Chen - DhpMonitor - provide an overview of the system.

Before trying to determine any comparison parameters normalise the reference and the problem histograms and overlay.

These are a few tips I was given for 1D histograms by Frederic Brochu who has now left the collaboration. Not sure though whether they add to the info in his BaBar notes. The Kolmogorov-Smirnov test is much less stringent and is less sensitive to normalisation problems than the chi squared one but 1DRefHist does not work as I wish with either statistical tests. 1DBin is very useful to monitor the apparition of tails (collection of HOT consecutive channels at the beginning or the end of the distribution). I used 1DFunction for distributions which looks really Gaussian (not approximately). 1DFit is 1DFunction for a restricted range in the histogram. It also allows to choose between 3 kind of responses, CLRange, ParRange and ParConsistency. ParRange is very useful if you expect, for example, that your Gaussian mean may shift a little bit around the central value you gave on input, and that you expressly wish to keep your central value in a specific range without having to try to guess which CL threshold correspond to the central value range you require.

For testing CRI files there may be a release already set up (possibly in the OEP coordinators home directory) but if not you need to get the latest set of tags to build the DhpToken binary. You might be able to get the latest tags with the following

cd /nfs/bbr-srv02/bfdist/FastMonAuto
srtpath
showtag
		
The old list of packages that were needed to build the DhpToken binary were
addpkg DhpDistCorba
addpkg DhpDistCorbaIDL
addpkg DhpMonCorba
addpkg DhpMonFit
addpkg DhpMonInterfaces
addpkg DhpMonitor
addpkg DhpMonitorApps
addpkg DhpToken
		

Once you have DhpToken compiled you now have the choice either to use the live histograms in IR2 and an HBOOK file as the reference, or to use two HBOOK files (one reference file and one test file containing problem histograms). For the hbook files you should use reduce.kumac (in EmcMonTools/FastmonHTML/) so they only contain the EMC histograms you are interested in.

For IR2 + HBOOK

1. First check to see that the LiveFastMon server is running using:

TaoNSDumper | grep LiveFastMon

Note that "LiveFastMon-*" servers are not the same as the LiveFastMon server.

2. You need to specify your HBOOK file and CRI file using the environment:

setenv CRIFILENAME myCRIFILENAME
setenv REFINTERFACENAME myHBOOKFILE

3. Run application using:

DhpMonApp DhpMonIR2.tcl


For HBOOK + HBOOK

1. You need to specify your HBOOK files and CRI file using the environment:

setenv CRIFILENAME myCRIFILENAME
setenv REFINTERFACENAME myREFHBOOKFILE
setenv LIVEINTERFACENAME myLIVEHBOOKFILE

2. Run application using:

DhpMonApp DhpMon2HBOOK.tcl
		

In both cases, DhpMonApp will execute one loop of comparisons. The messages will not be sent to the CMLOG server because one is not specified, so CMLOG log file responses get outputted to the screen instead. Make sure that file stream responses do not output to the production log files in /nfs/bbr-srv02/u4/Monitoring/u4/Outputs/LiveFastMon/FastMonAuto.

When you are happy that your CRI file works as it should, tag a new version of EmcMonTools and update the production area as described in the section above. The automated monitoring system (DhpMonApp) needs to be restarted for a new CRI file to be recognised in IR2. You can do this yourself or ask the shifter. The following should work (more details here)

cd /nfs/bbr-srv02/bfdist/FastMonAuto/
srtpath
cd workdir
DhpMonAdmin -n LiveFastMon-mon -s
		

This will stop the DhpMonApp process which will then be automatically restarted (now using your new CRI file) within five minutes.

Keep track of error messages, output to the shifters CMLOG screen, by checking the log file:

/nfs/bbr-srv02/u4/Monitoring/Outputs/LiveFastMon/FastMonAuto/EmcMonTools.log
		

(Note, however, that in this logfile you should subtract one from the "sliced bin #" to get the phi channel number, but the theta channel number, or "channel #" is correct. In the future there should be an option to change the numbering and format of the CMLOG output. This would avoid the wrong phi number being reported to the EMC shifter or as sometimes happens the subtraction being applied twice.)

Updating and Testing EmcOepDiag.cc

At some point you may want to add new histograms or change existing ones at the source. The histograms are created by EmcOepDiag.cc within the EmcOepMon package, which also includes the emcmoni script that generates the OEP postscripts on the Emc Monitoring web page. Firstly you should get the approval of the EmcOepMon package coordinator (currently Danning Dong). After editing EmcOepDiag.cc and EmcOepDiag.hh from the HEAD of the EmcOepMon package I have the following probably out-of-date instructions for testing the changes:

bsub -q bldrecoq -o gmake.log gmake lib EmcOepMon.bin
cd workdir
EmcOepReadXtc -f /nfs/tcstage/tcfiles/FILENAME -n NEVENTS ../EmcOepMon/EmcFastMon.tcl
paw
exec ../EmcOepMon/emcmoni.kumac
quit
		

where FILENAME is the name of the xtc file you wish to run on and NEVENTS is the number of events. Note that the job can take quite a while to run so stick to low values like 1000 unless you want to check the quality flag when you should have NEVENTS greater than 1000. 15000 or more events are considered to give an accurate quality flag number but it takes hours to run on this many events.

When you are happy with your new version of EmcOepMon, tag it and ask the current OEP coordinator Not Know for it to be included in the next LiveFastMon build. The changes will not take effect until the whole of LiveFastMon is rebuilt.

Useful Links

You should subscribe to the following HyperNews groups:

The high traffic BaBar Detector Operations HN is here.

When at SLAC you should try to attend the fortnightly OEP meetings in IR2 on Wednesdays at 4pm - more info on the OEP home page.

The BaBar JAS home page is here.

The instructions for the shifter, which includes useful info on the Live Fast Monitoring scripts, are here. It may be necessary to provide the shifters with information for the short-term, which is done on the important short-term DQM shift information page.

The CVS history of EmcMonTools can be found here.

You can view the EMC HTML files used by JAS here.

EmcMonTools is now included in online releases. If you are ever asked to update your tag you can do that here. (This does not effect the version used in IR2 as described above).

The EMC home page has many links although none are directly related to the responsibilities of the EmcMonTools package coordinator except the OEP monitoring page (and this page itself).

Some Useful Tips

Setting up the following symbolic links makes life easier:

EmcMonTools -> /nfs/bbr-srv02/bfdist/MonitoringData/EmcMonTools
EmcRefHistos -> /nfs/bbr-srv02/u4/FastMon/References/Emc/LiveFastMon
LiveHistos -> /nfs/bbr-srv02/u4/FastMon/Outputs/LiveFastMon
EmcMonTools.log -> 
    /nfs/bbr-srv02/u4/Monitoring/Outputs/LiveFastMon/FastMonAuto/EmcMonTools.log
		

As does the alias to the RemoteBabarJas script (which can be added to your login file):

alias jas "/nfs/bbr-srv02/bfdist/LiveFastMon/bin/SunOS5/RemoteBabarJas"
		

There are useful kumacs in the EmcMonTools/FastMonHTML directory: fillingaps.kumac reinstates dead channels in the channel map creating a hbook file which can be used in testing CRI files for finding dead channels; and reduce.kumac along with myhcopy.kumac (used above) produces a reduced hbook file containing certain emcoepdiag histograms. (I've never used and mark never mentioned addnoise.kumac so not sure if this is useful.)

Finally please keep this page up-to-date!

Valid HTML 4.01!


Page Maintainer: Huw Shorthouse
Page Creators: Mark Ian Williams and Huw Shorthouse
Last significant update: Sep-26-2002
Last minor update: Dec-16-2005 (H Flaecher)
Expiry date: Jan-1-2006