EmcMonTools How-To Page
This document will attempt to explain how to keep the EmcMonTools package up-to-date and
the other responsibilities of the EmcMonTools package coordinator.
This package contains the code for the EMC pages that are monitored by the shifters in IR2.
It also contains ASCII files used for the automated monitoring of the online EMC histograms.
The EmcMonTools Package
The EmcMonTools package is where the EMC's Java Analysis Studio pages are maintained.
Java Analysis Studio or JAS
is a data analysis tool developed at SLAC mainly for the online monitoring
of BaBar data but also now used outside of SLAC.
The HTML and XML pages used by JAS for the monitoring of the EMC are kept in EmcMonTools.
Online histograms can also be automatically monitored using a number of other BaBar packages.
The automatic monitoring of the EMC is controlled using the CRI (ASCII) files in EmcMonTools.
The files that are currently being used in IR2 are kept in the following self-service area:
/nfs/bbr-srv02/bfdist/MonitoringData/EmcMonTools
This area is the production release of the EmcMonTools package, which has the following four subdirectories:
(The two OPR (Online Prompt Reconstruction - detailed monitoring) subdirectories are not really the responsibility of the
EmcMonTools package coordinator and have only been used once or twice by the EMC OPR experts.
The EmcMonTools package coordinator is the EMC Live Fast Monitoring OEP (Online Event Processing) 'expert'!)
FastMonHTML/
HTML and XML files for presentation of OEP histograms in JAS
FastMonAuto/
CRI files to provide automatic validations of OEP histograms
OprMonHTML/
HTML and XML files for presentation of OPR histograms in JAS
OprMonAuto/
CRI files to provide automatic validations of OPR histograms
Warning: If you edit files directly in the production area
and don't commit the changes to CVS then you are in danger of loosing all your work.
To avoid this happening set up a test release, as is done for the editing of all BaBar software, and tag any
changes in CVS before copying them to this self-service area.
The CVS history of EmcMonTools can be found here.
Reference Histograms
The most regular responsibility of the EmcMonTools package coordinator is to update the reference histograms
from a recent representative run.
These are used in the Java Analysis Studio pages monitored by the shifters in IR2 and for the automated
histogram comparisons.
The EMC reference histograms often need to be updated as new dead channels crop up or are masked out.
If you are not aware when this has happened someone will usually send a mail informing you of new
dead channels or any other reasons for the references to be updated.
Useful (netscape) back to the top link!?
Updating the reference histograms is pretty easy and only really requires care to choose a suitable run to
take the references from. This is usually a recent run that has collected a large amount of data.
To check which recent runs have collected the largest amount of data and did not occur during adverse conditions see the
electronic logbook.
Once you have chosen a run go to the
EMC monitoring pages
and check the postscript file of the run (under EmcMoni)
to see if the histograms represent a typical run. (You can do this by comparing the histograms
from a couple different, largish, runs.)
Now you know the run from which to extract the references you need to setup a link to the appropriate hbook file.
You can pick up recent LiveFastMon hbook files from:
/nfs/bbr-nfs102/pubfastmon/Monitoring/OutputArchive/YYYY/MM/LiveFastMon
The hbook files are not named by their run number but instead by the start time of the run,
which is similar to that in the electronic logbook (although not always exactly the same).
Note that there is often more than one hbook file for large runs and so it is preferable to choose the largest of these.
So the link to the hbook file, which must be called inputlivefastmon.hbook, is setup with
ln -s /nfs/bbr-nfs102/pubfastmon/Monitoring/OutputArchive/YYYY/MM/LiveFastMon/LiveFastMon.YYYYMMDD-XXXXXX.hbook
inputlivefastmon.hbook
where YYYY is the year, MM is the month, DD is the day, and XXXXXX is the time at which the hbook
file was created.
Now just execute reduce.kumac, which is in EmcMonTools/FastMonHTML/ , on this link
(reduce.kumac needs EmcMonTools/FastMonHTML/myhcopy.kumac to be in the same directory)
paw -b reduce
This is a bit of a hack that strips out and renumbers the EMC histograms that are needed,
producing an hbook file called outputlivefastmon.hbook (ignore the error messages
that appear in paw when you execute reduce.kumac).
This has to be done as the LiveFastMon hbook files are large and the histograms are randomly numbered.
Now you are ready to install these new reference
histograms. Write access to the output
directory is restricted to very few machines.
So ssh to bbr-reflin, possibly as user babaremc.
First copy the outputlivefastmon.hbook file to the EMC livefastmon directory (give it an
descriptive name, including the date or the run number), and make a symbolic link to it in the
same directory called LiveFastMon.hbook.
Then run a script to merge this hbook file with those from other subsystems.
(editing of any online code must now be done
from a special IR2 machine, bbr-reflin.):
cp outputlivefastmon.hbook /nfs/bbr-srv02/u4/Monitoring/References/Emc/LiveFastMon/emc_ref_run[RUNNUM].hbook
cd /nfs/bbr-srv02/u4/Monitoring/References/Emc/LiveFastMon/
ln -s emc_ref_run[RUNNUM].hbook LiveFastMon.hbook
Accoring to the updated instructions from Jim Hamilton all you need to do now is to type
/nfs/bbr-srv02/bfdist/MonitoringData/LiveFastMon-hbook-install
LiveFastMon-hbook-install merges the EMC histograms with those of the other detector
subsystems, currently being used, into one hbook file.
This hbook file is used by the JAS reference server (LiveFastMon-ref) and in automated
histogram comparisons.
You will be prompted for a one-line description of your change.
After you have updated the merged reference hbook file, the new reference histograms
will not show up on the JAS
pages until the LiveFastMon-ref server is
restarted. This can be done by the DQM shifter via the FastMonControl gui on bfo-con05. Select the appropriate green button labeled "Reference" and choose "Restart" from the pulldown menu.
Now you can view the results in the JAS display.
The following is probably obsolete:
cd /nfs/bbr-srv02/u4/Monitoring/MonitoringData
srtpath
LiveFastMon-hbook-install . LiveFastMon
(you will be prompted to write a short explanation of the reason for the
update, which will be mailed to Edward Chen and Gregory Dubois-Felsmann)
tcsh (if not already using the tc shell)
chmod -R g+w References/merged/LiveFastMon.hbook
(write permission is now set by LiveFastMon-hbook-install)
This will be done by the OEP coordinator (who is automatically mailed by LiveFastMon-hbook-install) when they have a chance.
If you want to make sure it is done immediately you can ask the shifter by mailing them the instructions below.
You can do this yourself but it freezes the JAS system for a couple of minutes surprising the shifter and causing them to
page the OEP coordinator! The server automatically restarts within a couple of minutes after it is shut down.
When it restarts it will use the new references. So if it is not already running when you update the references
there is no need to restart it.
To check whether the server is currently running type
TaoNSDumper | grep LiveFastMon
A list should appear, which will contain "LiveFastMon-ref" if the server is running.
For the shifter to restart the server they need to do the following as user babar from bbr-con18
cd /static/logfiles/Oep/tmp/refServerTest
srtpath
DotAdmin -n LiveFastMon-ref -s
hbsRun >& /dev/null &
If you want the automated monitoring to start using the new references immediately the
system needs to be reset.
You can do this yourself or ask the shifter. The following should work for user babar on bbr-con03 (more details
here)
cd /nfs/bbr-srv02/bfdist/FastMonAuto/
srtpath
cd workdir
DhpMonAdmin -n LiveFastMon-mon -s (or FastMonAutoAppKill)
This will stop the automated monitoring process which will then be automatically restarted within five minutes.
If the reason for updating the references effects the automated monitoring but is not an immedite problem
and you'd prefer to let the OEP coordinator restart the server you must make a note of this
in the mail you submit with LiveFastMon-hbook-install script.
Otherwise the OEP coordinator will probably only restart the JAS reference server.
Some of the references, such as the one that gives the current dead channels, are used for the automated
monitoring. So if the server is not restarted
EMC error messages will continuously be generated on the shifters CMLOG screen from the old references.
(More on automated monitoring below.)
(Note: You can also pick up orderly numbered histograms
in the EMC hbook files which are created by
the EMC OEP fast monitoring
cronjob and deposited in:
$BFROOT/detector/emc/hist/
but you should not use the hbook files in this area to make reference histograms since
a different trigger is currently being used.
For the EmcLive job, it skips 1000 events and for the next 4000 events
it takes 1 out of every 5 events.)
Using JAS to View Live Histograms
To connect to the JAS server in IR2 and view live data, type:
/nfs/bbr-srv02/bfdist/LiveFastMon/bin/SunOS5/RemoteBabarJas
or if that fails, the old alternative was:
BabarJas -c bbr-srv05.slac.stanford.edu/Histograms
-s file:$BFROOT/www/Detector/Operations/shifts/LiveFastMon-html/EmcFastMon/index.html
It is probably better now to mail the online GUIs HN or
Victor Serbo if the first command doesn't work.
Using JAS to View Histograms from HBook Files
To view the histograms from an hbook file in a JAS browser you need to run three processes
from three different terminals: a histogram reference server; a JAS interface server; and a JAS browser
connected to the JAS interface server. The fuller, more general details on how to do this are
here, but the instructions below are enough to do this for the EMC.
For now this only works on SUN machines.
-
Firstly, open up 3 terminals on a SUN machine (bbr-dev20 was recommended to me).
In the first terminal create the release (if not already created), cd into the release directory and do
(at the time of writing the only release that could be used for this was 8.6.3-online-1, you could
instead try the latest online release)
srtpath
To be on the safe side cd into this release and issue the srtpath command in the other 2 terminals. Now,
back in the first terminal (if not already done) add the latest version of the EmcMonTools package so you
can develop it locally (using the HEAD tag ensures that the latest version is used).
Now choose a hbook file (see previous section) and start the hbook reference server:
HbkServer -n emcserver -f myfilename.hbook
where emcserver is the name of your server (you can call it anything) and
myfilename.hbook is the name of your hbook file which has to be in lower case ("HbkServer -h"
explains the options).
When "Server will not connect to a requestor" appears in the terminal you can
proceed to the next step.
-
In the second terminal (logged onto the same machine to make things easy)
create a JAS connection to your hbook server from the same release directory (and after doing srtpath) with
JasInterfaceServer -n LiveFastMon@Dot/emcserver -u 5 -V
where emcserver is the name of your hbook server and LiveFastMon is the
top level folder for the histograms seen by the DQM in IR-2 of which the
EMC histograms are a subset ("JasInterfaceServer -h" explains the options).
-
Then from the third terminal start JAS and connect it to your
JasInterfaceServer and the JAS page that you want to view using:
BabarJas --connect myhost.slac.stanford.edu/Histograms
--startPage file:pathToYourRelease/EmcMonTools/FastMonHTML/index.html
where myhost is the machine on which you are running the JasInterfaceServer
and pathToYourRelease is the full directory name of your test
release - for example /u/ec/huw/EMC/OnLine863 (again "BabarJas -h" explains the options).
If you want to check out the version being used in ir2 you should be able to replace pathToYourRelease
by /nfs/bbr-srv02/bfdist/Builds/MonitoringData although there is a single script for viewing exactly
the same JAS pages as in IR2 as given above.
The EMC's histograms are in the JAS folder (or "path")
Histograms/LiveFastMon/EmcOepDiag and the reference histograms are in the path
Histograms/LiveFastMon-ref/EmcOepDiag.
When you are done with JAS, you have to close down the three processes in the following order:
- Exit from the JAS browser itself and selecting the "no" option when asked if you
want to save any histograms.
- Press return in the window running the JasInterfaceServer
- Type "^C" once in the window running the HbkServer
Updating JAS HTML and XML Pages
The HTML and XML files which are used by JAS in the Live Fast Monitoring are stored in EmcMonTools/FastMonHTML .
You can edit the HTML or XML files in your test release
and then see the results by reloading the HTML page in the JAS browser.
The BaBar JAS page
has some useful links including
instructions
for creating XML files. Below is essentially a summary of these instructions.
If you have problems with JAS functionality then mail questions to
Victor Serbo and/or the
Online GUIs
HyperNews.
If you wish to make
a new XML file you can either copy and edit an existing one or you can load an individual
histogram from the JAS tree in the frame on the left of the JAS browser, right-click on
the histogram to change the way in which the histogram is displayed and then, by left-clicking
on the "Save Plot As.." option, save it as an XML file with the live data option set. You will then
have to edit the XML file itself to include the path to the histogram you want using the LiveFastMon
server for live histograms or the LiveFastMon-ref server for reference
histograms. (You may also want to edit the XML file to improve on the standard formatting.)
In EmcMonTools the XML files for reference histograms have filenames ended by "_ref.xml"
but otherwise have the same naming convention as the live versions.
Here is an example of path to a live histogram:
<class name="jas.bean.TreeDataSourceProvider"
param="Histograms/LiveFastMon/EmcOepDiag/diag(302): Hitmap"/>
The path to the corresponding reference histogram would then be:
<class name="jas.bean.TreeDataSourceProvider"
param="Histograms/LiveFastMon-ref/EmcOepDiag/diag(302): Hitmap"/>
If you want to make a fixed reference histogram which does not change when the histogram
reference server is updated
then make an XML file in the same way as described above
but with the "save current snapshot of data" option set so that the bin contents
of the on-screen histogram are stored within the saved file.
If you just want to update the contents of a customised, nicely formatted, XML file which has stored bin data in it,
then use JAS to make a new fixed reference XML file (as just described), delete all the badly formatted
lines around the bin contents, copy the nicely formatted lines from your customised XML file, and, finally,
overwrite your customised XML file with the new, updated version you have made.
Updating EmcMonTools in the Production Area
To install a new, official version of EmcMonTools in the production area, just do
(editing of any online code must now be done from an IR2 machine, such as bbr-dev100)
cd /nfs/bbr-srv02/bfdist/MonitoringData/
srtpath
addpkg EmcMonTools VXX-XX-XX
where VXX-XX-XX is the version number. Then, if you have modified the HTML files, type:
LiveFastMon-html-install
(you will be prompted to write a short explanation of the reason for the
update, which will be mailed to Edward Chen and Gregory Dubois-Felsmann)
tcsh (if not already using the tc shell)
chmod -R g+w .
Automated Monitoring
The automated monitoring is performed by comparing reference histograms with live ones
(the references can be different from the ones used for the JAS pages).
Histogram comparisons are determined by Comparison Record Info files.
The main CRI file for automated monitoring of EMC histograms is FastMonAuto/main.cri which simply
refers to other CRI files being used, one for each histogram comparison.
If your comparison finds an error (or warning) with the data currently being collected the messages
are sent to the CMLOG screen, which is constantly monitored by the shifters. If the shifter decides
the problem warrants it they will page the EMC weekly shifter. You can provide the shifter with
additional information on possible problems causing the messages and their severity on the
FastMonHTML/automain.html page, which is linked in the main navigation bar (this is a bit
out-of-date at the moment). You can check all the messages sent to the CMLOG from the file:
/nfs/bbr-srv02/u4/Monitoring/Outputs/LiveFastMon/FastMonAuto/EmcMonTools.log
The most recent documentation on CRI files can be found from the
OPR home page. The QA OPR group have done a lot of work on this recently and have much improved
the methods that can be used in automated monitoring since it was first started in 1999 (which was
essentially the system that I used). Since I have not done any work on this for over a year I have not
had a chance to try the new system or read the documentation. The two BaBar notes by Frederic Brochu
seem a huge improvement on the original documentation. (Although Frederic Brochu has now left the collaboration
and there is currently no package coordinator for DhpToken.)
The original documentation is
DhpToken/ExampleTokenFile.cri and DhpToken/README. Also a useful example of a CRI file
that performs bin-by-bin histogram comparisons was DhpMonitorApps/DhpMonAppTest2.cri.
More details on bin-by-bin comparisons can be found
here but the rest of the web pages
are quite out-of-date now. Although
the original web pages by Paul Bright-Thomas (who has also left) -
writing CRI files -
and Ed Chen -
DhpMonitor -
provide an overview of the system.
Before trying to determine any comparison parameters normalise the reference and the problem histograms and overlay.
These are a few tips I was given for 1D histograms by Frederic Brochu who has now left the collaboration.
Not sure though whether they add to the info in his BaBar notes.
The Kolmogorov-Smirnov test is much less stringent and is less sensitive to
normalisation problems than the chi squared one but 1DRefHist does not work as I wish with either statistical tests.
1DBin is very useful to monitor the apparition of tails
(collection of HOT consecutive channels at the beginning or the end of the
distribution).
I used 1DFunction for distributions which looks really Gaussian (not
approximately). 1DFit is 1DFunction for a restricted range in the
histogram. It also allows to choose between 3 kind of responses,
CLRange, ParRange and ParConsistency. ParRange is very useful if you
expect, for example, that your Gaussian mean may shift a little bit around
the central value you gave on input, and that you expressly wish to keep
your central value in a specific range without having to try to guess
which CL threshold correspond to the central value range you require.
For testing CRI files there may be a release already set up (possibly in the OEP coordinators home directory)
but if not you need to get the latest set of tags to build the DhpToken binary. You might be able to get the
latest tags with the following
cd /nfs/bbr-srv02/bfdist/FastMonAuto
srtpath
showtag
The old list of packages that were needed to build the DhpToken binary were
addpkg DhpDistCorba
addpkg DhpDistCorbaIDL
addpkg DhpMonCorba
addpkg DhpMonFit
addpkg DhpMonInterfaces
addpkg DhpMonitor
addpkg DhpMonitorApps
addpkg DhpToken
Once you have DhpToken compiled you now have the choice either to use the live histograms in IR2 and
an HBOOK file as the reference, or to use two HBOOK files
(one reference file and one test file containing problem histograms).
For the hbook files
you should use reduce.kumac (in EmcMonTools/FastmonHTML/)
so they only contain the EMC histograms you are interested in.
For IR2 + HBOOK
1. First check to see that the LiveFastMon server is running using:
TaoNSDumper | grep LiveFastMon
Note that "LiveFastMon-*" servers are not the same as the LiveFastMon server.
2. You need to specify your HBOOK file and CRI file using the environment:
setenv CRIFILENAME myCRIFILENAME
setenv REFINTERFACENAME myHBOOKFILE
3. Run application using:
DhpMonApp DhpMonIR2.tcl
For HBOOK + HBOOK
1. You need to specify your HBOOK files and CRI file using the environment:
setenv CRIFILENAME myCRIFILENAME
setenv REFINTERFACENAME myREFHBOOKFILE
setenv LIVEINTERFACENAME myLIVEHBOOKFILE
2. Run application using:
DhpMonApp DhpMon2HBOOK.tcl
In both cases, DhpMonApp will execute one loop of comparisons. The messages
will not be sent to the CMLOG server because one is not specified, so CMLOG
log file responses get outputted to the screen instead. Make sure that
file stream responses do not output to the production log files in
/nfs/bbr-srv02/u4/Monitoring/u4/Outputs/LiveFastMon/FastMonAuto.
When you are happy that your CRI file works as it should, tag a new version of EmcMonTools and update
the production area as described in the section above.
The automated monitoring system (DhpMonApp) needs to be restarted for a new CRI file to be recognised in IR2.
You can do this yourself or ask the shifter. The following should work (more details
here)
cd /nfs/bbr-srv02/bfdist/FastMonAuto/
srtpath
cd workdir
DhpMonAdmin -n LiveFastMon-mon -s
This will stop the DhpMonApp process which will then be automatically restarted (now using your
new CRI file) within five minutes.
Keep track of error messages, output to the shifters CMLOG screen, by checking the log file:
/nfs/bbr-srv02/u4/Monitoring/Outputs/LiveFastMon/FastMonAuto/EmcMonTools.log
(Note, however, that in this logfile you should subtract one from the "sliced bin #" to get the phi
channel number, but the theta channel number, or "channel #" is correct.
In the future there should be an option to change the numbering and format of the CMLOG output.
This would avoid the wrong phi number being reported to the EMC shifter or as sometimes happens
the subtraction being applied twice.)
Updating and Testing EmcOepDiag.cc
At some point you may want to add new histograms or change existing ones at the source.
The histograms are created by EmcOepDiag.cc within the EmcOepMon package,
which also includes the emcmoni script that generates the OEP postscripts on the Emc Monitoring web page.
Firstly you should get the approval of the EmcOepMon package coordinator
(currently Danning Dong).
After editing EmcOepDiag.cc and EmcOepDiag.hh from the HEAD of the EmcOepMon package
I have the following probably out-of-date instructions for testing the changes:
bsub -q bldrecoq -o gmake.log gmake lib EmcOepMon.bin
cd workdir
EmcOepReadXtc -f /nfs/tcstage/tcfiles/FILENAME -n NEVENTS ../EmcOepMon/EmcFastMon.tcl
paw
exec ../EmcOepMon/emcmoni.kumac
quit
where FILENAME is the name of the xtc file you wish to run on and NEVENTS is the
number of events.
Note that the job can take quite a while to run so stick to low values like 1000
unless you want
to check the quality flag when you should have NEVENTS greater than 1000. 15000
or more events are
considered to give an accurate quality flag number but it takes hours to run on this
many events.
When you are happy with your new version of EmcOepMon, tag it and ask the current OEP coordinator
Not Know for it to be included in the next LiveFastMon build.
The changes will not take effect until the whole of LiveFastMon is rebuilt.
Useful Links
You should subscribe to the following HyperNews groups:
The high traffic BaBar Detector Operations HN is
here.
When at SLAC you should try to attend the fortnightly OEP meetings in IR2 on Wednesdays at 4pm - more info on the
OEP home page.
The BaBar JAS home page is
here.
The instructions for the shifter, which includes useful info on the Live Fast Monitoring scripts, are
here.
It may be necessary to provide the shifters with information for the short-term, which is done on the
important short-term DQM shift information page.
The CVS history of EmcMonTools can be found
here.
You can view the EMC HTML files used by JAS
here.
EmcMonTools is now included in online releases. If you are ever asked to update your tag you can do that
here. (This does not effect the
version used in IR2 as described above).
The EMC home page
has many links although none are directly related to the responsibilities
of the EmcMonTools package coordinator except the
OEP monitoring page (and this page itself).
Some Useful Tips
Setting up the following symbolic links makes life easier:
EmcMonTools -> /nfs/bbr-srv02/bfdist/MonitoringData/EmcMonTools
EmcRefHistos -> /nfs/bbr-srv02/u4/FastMon/References/Emc/LiveFastMon
LiveHistos -> /nfs/bbr-srv02/u4/FastMon/Outputs/LiveFastMon
EmcMonTools.log ->
/nfs/bbr-srv02/u4/Monitoring/Outputs/LiveFastMon/FastMonAuto/EmcMonTools.log
As does the alias to the RemoteBabarJas script (which can be added to your login file):
alias jas "/nfs/bbr-srv02/bfdist/LiveFastMon/bin/SunOS5/RemoteBabarJas"
There are useful kumacs in the EmcMonTools/FastMonHTML directory: fillingaps.kumac reinstates dead channels
in the channel map creating a hbook file which can be used in testing CRI files for finding dead channels;
and reduce.kumac along with myhcopy.kumac (used above) produces a reduced
hbook file containing certain emcoepdiag histograms. (I've never used and mark never mentioned addnoise.kumac
so not sure if this is useful.)
Finally please keep this page up-to-date!
|
