Be responsive to pages, always carry the pager and carry the cell phone if you are not near another phone. If you have to rely on the cell phone, make sure that the reception is adequate in your area. Some locations in the Bay Area (e.g. parts of Redwood City and San Carlos) are effectively blacked out.

When you start being on call or backup expert, perform the following tasks

Make yourself familiar with the DIRC system and the current run conditions:

• check the DIRC commissioner logbook;
• read the minutes of the last week IR-2 meetings in the Detector Operations Hypernews;
• do the daily DIRC shift tour and check the current status of the DIRC;
• talk to the previous commissioner (mandatory!).

Update all contact information

• on the white board in IR-2;
• on the main operations web page (move the red ball in front of your name!);
• in the DIRC commissioner logbook (move the red bar).

Attend the operations meeting in IR-2

The daily BaBar operations meeting takes place at 3:45 pm in the IR-2 Cornelius Conference room Monday through Friday. A DIRC commissioner has to represent the DIRC there every day. If the on-call commissioner is unable to attend it, he/she should make sure that another DIRC representative will be present to report the DIRC current status.

To prepare this meeting, the DIRC commissioner has to make sure that all DIRC systems work nominally or that eventual problems are being addressed by the proper experts. The relevant DIRC systems are:

Hardware

Photomultipliers (PMTs), Front-End Electronics (FEE), High Voltage (HV), Electronics Cooling Water Chiller, Water Plant (including SOB Water Dump system), Nitrogen Gas Circulation, LED Calibration System, Data Acquisition System (ROMs).
Taking a daily tour of the water plant, gas rack and electronics house to convince yourself that everything is working well is mandatory.

Detector Control

• Check that the DIRC epics panels are all green and that the main parameters (crate temperatures, water and N2 flows, HV channel status etc.) are nominal.
Make sure that the front-end crate fans have been working steadily in the last 24 hours. The logfile of the corresponding cronjob is ~babardrc/CronJobs/plotFans.log. Look for errors in it if you have any doubt about the freshness of the plots displayed on the webpage. The cronjob can be reran by hand using the scripts ~babardrc/CronJobs/fansMonitoringCronJob.
• The chiller power SIAM and water level SIAM can be monitored here.
• The Wiener Crates can be monitored here.
• The dead pmts can be monitored here.
• It is useful to have epics, the alarm handler and some scaler stripcharts always running on your office's (home) computer.

DAQ and Data Quality

• The fastmonitoring plots of all runs must be checked daily (weekends included...) to make sure that the DIRC status remains fine. Instead of checking all runs taken during the last 24h in a row, it is better to look at an handful of runs a few times per day. This should also allow you to catch new problems quicker. The corresponding documentation is available here.
•On-Call person should check the Odd runs and Back-Up person the Even runs.
• Check the background levels in the last 24 hours as well. Minimal documentation is provided with the plots. A README describes the DrcBackgroundMonitor package which is used to produce the plots automatically. The logfiles of the corresponding cronjobs are ~babardrc/CronJobs/bkgMonitoring_daily.log and ~babardrc/CronJobs/bkgMonitoring_biWeekly.log respectively. Look for errors in it if you have any doubt about the freshness of the plots displayed on the webpage. They can be reran by hand using the scripts ~babardrc/CronJobs/dailyBkgMonitorCronJob and ~babardrc/CronJobs/biWeeklyBkgMonitorCronJob.
• OPR QA postscript plots can be located using the PR QA Output page. The current OPR status can be checked in the PR Run Progression webpage. Checking the run quality after reconstruction is not among the ops. crew task: there is a monitoring expert checking the DIRC QA.

Calibration

Global calibrations are taken once every 1-2 days by pilot shifters when beams are lost and PEP does not reinject immediately. Each time new DIRC calibration data are taken, an e-mail is sent to babardrc. When you see this message, you should check the quality of the calibration asap.
Bad calibrations can always be overwritten in the database but the first processing of mis-calibrated runs will be bad. Fixing it will require to reprocess these data which will delay the use of these runs for physics analysis. In addition, bad calibrations can also trigger hardware problems with the calibration systems which need to be addressed promptly.

If you notice during your checks that one automatic script seems not to be working, you should try to fix the problem asap. If it is not possible, make sure that the proper expert is aware of the situation. Having no monitoring data is as worrysome as having bad data!

Maintain the Commissioner Logbook

To help in keeping track of problems that may influence data quality, the DIRC commissioner has to update the DIRC commissioner logbook every day.
A simple 'Nothing to report' (NTR) is much better than no entry at all!

E-logbook

Read the e-log at least once per shift and respond to any comment regarding the DIRC. Report any work on the DIRC in IR-2 in the logbook: remember that you get a proper work authorization by checking in with the pilot before starting to work and by making an entry in the e-logbook.
Start any entry in the e-logbook by [DIRC myName] to allow an easy identification of who wrote what.
If you're talking by phone with the pilot (e.g. after a page), make sure that he/she understands what you're saying and that a summary of the discussion appears in the logbook. If this is not the case, call again or make the entry yourself.
You can also check the new MCC e-log although it is likely to be too technical.

Maintain Shift Instructions

In order to propagate the information on DIRC problems to the BABAR shifters, the DIRC commissioner has to maintain several short-term and long term instruction pages:

• short-term DQM instructions
• long-term DQM instructions
• short-term pilot instructions
• shift leader concise material

Remember that shifters are taught to read long-term instructions once per block of shifts and short-term instructions once a day. So any new guidance should first appear in the short-term instructions (and in the long-term ones if needed).
If new instructions apply for the two shifters, include them in both manuals. Experience shows that pilots don't read DQMs instructions (and vice-versa) and that the quality of the communication between the shifters is not always perfect.

Check reference histos

Browse periodically the JAS plots to make sure that the reference histograms they contain are still up-to-date. Discuss any (valid) DQM comment with your FastMonitoring expert.

Maintain Contact Info

Maintain the DIRC Operations web page. Also, feel free to make changes to the ops. manual, there is plenty of room for improvement!

Maintain the DIRC system contact info in the

• Care and Feeding Manual . This info is updated via cvs -- the first section of the Care and Feeding manual describes the manual's update procedure.
• IR2 contact Database . Check periodically that all DIRC information it contains is up-to-date.

Keep the other commissioners and the DIRC group informed

In addition to updating the daily log, send emails about questions, news or work done to babardrc and post important problem reports or activities in the DIRC Operations Hypernews forum .

Once

• Subscribe to the DIRC Operations and BaBar Detector Operations Hypernews forums.
• Add your e-mail address to the .forward file of the babardrc account (~babardrc/.forward). This e-mail address is used to exchange and discuss information related to the DIRC operations.
• Subscribe to the PEPIIBBR bfmail distribution (pepiibbr) to be notified when BaBar/PEP meetings are scheduled. The subscription procedure is explained here.
• Print a copy of the DIRC NIM paper which is THE reference for the subdetector.
• DIRC commissioners are encouraged to take responsibility for a detector analysis project in coordination with the DIRC software experts and the ops. manager(s). Both actitivies would benefit one from the other!

Weekly

There is a weekly PEP/BaBar meeting, usually every Friday at 1.30 pm. That's the best place to learn things about PEP status and plans. Each subsystem should have at least one representative in the meeting, normally the oncall expert. If you're oncall and can't go, make sure someone will replace you.
BaBar/PEP meetings are announced via the bfmail distribution PEPIIBBR to which you should subscribe.
Every Tuesday night, the oncall commissionner must prepare a short weekly report summarizing what happened in the last week. Just copy and rename the report from the previous week and fill in the new information required. Avoid to be too verbose (details should be put in the DIRC logbook which is our main tool to monitor daily operations) and don't spend much time on hunting lumi numbers: rough precision is enough! On the other hand, post the summary as early as possible (on Saturday at the latest). Then, send a Hypernews to DIRC-Operations.

Bi-weekly to monthly

The commissioner should give a report about the activities since the last DIRC meeting in the DIRC meeting, held on demand, typically once or twice per month, on Wednesday at 9 am in the Sierra room (central lab. 1st floor). If useful, prepare a small talk summarizing the main points to be discussed -- otherwise just go through the last 2 weekly reports. Post any presentation to the DIRC ops. hypernews before the meeting: this will ensure that the provided information is easily available and can be viewed by people outside SLAC.
The DIRC meeting allows for a phone conference. The current phone number is (510) 665-5437 and the meeting ID 5418. If you're not on site and want to connect to it, crosscheck these details with the meeting announcement e-mail posted on the hypernews.

Monthly

One member of the DIRC ops. crew (commissioner or operation manager) has to give the DIRC monthly report. Such reports are given on Wednesday after the IR2 3.45 pm meeting. The monthly report planning is usually written on the white board in the Cornelius room. In any case, a subsystem knows at least one week in advance that it has to give a monthly report.

Collaboration Meeting

At each BABAR collaboration meeting, there is a DIRC talk in one of the detector plenary sessions. One member of the DIRC ops. crew (commissioner or operation manager, alternatively 'European' and 'American') gives a report about the activities since the last meeting and presents the status and the future plans for the DIRC.

The DIRC Coordinate Systems (more information concerning the DIRC Numerology):

The DIRC detector space (dTag) is defined by module, section, chip and channel, like for all BaBar subsystems. The specific DIRC dimensions are (counted as: 0...n-1):

• 6 Read-Out-Modules (ROMs) which perform the FEE data acquisition;
• 28 Sections (DIRC Front-End Boards, DFBs in short) per module;
• 4 TDC chips per section;
• 16 PMT channels per TDC chip, resulting in a total of 10752 = 6*28*4*16 channels (PMTs) for the whole DIRC. Upstream from the TDC chip, there are 2 analogical chips which get data from 8 PMTs each.

The ROMs communicate with the FEE through p-address links. The module, chip and channel coordinates are identical to the dTags and the only difference concerns the sections: p-sections follow the actual slots in the FEE ("Wiener") crates while the dTag sections count in the inverse direction. There are 16 p-sections (slots) in an FEE crate but only 14 of them are actually used for the DIRC.
The lowest non-empty slot (slot 05) is occupied by the DIRC Controller Carte (DCC) which

• steers the input/output of the Fast Control (FC) commands via the Control Link (CLINK);
• sends the measured TDC and ADC data via the Data Link (DLINK);
• provides and feeds slow control CANBUS channels.

The 2 following slots (right after the DCC) are empty. The next 14 slots (slots 8 → 21) of the VME crate are filled with DFBs. The last slot in the crate has the p-section bit (link) 15, while it has the d-section (DFB, also called "harness") 0. The first slot occupied by a DFB (after the DCC and the 2 empty slots) has the p-section bit 2 and the d-section address 14. This is an important difference. For instance, the individual register configuration of the 8 analog chips per DFB is addressed in p-space: the links go from 2^2=0x0004 to 2^15=0x8000. On the other hand, the electronics coordinates (dTags) used in the Online Event Processing (OEP) go from section 0 to 13. For convenience, the distinction between chips and channels is sometimes kludged together to one "DFB channel" coordinate, going from 0 to 63.

The DIRC geometry space consists of (counted as: 0...n-1):

• 12 SOB sectors;
• 41 rows of PMTs;
• 30 columns (the number of columns in a row varies: it is the smaller the lower the row number is).

The sectors correspond to a combination of modules and sections in d-space. The conversion formula reads: For instance, module 0 hosts the sectors 0 (modules 0 → 13) and 1 (modules 14 → 27).

The conversion from "harnesses" (which is equal to a DFB and to a section, for section < 14), and DFB channels within a given sector to columns and rows is non-trivial. A (supposedly) self-explaining FORTRAN program which is the basis of the DIRC geometry model used in the reconstruction code is available for this purpose. It may be directly called from its location using the command (working on Sun machines only):

The conversion is particularly important when one wants to identify bad tubes in a sector. It is more convenient to count columns and rows, but the cables are labeled using harnesses and channels (the actual harness numbering on the detector counts clockwise from 0 to 167 = 12*14-1, seen from outside the SOB).

The DIRC HV channels are organized in groups of 16 PMTs, all equally distributed between 2 harnesses of a same sector. Thus, there are 56 HV groups per sector and 672 for the whole DIRC. The individual HV settings have been chosen in order to maximize the efficiency and purity of the PMT signals. It is therefore crucial to make sure that the group settings visible through EPICS panels correspond to the nominal values.
The sector plus HV group to HV setting conversion is also a feature of the above "position.run" executable.

Troubleshooting: The DIRC coordinates are closely related to the actual cabling of the crates, boards (harnesses), HV groups and signal channels. Miscabling of a HV group will show up as a hot group of 16 channels and a corresponding cold group in the same sector. Miscabled optical fibers will interchange whole sectors. This shows up as interrupted Cherenkov rings in the Event Display. Interchanged harnesses or signal cables however require a high statistics single tube Cherenkov angle analysis. Cable errors are corrected via software during the dTag-to-gTag conversion.

The module IDs (0...5) are set by dip switches on the controller card of the ROM. Wrong Ids show up as invalid or doubled module numbers printed on the xyplex windows after a reboot.

PMTs: The quality of the PMTs is primarily controlled by the periodical calibrations and the fast monitoring data.

Front-End Electronics: The FEE boards (14 DFBs and 1 DCC per sector) are located within the magnetic shielding which renders them inaccessible during normal BaBar and PEP-II operation. The Fast Control global commands in one direction, and the FEE data in the reverse direction, are sent through optical fibers (2 per FEE crate) which connect the 6 ROMs to the 12 DCCs.

Front-End crate Power: The 12 DIRC front-end crates take power from two power boxes located on the backward sides of BaBar: one on the west side (powering sectors #0 to #5) and one on the east side (powering sectors #6 to #11). Labels have been put on the power cords to allow an easy identification of the sector powered by a given cable. The picture below shows the way the 12 cables were plugged in during the October 2005 shutdown.

Front-end crate power cord status on 2005/10/19.

If some cords are moved in the future, it would be nice to update this picture: the xfig file in $BFROOT/www/Detector/DIRC/Operations/ should be used to overwrite the gif picture displayed here.

The two boxes are powered by a panel located on top of the detector. Between this panel and the sector crates, there is a relais (also located on top of the detector, about 2 meters away from the main box) which is controlled by the DIRC chiller: chiller running ↔ relais closed; chiller not running ↔ relais opened. The control line of the relais is supplied by 24 V provided by a power supply located in the electronics hut in a DCH rack (B620B-10, the PS is actually hidden behind a front plate!). The open/close status of the 24V is given by the chiller SIAM output.

High Voltage System: The DIRC HV power is supplied by 6 SY527 CAEN crates. Each crate is powering 2 consecutive SOB sectors. They are located in Rack #5 and #6 in the Electronics House. Each CAEN crates contains 8 independent HV boards which can power each 16 HV channels (called groups). Their status and the HV settings are monitored by dedicated EPICS panels.

An alternative, but discouraged way to connect to the crates are the serial "xyplex" lines, available for each CAEN crate using the command:
xyplex drc-caeni where i=1...6 is the number of the CAEN crate.
Troubleshooting: For security reasons, the access to all HV power supply serial lines has been restricted on August 12th 2005. Therefore, these connections can now only be done from a limited number of machines which have AFS and regular home directories: bbr-reflin, bbr-reflin1 (Linux machines) or bbr-refsun and bbr-refsun1 (SUN machines). By default, users are not allowed to log in these machines. If you think you need such permission for DIRC-related activities, please send an e-mail justifying your request to ir2-admin with the DIRC ops. manager(s) in cc. Otherwise you can connect to these machines using the DIRC generic account babardrc.
Once the xyplex session has been setup, pressing "D" displays

• the individual HV settings;
• the voltage and current 'absolute' limits (respectively 1600 V and 2200 uA);
  The channel voltage is hardware-limited: a small screw on the back plane of each HV board allows one to adjust this limit.
• and the monitored HV and current values.

It is under all circumstances forbidden to operate the HV when the SOB is open to light.
There are 6 closable glass windows in the SOB which allow visual inspection of the inside. One should check that all of them are closed before turning the HV ON. In addition, if one of the optical fibers used to transport light from the DIRC light generator crate to the SOB is unplugged, some light enters inside as well.

In addition to turning the HV off with epics, the following actions are mandatory before starting any work involving the DIRC HV system.

1. Read and sign the non-routine JHAM describing the work planned.
2. Turn off the HV crate using the unique key which should be in one of the crates. This hardware interlock switch guarantees that the HV supply is interrupted.
   Keep the key in your pocket while you're working on the DIRC HV.
   If the key get lost, don't panic: any IR2 CAEN crate key should fit!
3. If you plan to work on an HV board, unplug the power cords at the back of the three crates which are in the rack. Then, wait for the red light to turn off to make sure that all capacitors have dissipated their charges.

Faulty HV boards (or crate) must be brought to Paul Stiles, the SLAC technician in charge of them. Ask Ray Rodriguez for more information if needed.

Electronics Cooling: When the doors of the magnetic shield are closed, the FEE crates need to be cooled to evacuate the power dissipated during the running of the detector. Therefore, cooling radiators are attached to the inner side of the shielding and come close to the crate fan trays when the doors are closed. The cold water circulating in these radiators is supplied by the DIRC Water Chiller. The inspection of the chiller is part of the routine BaBar shift tour.

Water Recycling System: The SOB water circulation system is located behind the Electronics House in the IR2 entrance hall. The inspection of the water system is part of the routine BaBar DQM tour, done once per shift (see the DQM checklist for details). An EPICS panel informs about water level, temperature, pH values. A detailed DIRC expert checklist provides guidelines about how to fill the SOB and how to maintain the cleaning system, and what to do to recover from a power outage.
The DIRC water plant is a sensitive part of the DIRC system. Commissioners should not try to fix anything on it unless they were given special instructions to do so. The safe procedure is to page the water plant experts, Matt McCulloch and Jerry Va'Vra -- if needed.

SLAC pressured air supply: The valves in the Water Recycling System are controlled by pressured air.

Nitrogen Gas Circulation and SOB Water Dump System: The DIRC DIRC Gas System detects water leaks by measuring the humidity level of N₂ within each barbox and looking for water inside the barbox slots. There is a total of 4 leak sensors per barbox, although some are shared between barboxes. Signals on any two sensors on the same barbox will trigger a SOB water dump. The sensor electronics is located in Rack 5, on top of the detector, and the majority logic electronics that creates signal for the dump is located in Rack #38 on the top of the Electronics House. The humidity level and the N₂ flow are read out via the DIRC Gas System Epics panel.

Power Outages: Relatively frequent power outages in the Electronics House due to cooling problems or VESDA alarms need the following actions:

The DIRC has 3 independent Input-Output Controller (IOC) modules to which serial link connections can be established from the proper IR2 machines:

The MON IOC

(serial link: xyplex drc-mon) controls 12 PMT hit rates (scalers), low voltage, temperature and status of the FEE boards and crates, the values of the magnetic sensors and the light transmission intensity of the optical fibers through 2 daisy-chained CANBUS systems, one on the Wiener FEE crates and the other on the DCCs. In addition, the gas humidity and flow readout as well as the SOB status are controlled here. The corresponding setup files (database libraries) are found in the directory:

/nfs/bbr-srv01/u1/babar/boot/apps/drc-mon/db/

The HV IOC

(serial link: xyplex drc-hv) controls the HV configuration and status. The database setup files are found in:

/nfs/bbr-srv01/u1/babar/boot/apps/drc-hv/db/

More information on the HV control system can be found here.

The Gas SIAM IOC

(serial link: xyplex drc-gas) controls the SIAMs of the various gas humidity sensors. Equivalently to the other IOCs, the database configuration files reside in:

/nfs/bbr-srv01/u1/babar/boot/apps/drc-gas/db/

In fact, drc-mon, drc-hv and drc-gas are 3 symbolic links which point to the last versions of the packages, currently used in production. The BaBar packages containing the IOC codes are epics-drc-mon, epics-drc-hv and epics-drc-gas.

Epics variables are normally initialized with upper and lower alarm thresholds. These limits are set in the db/ subdirectory of the corresponding IOC package. The alarm status of variables is visible on the epics panel through a color code (OK, minor alarm, major alarm or not connected) and variables linked to the ALarm Handler (ALH) produce audible and visible alarms on the pilot console when they go outside their nominal range or loose connection.
The DIRC ALH configurations are stored in the alh/ subdirectory of the IOC packages. There are usually two files per ALH section: one for the normal data taking conditions with all alarms enabled; one for the shutdown periods with several alarms disabled. As parts of the DIRC are usually turned off during shutdowns, disabling the alarms prevents from overloading the ALH panels with fake alarms.

To know more on an epics variable middle-click it on the epics panel. This should open a small grey popup window displaying the name of the variable. By clicking then on 'Examine', you'll access some important parameters of the variable: its scanning frequency and its alarm thresholds.

Ambient database: Several DIRC epics variables are archived in the ambient database. These data allow one to monitor the DIRC behavior over long-time periods or to make detailled studies of the DIRC status at a given time in the past. In particular, two important components of the DIRC monitoring (front-end fan speed monitoring and background monitoring) are extracting their data from this database.

Among the monitored variables, one can find:

• the 12 scalers;
• several parameters from the front-end crates;
• currents and voltages for all the HV channels;
• various environment data: N₂ flows, B-field sensors, neutron counter rates, water plant main parameters...
• background monitoring variables of various types.

There are two main ways to access ambient data.

1. To use the ambient browser.
   This is the simplest way to create quick plots. If you're working on a linux SLAC workstation, a convenient way to start it is to click the button 'Ambient Browser' on the right part of the main BaBar epics panel. This should open a window similar to the picture below.
   Screen shot of the ambient browser showing some of the DIRC epics variables archived in Ambient.
   If you're working from a remote site, it is much simpler to install JAS on your computer and to connect to the BaBar ambient server from there. This will prevent you from importing the browser from SLAC in addition to the monitoring data you're interested in! Otherwise, the browser will be very slow and may crash quite often...
2. To use OdcNtupleMaker.
   This nice tool (managed by Andy Salnikov) uses a time range and a list of epics variables as input. It produces a ntuple (PAW or ROOT format depending on the user's request) which contains the time history of all the selected variables over the chosen range. The ntuple can then be analyzed offline using suitable macros.
   The directory $BFROOT/detector/drc/Odc/ shows several examples of input datacards (files with the .inp extension). The conventions used in these text files should be pretty straightforward. The file analysis.kumac contains PAW macros which can be used to analyze ntuples produced by using the .inp input files.
   • TIME_WALK_SCALERS_NEW: plots of the scalers, PEP-II luminosity and LER/HER currents versus time;
   • FREQ_VS_I: the scaler rates versus LER current or luminosity for all sectors including fits of the linear and quadratic components, if required;
   • I_AND_HV, ALL_V: the current and HV stability;
   • PLOT_TEMP: the FEE temperatures;
   • TIME_WALK_HUM_FLOW: the gas flow and humidity versus time.
   Warning: this directory has not been used for years and so its contents are expected to be quite old-fashioned. Therefore, use them with care: they are good examples of what one can do with OdcNtupleMaker but they may not be directly usable.

Adding new channels in Ambient is quite straightforward. Details can be found in a dedicated guidance webpage or in this hypernews.

Data Acquisition System (ROMs): The DIRC DAQ system has 6 "segment" level ROMs and 2 "fragment" level (slot-one) ROMs divided up into 3 logical DAQ crates, located in the same physical crate in the Electronics House. In the dataflow conventions, these crates are 18 (0x40000), 19 (0x80000) and 20 (0x100000). The "segment" level ROMs communicate over optical fibers through the DCCs with the DFBs, send commands and receive the TDC and ADC as well as header and status data.

The datagrams are received with a clock frequency of 15MHz. Once arrived in the single board computer in a ROM, they are "feature extracted" (FX) i.e. cleaned up from header and status words and checked on their structure. The results of these checks are sent to the event level as an FX-status word which is traced in the DIRC fast monitor. Status values above 20 are fatal errors for which FX sets the dedicated FX-damage bit. Currently, all ADC data are truncated within FX in order to speed up the dataflow within the ROMs and to prevent dead time. In presence of high machine background, it may happen that the FX status reaches 9 or higher which means that TDC data are truncated too. This must be observed as it affects the DIRC reconstruction (loss of data).
The most recent (August 2005) information on the FX status can be found here.

The "fragment" level ROMs collect the data from their logical DAQ crates and send them to the event level (UNIX).

In order to debug ROM functions like library loading, configuration, calibration or running, it is useful to open serial lines through the xyplex board. The command

opens 6 xyplex connections (xterms) for the 6 segment level ROMs. As only one serial line can be open at a time, one may need to use first which closes all existing DIRC DAQ xyplex connections.

FEE and Finite State Machine (FSM) Configuration: Before starting a data taking run or a calibration, the individual registers of the DFBs need to be configured. This is performed during the so-called "Configure Transition". The system which drives the configuration is "Reverse Dataflow" (RDF) which itself is driven by "keys". Each key (top-key) is linked to a hierarchy of configuration data for each subsystem involved. These data are usually read from files (in xtc format) and the correspondance between a key and the files in the configuration tree is fetched from the configuration database during the configure transition. The DIRC has 2 types of configuration files: DrcRegisters for the FEE register configuration, and DrcCycles for the FSM and FX or Calibration configuration. The corresponding files are located in:

There are 6 different DrcRegisters configuration files, one for each ROM. The executable TestDrcRegisters (in the same repository) allows to transform a xtc file into readable ASCII format. The structure of the FEE configuration is the following:

where the groups are data taking (0) and calibration (1). The links determine which FEE element is addressed by a particular register configuration. LinkA talks to the even sector number served by a ROM and linkB addresses the odd sector number. The only individual register settings define the gains of the analog chips (8 ADCs per DFB, register addresses 0x0-0x7) which are different for each ADC. The DCCs have 2 sectors (calibration strobe ON and the calibration strobe delay) which must be addressed through the first DFB in the front-end crate which is linkA/B = 0x1.

See also the section Masking a noisy channels.

FastMon: For information about the Fast Monitoring, check the dedicated webpage.

Calibration: Many of the final tools of the DIRC calibration system are already in place, some are still in development. All DIRC calibrations are started with the help of the calibration GUI (tk library):

Which is an alias defined, e.g., in the setup script of the DIRC account: babardrc. In future this is supposed to be an expert tool for deeper studies and debugging, while all default calibrations are performed by the BaBar DAQ shifter using Run Control (ORC). All calibrations exist in a monitor and a database mode. Both modes produce ntuples which can be analyzed with the help of the various "Check Results"-type buttons. The GUI is a simple command line executer. It starts tcl and csh scripts which then calls the respective actions. All relevant calibration scripts reside in the directory: Temporary files needed for the database validation (there is an absolute validation of the calibration results with respect to some initial ranges and a relative validation by comparison with the previously stored results), various checks, the hbook ntuples and the log files of the different calibration levels (event level, control level) are saved at: The individual DIRC calibrations are:

Online T0 calibration

(using LED flasher): fits the individual T0's of all PMTs. The most important contribution to the T0 derivation between channels originates from the different HV settings: dT0/dHV = (-15 +/- 1) ps/V.

ADC calibration

(using LED flasher): fits the single photoelectron peak and the threshold of the ADC spectrum

Electronics calibration

(using DFB's internal charge injection) -STILL IN PREPARATION-: fit the gain (slope) and the pedestal (ADC offset at zero charge injection) and the threshold (ADC signal rise) using the charge injection facility of the DFBs.

Occupancy calibrations

-STILL IN PREPARATION-: there are 3 different types of occupancy calibrations

• PMT noise calibration taken without beam and LED
• Background calibration taken in presence of beam
• LED occupancy calibration used in order to monitor the LED light intensity and to take fast snapshots of the system for debugging purposes.

DFB serial number calibration

this "calibration" is used in order to attribute the correct gain settings (lab measurements connected to the DFB serial number) to the FEE elements used in the detector. It only needs to be performed when DFBs have been exchanged.

Additional Features: The calibration GUI contains two panels that are very useful shortcuts to frequently used tools: a) the DRC scaler rate history and correlation with the LER current and b) the time history of the calibration values. Both are started by clicking on the aprropriate panel and entering the time interval into the pop-up window (comply with the format given).

Plot Scaler Rates

Clicking on this panel produces an ntuple using the above mentioned OdcNtupleMaker with the standard ntuple variables: HER+LER currents, luminosity, and scaler rates. The input file is defined in the shell script

$BFROOT/detector/drc/DrcCalScripts/runOdcNtupleMaker.csh

The output file is written to $BFROOT/detector/drc/tmp Then the macro $BFROOT/detector/drc/DrcCalScripts/ntuplemaker.kumac is run on the ntuple and produces a 2 page postscript file. Finally, a ghostview process displays the PS file. Quitting ghostview returns you to the GUI.

Plot calibration history

This command used the DrcCondNtupleMaker to plot a large number of calibration constants as a function of time. It generates a 26 page postscipt file and launches a ghostview process to view the output.

Teststand: Upstairs in IR2 (bldg 621, room 201) we have a teststand which is used to test front-end components (power supplies, fans, boards etc.). Spare HV boards, tools etc. are also stored there -- we also have 2 cabinets in this room; the one closer to Ray's office contains the front-end spare boards and the tools used by Orsay engineers to test and fix the DFBs.
Safety guidances:

• Never leave a crate ON on the front workbench (the one facing the entrance from outside) if you're not around.
  This restriction does not apply on the other workbench.
• If you're burning PS or fans, do not put front-end boards in to avoid wearing them out.
• Once you're done with the DCC located in the spare crate, put back the covers on the finisar outputs to prevent dust from going in.

1. First and foremost, never start this procedure unless you are in the BaBar control room. Make sure that the pilot is aware of your work and gives you the green light.

2. Obviously, the DIRC HV need to be set a nominal voltage to take meaningful data.

3. Find in the logbook the last Physics run and write down the current physics key (this is 0x7ba (=1978) as of January 22nd, 2003).

4. Log in to bbr-farm02 (or any other not-overloaded dataflow machine like e.g. bbr-farm04 or bbr-farm07) and open two more terminals.
   (Note: Do not use x-terms because fcgui below may not work and if fgui still does not work below try sourcing drcSetup.csh).

5. Find out what the current release is
   bbr-farm02:~> which showPartitions
   showPartitions: aliased to /afs/slac.stanford.edu/g/babar/dist/online_releases/8.14.1/bin/SunOS58/showPartitions
   In this case it is 8.14.1

6. In one terminal, type (replace 8.14.1 with the current online release)
   cd /afs/slac.stanford.edu/g/babar/dist/online_releases/8.14.1/
   srtpath
   
   <RETURN>
   <RETURN> ← WARNING: before pushing <RETURN> the second time, check that the default BFARCH is the same as the one included in the showPartitions path, normally SunOS58_sparc_WS6U1. If this is not the case, select the correct BFARCH.
   
   cd $drc/bin
   DrcOepLog
   to start OEP (the event level). This will open a new window called "OepLoggingApp". If this doesn't work, check the BFARCH value (see above).

7. Then go to the second terminal and type
   cd $drc/bin

8. Add the directory $ODF_ROOT/package/shlib/$BFARCH to the LD_LIBRARY_PATH:
   setenv LD_LIBRARY_PATH "${LD_LIBRARY_PATH}:$ODF_ROOT/package/shlib/$BFARCH"

9. In this terminal, type
   fcgui -k 0x7ba -c 0x1c0000
   where 0x7ba is to be replaced by the current key (preceeded by the 0x!) and 0x1c0000 is the DIRC crate mask (do not change it!).
   This will open an "ODF Sequencer" window which looks like this:
   

10. Click on the 'Allocate' button.

11. Click on the 'Arguments' button. This will open the following window:
    
    Check that the configuration key is correct; then close this window using the 'Close' button.

12. Click on the button 'MinorCycle'. The button label will first go red and then green. In the message window at the bottom, you will see the 'Last received transition' change. The 'Last damage:' has to stay none. In case of a problem in the partitioning or configuring phases, the key and/or crate mask is likely not to be correct: crosscheck your inputs!
    Another possible problem: the ROMs may need to be rebooted. For instance, ROMs must be rebooted each time the FEEs are switched on... Try to avoid rebooting the ROMs if this is not mandatory.

13. Click on the 'Calibration' button. This will open another dedicated window:
    
    Change the settings in the window to match the ones in above picture:
    
    • Cycle is Internal.
    • Commands 1, 2 and 3 enabled (i.e. button pressed, 'Disable' is visible).
    • Command 1 is set to L1Accept.
    • Command 2 and 3 are set to NoOp with a delay of 64000 (ticks).
    • Counts field is set to 0.
    Once this is done, click on 'Apply' and then 'Close' the window.

14. Click on the 'Enabled' transition in the main 'ODF Sequencer' window. This will start the standalone run.
    Check that data are indeed accumulating on disk in the new xtc file created by the fcgui command. You can use the following command line:
    ls -ltrh /nfs/bbr-srv02/calib/drc/conditions/drc/ | tail -3
    Check the file size regularly to make sure that it is growing at a reasonnable rate.

15. Typically, you want at least 150 kEvents for a random trigger run. This corresponds to an xtc file size just under 100MB.
    Obviously, if some PMTs are noisy (e.g. because there is a Christmas tree ON) events will be much bigger and so the number of recorded events should be significantly reduced. Otherwise, analyzing the hbook will take ages and may even fail!

16. If you take a random trigger run to analyze dead and inefficient PMTs you will have to collect more events. For a typical analysis, you need to take about 5M events. To avoid problems with disk space or hbook file size, please split the 5M events in smaller runs of about 1M triggers each. Remember to check the available disk space for the xtc files on the output disk:
    df /nfs/bbr-srv02/calib/drc/conditions/drc/
    A run with 1M events will take 500-700MB for the xtc file. It may therefore be necessary to move each xtc file to the DIRC NFS disk /nfs/farm/babar/drc/babardrc.
    Don't fill up the /nfs/bbr-srv02/calib/drc file system! If this disk area gets 100% full, the calibration monitoring job will fail next time a calibration is taken!

    Please remember to clean up after yourself when the xtc files are no longer needed.

    If the /nfs/farm/babar/drc/babardrc file system is filling up and if you need disk space there to move files from the bbr-srv02 file system, please consider backing up the data from /nfs/farm/babar/drc/babardrc to mstore. The path to the babardrc mstore repository is /nfs/mstore/u/babardrc. A typical sequence of mstore commands would be:

    • The minimum file size for mstore is 100MB, the maximum 10GB. For larger files use the astore command. If the files are smaller, create a compressed tar archive of the files you want to archive, let's call that myruns.tgz (you could use the $BFROOT/work/b/babardrc area for that purpose).
    • Create an entry in mstore with the command
      mstore create /nfs/mstore/u/babardrc/myruns.tgz
    • Copy the file to the mstore entry with the command
      cp myruns.tgz /nfs/mstore/u/babardrc/myruns.tgz
    • Write the file to tape with the command
      mstore put -p -v /nfs/mstore/u/babardrc/myruns.tgz
    • Now you can safely delete the xtc files and the tar file on /nfs/farm/babar/drc/babardrc. (You will be able to retrieve the file from mstore with the mstore get ... command.)
    • To see what has been stored, just type
      mstore ls /nfs/mstore/u/babardrc
      .
    Don't forget to check first if other users (not babardrc) are responsible for filling up /nfs/farm/babar/drc/babardrc. If someone other than babardrc is using more than a few hundred MB, ask them politely to delete the files asap or have the Slac Computing Service (SCS) delete the files for them...

17. Once you have accumulated enough data, end the run by clicking on the button labelled 'Halted'. This button will end the program and release the partition. The ROMs should all be ready to run again without rebooting.
    Please note, If the run ends abnormally, use the blue buttons labelled 'Reboot' and 'Dissolve'. Click the reboot button first if the ROMs need to be rebooted. You can check in the xyplex windows (you can start those with the
    DrcStartXyplexes
    command) that all the ROMs began to reboot, and use ctrl-X to reboot any which didn't. Then click the 'Dissolve' button. This will gracefully clean up the partition. Failure to do this (e.g., killing the GUI with ctrl-C) may require manual intervention in the master crate.

18. Tell the shift leader that you are done and clean up the terminals.

Looking at the results of a DIRC standalone run

First process the run to make an ntuple - DrcOedPlayback is a convenient way to do that.

• Log in to a fast SunOS IR-2 machine (e.g. bbr-refsun) as babardrc
  cd $drc/bin
  mkdir $BFROOT/work/b/babardrc
  setenv LD_LIBRARY_PATH /afs/slac/package/amulet/SunOS5/lib
  DrcOedPlayback -n <numberOfEvents> -f <filename> tcl/myDrcOed.tcl

  Note: the babardrc work directory might exists, in that case you will see an error message you can ignore. The mkdir command is here to make sure that the default output directory for the hbook file produced by DrcOedPlayback exists -- this directory gets deleted automatically if it remains idle during one week.

  filename is the file created above

  /nfs/bbr-srv02/calib/drc/conditions/drc/filename.xtc
  The job will take some time (~2-3 mins per 100K events) and produce a hbook file named $BFROOT/work/b/babardrc/drcoed.hbook
  On completion it will print the number of events processed. Note this for the next step below if you did not run xtcCat before.

• Now look at it with Paw:
  exec random_run <#events> '<title>' ['<filename>'] ['<histoname>']
  where #events is the number of events processed above. Specify a title for the top of the PS output, (for instance the date of the random trigger run) and the filename of the postscript output file (the default name is random.ps in the current directory). You can also specify the full file name (including full path) of the hbook file that you want to use as histoname (it defaults to $BFROOT/work/b/babardrc/drcoed.hbook).

  Another useful plot can be made for sector <sector> using:

  n/pl 2.row%column (1./<#events>)/0.6E-3.and.sector=<sector> option=colz
  Note: the weighting factor converts the random triggers into a noise level in kHz:
  0.6E-3 = 600E-7 (width of the window in which hits are accepted) * 1000 (Hz to kHz conversion)

  To draw the sector boundaries on top of the previous plot, use the command

  exec ~babardrc/kumac/drawSector.kumac
  right after the previous one. The borders should appear in red. If it doesn't work, type filecase keep and try again -- by default paw doesn't differenciate capital and small letters whereas unix does...

  Please remember to clean up the area when postscript files are no longer needed.

• You may want to keep a copy of the hbook file for future reference. The dedicated disk for that is
  /nfs/farm/babar/drc/babardrc/histo
  Please give the hbook file a name that identifies it for future use (like date and number of events).

Last global calibration could not be validated. (...) Current DIRC offline-online release XX.X.X doesn't match production offline YY.Y.Y

To find the location of a noisy channel:
Open paw with from your home directory:
paw
Then inside paw use the following commands:
h/file 1 /nfs/bbr-nfs102/pubfastmon/Monitoring/OutputArchive/2007/03/LiveFastMon/LiveFastMon-0071363-20070321-115817.hbook 0
where LiveFastMon-0071363-20070321-115817.hbook corresponds to the RUN you want use.
ldir
cd drcfm
h/l
which will list all the histograms
set ncol 50
palette 1
op logz
h/pl 45 colz
where 45 is the 2-D histogram correspoding to the appropiate sector. The noisy pmt will show as a hot bin in this histogram, maybe 100 or more times hotter than the average pmt.
exec ~babardrc/kumac/drawSector_FM.kumac
locate
which allows you to get the coordinates of the noisy channel when you click on the channel bin on the plot
q
to exit.

To mask the noisy channel:
There are two new scripts that make the jobs much easier:

to find the list of channels masked in the given key and

to mask an addition channel.

Things to know before following the instructions:

Terminology of the DIRC geometry and basics of the readout.

This includes the terms: ROM, sector, harness, channel, TDC, and DFB. See here for details.

Which channel is noisy (sector, row, column) ?

To get this information, do a standalone run and look at the sector plane:

n/pl 2.row%column sector.eq.# option=colz
locate

with # replaced by the number of the noisy sector. Please make sure that you get the row and column number in the format starting at 0. To check plot:

n/pl 2.row%column sector.eq.#.and.(row.ne.#.or.column.ne.#) option=colz

with the #'s replaced by the corresponding numbers. The noisy PMT should no longer be visible, but a white square should appear at its coordinates.

Is it electronics noise or a noisy PMT ?

If the neighboring PMTs see light from the noisy PMT, it is a X-mas tree and not a noisy tube. Otherwise, switch off the HV for the corresponding HV group and make another standalone run and check whether the PMT is still hot. In case it is a PMT related problem, try to lower the voltages in order to check whether you can save the PMT, otherwise, disconnect the HV cable from the PMT (see other texts for more information).

Gathering the information
Note that hexadecimal numbers are prepended with an 0x in this text. In the ASCII version of the xtc file that you will edit, all values are entered in hexadecimal format WITHOUT the leading 0x.

We now have the sector, row, and column, of the PMT. Next, we need the ROM, link, harness, channel, section, TDC, and TDC channel information.
The following table converts sector to ROM and link:

Sector	ROM	Link
0	0	A
1	0	B
2	1	A
3	1	B
4	2	A
5	2	B
6	3	A
7	3	B
8	4	A
9	4	B
10	5	A
11	5	B

In order to convert the row and column to harness and channel, run the program Select option 2 and enter the column and row numbers. The DFB number is the same as the harness number.

Get the section number using the following table:

Harness	Section if Link==A	Section if Link==B
0	13	27
1	12	26
2	11	25
3	10	24
4	9	23
5	8	22
6	7	21
7	6	20
8	5	19
9	4	18
10	3	17
11	2	16
12	1	15
13	0	14

We only need the section number modulo 14, i.e. note the number in the second column.

Get the TDC information by:
TDC number : channel / 16 (number from 0 to 3)
TDC channel : channel modulo 16 (number from 0 to 15)

Look for the current xtc file. You should ask the Configuration specialist (Vasia) about the current xtc files. You are likely to find those in:

Please note that you should never change or delete those files as they are part of the current configuration and if we loose those, DIRC will go non-runnable.

Find the correct xtc file for the ROM you want to change, it should be called Drc0#_DATE.xtc with # replaced by the ROM number (0-5).
Convert the file into a human readable format by running:

Open the file test.dat in your favorite editor. The format is line oriented. The columns have the following meanings:

Column	Description	Values
0	Group	0 : data taking
		1 : calibration
		2 : internal test
1	Address	0-7: ADC
		0x8-0xf: ANDer
		0x10 and up: TDC
2	Value	0-0xffff
3	Link A Mask	0-0xfffc
4	Link B Mask	0-0xfffc

More detail on all the fields, can be found here.

We are only interested in the Group 0. The channel will be masked in the ANDer (in theory we can also mask it in the TDC only but this will leave the ADC for this channel switched on). The Address values for masking are:

TDC number	Address
0	0x9
1	0xb
2	0xd
3	0xf

Look for the lines in the text that are Group 0 and the Address value you determined in the table above. A typical line (without any masked channels) will look as follows:

Please note that 0xfffc stands for the binary number %1111 1111 1111 1100, i.e. the upper 14 bits are set, the lower 2 not. Correspondingly, 0xffff stands for the binary number %1111 1111 1111 1111, i.e. all 16 bits are set.
Remember also that Link A and Link B stand for one full Front-End-Crate with 14 DFBs.

This line will cause the following configuration to be set:
In Link A all upper 14 bits are set. Therefore, the initialization is sent to all 14 DFBs. For Link B, all upper 14 bits are set to abd the initialization valid for those 14 DFBs, too. The Address 0xb tells us that the Value is set in the registers of the second TDC (number 1) on those boards.
The value itself is 0xffff, i.e. all 16 channels on that TDC are enabled.

Lets assume you want to switch off TDC channel 7 on the TDC number 2 on Link A section 5. This means that all TDC channels BUT TDC channel 7 have to be switched on for that TDC.
Lets assume the old line is:

Thus, the value will be 0xffff - 2^7, i.e. all bits set but the 8th: %1111 1111 0111 1111 = 0xff7f.
The Link B Mask should be 0, as none of the Link B DFB/TDCs should get this value. The Link A Mask should only contain the bit for the TDC on section 5. Remember that the bits are shifted by two, i.e. 2^(5+2) = 0x80.
The other TDCs should still be set to 0xffff, the mask for Link B is unchanged, the one for Link A is the old value minus the new Link A mask calculated in the last step.

Now you can replace the old line by the following new lines:

Before converting the file back into xtc format, make sure to do the following:

• Remove all none data lines at the header.
• Check that the last data line in the file has the full masks set in the Link A and Link B field as it is used as run mask, i.e. something like
  0 8 0 fffc fffc
  has to be in the last line.
• Add two lines to the end:
  FFFF
  and the file name of the xtc file. You should name the xtc file according to the rules as described above; use a new not yet used in order to not overwrite an old configuration !

Convert the ASCII file back to the new xtc file by running the command:

It is best to check that the translation was successfull by reconverting the new xtc file into an ASCII file and making a diff of the two files.

In order to make the changes offical, Vasia has to create a new DIRC key which has then to be included in a new BaBar key. Once data taking starts with this key, the masking is effective.

It is not recommended, but possible, to change the DIRC xtc file currently in use. In this case, the change will get used as soon as the shift leader goes to 'STANDBY' to allow a reconfiguration of the system. If you have to do this, make a backup copy of the original xtc file so that you can back out your changes in case of unexpected problems with the new key.

Please don't forget that the ROMs should not be rebooted 'just for fun'

1. Log in to a subnet 60 machine
   ssh bbr-farm02

2. Set some basics:
   setenv ODF_PLATFORM 127

3. Reboot the DIRC ROMs
   /nfs/bbr-srv02/dataflow/ir-2/drc/release/bin/SunOS58_sparc_WS6U1/rebootPartition -c 0x1c0000

Make sure that the main frame has a problem. In order to do that, do the following tests:

• Restart the CAEN HV crate
  Turn the key to the OFF position. Remove both cables from the CAEN serial line. Turn the key to the ON position. After about 10 seconds, you should see the colored LEDs blink and the LCD screen show the CAEN logo. If this comes up, there is no problem.

• Check the fuses
  Switch the power supply off; go to the back of the crate and disconnect the power cable. Wait untill the red light gets OFF. Use a screw driver to unscrew the black fuse covers. Check the fuses with a DMM (to measure its resistivity). Replace a broken fuse by a spare (upstairs in the Lab area, close to Ray's office). If Ray is around, you can ask him to help you.

• Try to remove load from the power supply
  Switch the power supply off; go to the back of the crate and disconnect the power cable. Wait untill the red light gets OFF. Pull 4 of the 8 boards out of the crate. Reconnect the power cable and try again to turn the crate ON.

If the crate needs to be replaced, go to the previous section of the manual and follow carefully the instructions.