Born on:March 12, 2000 

This page is intended as a record for important information, outstanding problems and progress that affects the DIRC run commissioner team and should help to ensure that all four DIRC commissioners stay up-to-date.
It should also help to correlate data quality issues with known problems - in fact, in many cases the daily log may be our only record of glitches, reboots and failures in DIRC hardware or software. Please keep a log of any reboots, power glitches and problems - even if nothing at all happened that day, during data taking please enter "nothing to report" for that day.

When you are on DIRC commissioner shift, please enter all relevant information into this file that is located at $BFROOT/www/Detector/DIRC/Operations/commissioner.html using your favorite editor.
Please do not use netscape or any composer as this typically screws up the whole file.


[ Problems  Schedule  Daily log   Links  Old_Schedules

Open problems:

Problem description: Approach to solve problem: Expert responsible:
Hardware and Online fixes (Nicolas, last update Mar 13th, 2006)
- - -
Software and other tasks (in random order, last update Dec 14, 2006, Joe)
Meantime: average is at 0.5nsec instead of zero. -
Study influence of deltaT resolution on DIRC fits. -
Background production processes; D/MC comparison, surface scattering, bar imperfections, lobes; use 90deg tracks. -

2008 schedule:

Week: Commissioner: Backup:
January 2 - January 9 Joe Jose
January 9 - January 16 Jose/Jennifer Joe
January 16 - January 23 Jennifer Joe
January 23 - January 30 Jennifer Joe
January 30 - February 8 Jennifer Joe
February 8 - February 13 Joe Jennifer
February 13 - February 20 Joe Jennifer
February 20 - February 27 Jose Jennifer
February 27 - March 5 Jose Jennifer
March 5 - March 12 Jennifer Jose
March 12 - March 19 Jennifer Jose
March 19 - March 26 Jennifer Jose
March 26 - April 2 Jennifer Jose
April 2 - April 9 Jose Jennifer
April 9 - April 16 Jose Jennifer

2008 Accounting
Owe:


Scheduled:
On-Call:

Back-Up:

Jose:

Foreseen out of SLAC periods:

Planned/pending/in progress activities:

Daily Logbook:

Jump to:
Year 2007 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Year 2006 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Year 2005 Jan Feb Mar Avr May June Jul Aug Sep Oct Nov Dec
Year 2004 Jan  Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Year 2003 Jan  Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Year 2002 Jan  Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Year 2001 Jan  Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Year 2000 Mar  Apr May Jun Jul Aug Sep Oct Nov Dec
Date Name Description
25 Augost 2008 JB The SOB has been emptied last week and is beeing dried. The Water plant was put in local circulation mode.
  • These are the changes made to the software:
    1) in $DAPPDIR/drc-mon/script/
    o comment out paging and e-mail commands in the |sob| and |water| scripts 2) in $DAPPDIR/drc-mon/alh/
    o edit DRC_GAS.alh
    disable (by replacing the second dash by a "D" in the mask at the end of the line) the following alarms:
    CHANNEL SOB_Water_System DRC::WTR:VSM:CH_1 -----
    CHANNEL SOB_Water_System DRC::WTR:VSM:CALC0 ---T-
    CHANNEL SOB_Water_System DRC::WTR:VSM:CH_5 -----
    CHANNEL SIAM_0 DRC:GAS:SIAM_0:TRP_0 ---T-
    CHANNEL SIAM_8 DRC:GAS:SIAM_8:TRP_0 ---T-
  • The DIRC oncall and backup pagers have been returned to SCS (Room 108). Also the oncall and ops. manager cell phones have been returned to SCS (Teri Church).
  • 3 July 2008 JB From the IR2 minutes today:
    DIRC (Matt): chiller has been emptied. The path to the electronics is valved off, as well as that to the backup chiller, where handles have been removed. The internals of the chiller have been blown dry.
    23 June 2008 JB Today Matt has turned off the degassing pump. Walt restarted the alarm handler.
    19 June 2008 JB (Yes its been a long time since we updated this logbook) Today I disabled the degassing pump automatic page and emails because Matt says he will be turning off the pump within the next week because it is not needed anymore. The file I modified was $DAPPDIR/drc-mon/script/degaspump . Nicolas did the following change to the file /afs/slac.stanford.edu/g/babar/repo/epics-drc-mon/alh/DRC_GAS.alh :
    > CHANNEL SOB_Water_System DRC::PUMP -D---
    'D' means disabled => no alarm on the alarm handler regardless of the pump condition. To have the change enabled, one just needs to restart the main ALH on con01 (pilot console).
  • Another thing that has been happening lately is that Matt reports that the incoming resistivity in the SOB water is lower than usual in the display by the water plant. It is ~16 instead of 18. However in epics and Ambient it looks fine. Maybe it is just a bad controller box.
  • 10 Apr 2008 JB All the cronjobs have been disabled except for DIRCLIVECronJob. This scripts are ~/CronJobs and one has edit ~/.trs/crontab to enable/disable them using the command "trscrontab -e".
    09 Apr 2008 JB
  • Today we turned off the Wienner crates. But not without trouble. When I pushed the On/Off button for the Wieener crates it was unresponsive. Rebooting drc-mon did not help. Finally we figured out that this was a side-effect of the code change that Matthias did last November.Nicolas undid this change (it was just one commented line) and after that we rebooted drc-mon again and this time most of the Wienner crates came back Off. We turned off the last 2 by hand. The file was: epics-drc-mon/db/feewiener.db and the which has un-commented is : field(PINI,"YES").
  • Then the chiller was turned off.
  • The ALH has been updated accordingly.
  • Also the key for the HV crates has been removed and hidden in the control room. It is in the keys box next to the computer alcove door with a label "DRC HV".
  • 08 Apr 2008 JB This is the end of data taking. The following activities have been done today:
    • A Global calibration was taken last night.
    • Cosmics were taken all last night.
    • Put the ALH in shut down mode:1)log into bbr-reflin 2)cd $DAPPDIR/drc-mon/alh 2)edit drc.alh : change DRC_HV.alh to DRC_HV-shutdown.alh , DRC_FEE.alh to DRC_FEE-shutdown.alh,
    • Took a long ADC calibration. The files are /a/bbr-srv02/calib/drc/hbook/calibration_adc_200804081230.hbook (.ps), hbook also copied into babardrc home directory.
    • Took random trigger run. The xtc file is at : /nfs/bbr-srv02/calib/drc/conditions/drc/tt1-004b2be5ce99252c-C.xtc also copied to /nfs/farm/babar/drc/babardrc/standalone_8April2008.xtc, here is the hbook file /nfs/farm/babar/drc/babardrc/OedNtuples/standalone_8Aril2008_50k.hbook
    • The HV crates have been turned off.
    The DIRC finishes in good shape. Only one Analog chip (DFB#7) and one TDC chip (DFB#11) broken (both in sector 8). And about 200 dead pmts. Also there are no noisy channels.
    07 Apr 2008 JB NTR
    06 Apr 2008 JB NTR
    05 Apr 2008 JB NTR
    04 Apr 2008 JB NTR
    03 Apr 2008 JB NTR
    02 Apr 2008 JB NTR
    01 Apr 2008 JB NTR
    31 Mar 2008 JB NTR
    30 Mar 2008 JB NTR
    29 Mar 2008 JB NTR
    28 Mar 2008 JB NTR
    27 Mar 2008 JB NTR
    26 Mar 2008 JB NTR
    25 Mar 2008 JB NTR
    24 Mar 2008 JP Reported at the 3:45 meeting that shifters have to try on the cell if no quick answer when paging ...
    23 Mar 2008 JP A page was sent to the on-call pager the at 8:00 but it never reached the page (happily the problem was not DIRC-related). Some tests with IR2 shown that the pager was not systematically working ...
    22 Mar 2008 JP NTR
    21 Mar 2008 JP Red alarms for scaler, sector #0. Jose moved up the threshold to stop these alarms bothering shifters.
    20 Mar 2008 JP NTR
    19 Mar 2008 JP NTR
    18 Mar 2008 JP NTR
    17 Mar 2008 JP A lot of backgrounds.
    16 Mar 2008 JP NTR
    15 Mar 2008 JP DQM shifter reported drops in some bins of the # hits per sector for sector #8 and #11. Things are back to normal in the next runs.
    14 Mar 2008 JP NTR
    13 Mar 2008 JP Made plots for currents in LST and DIRC scaler --> there is a correlation between LST backgrounds and our scaler alarms. We were right to leave the threshold to its current value.
    12 Mar 2008 JP Decision was made to leave the current threshold for scaler alarm to let PEP have something to tune against for the LST backgrounds.
    11 Mar 2008 JP Should we change the threshold of the alarms ? Yellow DIRC scaler alarms made silent by DAQ shifter ...
    10 Mar 2008 JP NTR
    9 Mar 2008 JP NTR
    8 Mar 2008 JP NTR
    7 Mar 2008 JP NTR
    6 Mar 2008 JB NTR
    5 Mar 2008 JB NTR
    4 Mar 2008 JB NTR
    3 Mar 2008 JB at 22:52 HV channel #14 in sector 9 got into an overcurrent state. It is normally at 1810uA and it jumped to 2500uA. I do not see a sign of a christmas tree in the fast-mon histograms. I lowered the voltage by 100 volts from 1199 to 1100 and the current dropped to < 1700uA, then I raised the voltage in steps all the way back to 1199 and it appears to be normal now.
    2 Mar 2008 JB NTR
    1 Mar 2008 JB NTR
    29 Feb 2008 JB NTR
    28 Feb 2008 JB calibration at 17:42 shows 16 dead channels in DFB#11 sector8. These channels correspond to a TDC chip#0. looking at the timing distribution of this chip in a cosmics run shows that the timing is discretized in steps of 17ns.==> This chips is broken, thats why it does not calibrate.
    27 Feb 2008 JB NTR
    26 Feb 2008 JB NTR
    25 Feb 2008 JB NTR
    24 Feb 2008 JB NTR
    23 Feb 2008 JB NTR
    22 Feb 2008 JB NTR
    21 Feb 2008 JB NTR
    20 Feb 2008 JS Got paged for a single slightly lower channel (DFB 4) in sector 8 occu. vs DFB. No problem - but not sure why DQM felt like ignoring instructions not to page...
    19 Feb 2008 JS A beam abort caused huge deadtimes in DIRC for two runs, data corruption caused fastmon to crash. Jim Hamilton investigated and during investigation problem fixed itself. The fix was rebooting the ROMs which was done by the pilot reconfiguring. For future reference we should ask the pilot to go to standby if large unexpected deadtimes happen. This will reconfigure and reboot the ROMs.
    18 Feb 2008 JS Got paged because of a bad trickle run (79643). Asked pilot to start new run, checked FastMon spew file, which showed pileup on the left of signal peak in line 1&8 as usual. Runs before and after are fine. Updated short-term DQM instructions to include guidance on trickle backgrounds again.
    17 Feb 2008 JS NTR
    16 Feb 2008 JS NTR
    15 Feb 2008 JS NTR
    14 Feb 2008 JS NTR
    13 Feb 2008 JS Got paged just after midnight. HV group 11 in sector 5 is much lower than the reference. Had been going on for some 7 hours but was difficult to see for commissioners and DQM alike. Put a note into the short-term to explain that this is not a problem. The channel returns to normal around 1:30am. During the day update the short-term to say that the DIRC expert should not get paged if 1 or two HV groups are much below reference or zero.
    12 Feb 2008 JS NTR
    11 Feb 2008 JS NTR
    10 Feb 2008 JS Around 4am the occupancy in sector 0 returned to normal = high in DFB 8.
    9 Feb 2008 JS Occupancy vs. DFB for DFB 8 in sector 0 dropped below reference following run 79340. It was too high (one of those containing HV group 20). Updated the short-term instructions to indicate that this is no problem. We may need to update reference if persists.
    8 Feb 2008 JS NTR
    6 Feb 2008 JP NTR
    5 Feb 2008 JP Joe discovered that the timing distribution for HV group #20, sector #0, contains a lot of background.
    The 8 channels in the problematic location in sector 8 are now dead.
    4 Feb 2008 JP NTR
    3rd Feb 2008 JP Last night's calibration not fully satisfying for sector #8, next one was better.
    Joe reports some runs with very high FX=9 error rate again ...
    2nd Feb 2008 JP NTR
    1st Feb 2008 JP NTR
    31 Jan 2008 JS/JP
  • The analog chip in sector 8 is starting to die again. DFB #7, channels 0-8 are fading away quickly since this morning. Probably same problem as last year when several DFBs were eaten alive in this slot.
  • Wiener crat alarm seen at 6:25 yesterday evening for sector #2, DFB #6, -5.2V : seems to be just a glitch.
  • Alarm in power chiller at 11:38 AM yesterday due to something being rebooted ...
  • 30 Jan 2008 JP Investigation showed that the noise in the fast mon plots for run #78909 and #78910 was due to 3 channels. Files saved in :
  • /nfs/farm/babar/drc/babardrc/OedNtuples/run78909_400.root
    (this one doesn't show anything even if it is bad because we picked up the first 400 events)
  • /nfs/farm/babar/drc/babardrc/OedNtuples/run78910_2000.root
    (shows clearly the problem)
    Noisy channels are : sector #8, DFB #7, channel #2 to 4. (2 is at 2200, 3 is at 1200 and 4 is at 700 whereas a normal channel is at less than 60 ...)
  • 29 Jan 2008 JP "Noisy channel" seen in fast mon plots for run #78909 and 78910, to be investigated.
    28 Jan 2008 JB Today me and Chris O'Grady installed a change the code of the DIRC calibrations to hopefully prevent bad calibrations like the one from last week from being written into the DB. The file we modified is DrcOdf/DrcT0End.cc. The code was installed in the morning and this required rebooting the ROMs at 7pm during a beam loss.
    27 Jan 2008 JP NTR
    26 Jan 2008 JP NTR
    25 Jan 2008 JB This morning we found out that the rolling calibrations were failing because the DRC T0 calibrations were bad. Later Nicolas found that the problem started on Wednesday (the 23rd) morning. This was correlated with the LED calibration taken at 4am that morning. After looking at the calibration history it was found that the calibrations started to fail on the weekend. Talkng with Chris O'Grady we found that the failed calibrations were correlated with the DRC being assigned to Partition Master #1, at 3pm the beams were lost and we took this oportunity to recalibrate the DIRC. This calibration was bad (but it was validated and written to the DB just like the ones from earlier in the week). Then Chris replaced the suspicious module. We took two more calibrations and they came out fine. The 3 calibrations are
    calibration_tadc_20080125_152052.hbook
    calibration_tadc_20080125_153400.hbook
    calibration_tadc_20080125_154108.hbook
    22 Jan 2008 JP NTR
    21 Jan 2008 JP NTR
    20 Jan 2008 JP NTR
    19 Jan 2008 JP
  • Joe and me investigated the bin6 FX errors. The errors are coming from ROM#4 only, so it is probably due to one DFB in sector #8 or #9.
  • Nothing else to report.
  • 18 Jan 2008 JP
  • HV trip (automatic page worked) at 8:00 AM : sector 2, channel 3 (the same than the previous day). Trip can be seen in run #78437 and #78444. Reset to nominal. Threshold should be changed .... This has been done. The threshold was moved from 2295 uA to 2345 uA.
  • Wiener crates minor alarms occured yesterday evening at 7:40 PM, but were just glitches.
  • We raised the threshold if channel #3, sector #2 by 50uA. This channel was normally at about 2190uA and the threshold was at 2295uA. The splashes seen by the other channels are spikes of about 60uA ...
  • 17 Jan 2008 JB Here is a summary of the problem with the timing distributions on the fast mon plots:
  •   This problem started on Sunday 13, with run 78206
  •   run 78259 had the distribution very bad.
  •   no trickle injection for both beams - run 78313
  •   trickle injection for only the LER - run 78315
  •   trickle injection for only the HER - run 78319
  • Clearly the problem is due to bad trickle injection in the LER. The background is only showing up on two trigger lines: #1 and #8. Trigger people say these are pure DCT triggers. Chris O'Grady has made some tests with the master crates to make sure that there is no problem with dataflow. Matt Weaver has shown that the bad timing distributions are coming from pulses with less than 50 revolutions.
    Also today I un-inhibited SIAM0 channel0 (the channel monitoring) the chiller on/off. Since the new flow sensor was installed on Jan9 this channel has been been very stable at OK. So the new sensor is working fine.
  • 16 Jan 2008 JB NTR
    15 Jan 2008 JP
  • Global calibration checked at 8:20 PM : looks fine.
  • Dirc tour at 8:10 PM : NRT
  • Fast monitoring plots checked --> trigger lines #1 and #8 show some asymmetry, peaking time problem still present.
  • HV trip on Sector2 Channel#3 at 6:30 am. This was due to high backgrounds.
  • 12 Jan 2008 JP
  • Global calibration checked at 10:30 AM ; looks fine.
  • Fast monitoring plots checked ; NTR.
  • Made the "DIRC Tour" at 11:00 AM ; NTR.
  • Circulating pump status : current value 0.836 (11:30 AM). See plots in : /afs/slac.stanford.edu/u/ec/babardrc/CirculationPumpAndSOBLevel_12jan2008_10AM.gif and /afs/slac.stanford.edu/u/ec/babardrc/CirculationPumpAndSOBLevel_12jan2008_12AM.gif --> value seems to be increasing ...
  • 11 Jan 2008 JP/JB NTR
    10 Jan 2008 JP
  • 12:00 PM : Noisy channel in sector #0 came back in run 78089 and 78090 (it had disappeared for the previous ones). Jose updates to code to disactivate it. Coordinates : 3rd row, 6th column --> DFB #5, channel #2
  • 9 Jan 2008 JB Activities for the ROD:
  • At 7:30am we switched to the BCS to allow HVAC people to install a new pressure sensor on on the water lines. They arrived close to 9am and were done installing it by 9:30am. The chiller SIAM was inhibited.
  • After the sensor was installed the opened the HV panel under the chiller hood and Matt and Jerry installed a toroid on a power line going to the pump.
  • At 10am we switched back from the BCS. Everthing went smooth. We verified that the pressure sensor gave open circuit when the chiller was off and a closed circuit when the chiller turned on. There are several terminals on this sensor, we will use the two left most. Also the toroid was verified to be giving ~3V AC, this will need to be connected with a long cable to a box by the water plant if it needs to be used.
  • At 10:30am Jerry, Matt and me made a controlled access to the top of the detector to look at the Nitrogen flow meter electronics. We could not reproduce the problem so we just cleaned the board as well as we could.
  • Michael was sick yesterday and is still not in today so we dont know if he has the parts for the new connector. Me and Matt will disconnect the two wires that are currently going to the power relay in the chiller and will connect them to the new pressure sensor. We will disconnect them from the connector on the back of the chiller. !We will finally have a good monitoring of the chiller power today!
  • at 2:30pm we finished connecting the new presure sensor. The correspoding channel on the SIAM reads OK. We will watch it for a few days before we enable this channel. Channel 1 (the flow monitoring) has been enabled now.
  • noisy channel in sector 0 to be determined from the following file: /nfs/farm/babar/drc/babardrc/OedNtuples/run78032_1k.hbook
  • 8 Jan 2008 JS NITR. Announced planned work for tomorrow's ROF in ops meeting: chiller and N2 flow. ROD to start at 6am, 4 shifts.
    7 Jan 2008 JS Persis announced that the BaBar run will be terminated early in March as soon as we reach 30fb-1 at the Y(3S).
    JBWith Matt we tested the connector at the back of the Chiller to see if it will glitch if we cool it. We put ice bags on it and waited about 2 hours. We also took the thermocouple from the reservoir and put on the connector, the temperature was about 9 degrees. No glitches seen, except for one when I inserted the thermocouple.
    6 Jan 2008 JS Got an e-mail from the pilot reporting very short silent alarms from DRC:BKG:ALARM:CURRENT_EXCESS in CMLog. Updated the short-term instructions to explain alarm and that it is just a sign of high backgrounds. DAQ reports via e-mail that there is a new noisy channel in sector 0, DFB #5, HV group #3. It's spurious, only a couple of runs per day show the noise peak. Will keep an eye on it, may be electronics noise requiring masking the channel.
    5 Jan 2008 JS First page of the year at 2:15am. DAQ saw few 10 entries in FX status bin 6 in run 77907, paged as instructed. Those events correlated with very high background. Did not see FX=6 events previously and not afterwards. Decided that such events are acceptable at low level, some 10-100 per run, less than 10^-3, and updated the short-term instructions accordingly.
    Feedback from cpo: the FX=6 bit means that the 4 TDC chips within one DFB do not all have the same coarse timestamp for a particular event. cpo thinks that it is probably due to data truncation in the event, either a TDC buffer filling up with too many hits of a FIFO filling up in this DFB. It's conceivable that one of the data truncations could cause the timestamp for one of them to get dropped. We don't know for sure which truncation occurred because the FX only reports the most severe error, FX=6, and not if also an FX=9 or FX=11 happened.
    4 Jan 2008 JS Power dip at 6:45 did not trip any of our crates, thus saving me from my first page of the year. N2 fluctuations are back. Will have to go back in again next week to see if we can find the source of the problem. The alarms have been disabled. Mike thinks that the chiller alarms are due to bad connector. Plans to replace it next week.
    3 Jan 2008 JS NITR, just updated some operations web pages
    2 Jan 2008 JB Updated JAS References, Updated Jas instructions, updated Ir2 contact database, Updated CareAndFeeding Contact page.
    1 Jan 2008 JB Happy New Year.
    31 Dec 2007 JB NTR
    30 Dec 2007 JB NTR
    29 Dec 2007 JB NTR
    28 Dec 2007 JB NTR
    27 Dec 2007 JB created hbook file for investigating the FEX Errors:/nfs/farm/babar/drc/babardrc/OedNtuples/run77606_9k.hbook. Turned out that it is DFB#9 in sector six which is causing the high number of FEX bin11 errors. It is very weird however because it is channels 8 to 21 but not every channel, it is every other channel.
    26 Dec 2007 RA NTR
    25 Dec 2007 RA NTR
    24 Dec 2007 RA NTR
    23 Dec 2007 RA NTR
    22 Dec 2007 RA Chiller alarms. I moved the short-circuit switch to the 'ON' position, and instructed the pilot to send us email and check on the chiller in case of further alarms. Further, a page during the night did not reach me; yesterday some pages got through and some didn't. I updated the short-term instructions; pilots should now call the cellphone and then the backup pager if the primary pager isn't responded to. (1510) The alarms are still occurring. The pilot asked permission to modify the ALH mask, and I gave it.
  • (Jose) I came to IR2 at 4:30 pm to look at the chiller. At 4:54pm I removed the on/off switch which is on the cables on the back of the chiller and I joined together the ends to make absolutly sure that the circuit was shorted. I still see the glitches going in the CMlog. This would mean that in fact the glitches are not coming from the chiller and are probably due the cable that goes to the siam. Last friday the 15th we probably did not see any glitches because the ambient temperature was warm.
  • 21 Dec 2007 RA NTR
    20 Dec 2007 RA Nicolas made a new webpage for monitoring the number of dead PMTs.
    19 Dec 2007 RA NTR
    16 Dec 2007 JB Last night we did not get any glitches.
    15 Dec 2007 JB Lat night we did not see any glitches. This means that the glitches are coming from inside the chiller. At 10am we have separated the wires to the siam channel 0 to continue to monitor the chiller. Also at 9am the N2 flow instabilities came back. The alarms are masked.
    14 Dec 2007 JB Lat night we still saw some glitches, but only a few not nearly as many as in the previous nights. At 10am we have shorted the wires and they will remain shorted for tonight to check if the glitches are coming from inside the chiller or outside. Another thing we did was to make an access to the top of the detector with Jerry to investigate the N2 flow instabilities. Jerry and Popat went to the top while I stayed in the control rooom monitoring the instabilities. The instabilities went away when Jerry removed a plastic cover on the back of the electronics rack.
    13 Dec 2007 JB/RA At 8am today we switched to the back-up chiller. After one hour Bob from HVAC had replaced the relay. We switched back from the BCS. The channel1 is still inhibited. Channel 2 which monitors the BCS is also inhibited. Only channel 1 (water flow) is un-inhibited. Another thing they installed in the chiller is a spare pair of wires connected to the end of the wires which go to the channel 0.These wires stick out of the back of the Chiller and can be shorted to verify that the glitches are coming from inside the chiller.
    12 Dec 2007 JB/RA We tested the siam0_channel1 which monitors the chiller water flow to see if it would trip when we turn off the chiller. We found that when we turn off the chiller and even when Matt pulled down the power handle on the wall to make sure that there was no residual power somewhere on the chiller this channel was still OK. It never tripped. Therefore it does not protect us if the chiller should be turned off. We called HVAC and tomorrow morning they will replace the relay which monitors the powerline inside the chiller.
    Did a dry run of changing over to the backup chiller. Tomorrow when we can get an HVAC guy, we will do the real thing so the HVAC can work on the main chiller.
    11 Dec 2007 JB/RA Matt and Jose looked at the chiller; fuller report to follow. (Midnight) Pilot paged me; persistent alarms in the chiller. Since the chiller was actually working, told him to disable the alarm and set up a strip chart of the Sector 3 WCR temperature, and page me again if it should rise.
    Matt,Jerry and Steve Dardin made a control access to the top of the detector to investigate the N2 flakiness. Fifteen minutes after they went in the flow values became stable again. They dont know what exactly fixed the problem, they just tested some voltages, so they think that a board which holds the GMG board could be going bad. Or Perhaps a loose connection? We will stay like this unless the flakiness comes back in the meantime they will try to find a spare board.
    about the chiller: Matt reads in the chiller manual that the chiller should not be operated under ambient temperatures below 13 degrees. Now we think this might the cause of the glitches. Also, togther with HVAC we tested that siam0channel1 monitors the water flow of the chiller. We inhibited the siam first, then disconnected the flow sensor. After about 1.5min the channel1 tripped in the EH. Matt still wants to do another test: to turn off the Chiller to simulate a no power situation and see if that will also trip channel1.
    10 Dec 2007 RA 1200 : Pilot called, worried about intermittent red alarms from the nitrogen flow. 1325 : All the nitrogen flows went red simultaneously, then came back to green after a short while (~two minutes?). Told the pilot to page Matt; going down myself to look at things. (1425) Today's huge glitch was apparently caused by some cryo work, which also tripped the DCH gas system; Karl used the phrase 'opened the wrong valve'. The more general problem of increased fluctuation in the system is still under investigation; Matt has said he will look for leaks and other changes to the lines tomorrow. (1700) Livio paged me, looks like another fluctuation in the flow. I asked him to page Matt, but in any case the fluctuation settled out quite rapidly, so I asked him to ignore these alarms unless they go on for some time. (1725) Jose reports that he and Matt think the problem may be in the electronics; he is trying to get hold of Jerry.
    09 Dec 2007 RA NTR
    08 Dec 2007 RA NTR
    07 Dec 2007 RA IR2 reports yellow alarms for N2 flow in sectors 0 and 10. Matt says that these flows have been more shaky than usual since last night, and he will investigate. Until we know more, we have changed the alarm thresholds.
    Jose moved the limits from 185 to 170 for sectors 0 and 10.
    06 Dec 2007 RA NTR
    05 Dec 2007 RA NTR
    04 Dec 2007 RA NTR
    03 Dec 2007 RA NTR
    28 Nov 2007 LE NTR
    27 Nov 2007 LE NTR
    26 Nov 2007 JB Matt lowered the Nitrogen flow for bar boxes 5,6, and 7 to the values before the water leak (~100).
    Also the backward west door was opened today for EMC work, calibrations show everything remained fine with the DIRC.
    25 Nov 2007 LE NTR
    24 Nov 2007 LE NTR
    23 Nov 2007 LE NTR
    22 Nov 2007 LE NTR
    21 Nov 2007 JB/LE
    • took a standalone: /nfs/bbr-srv02/calib/drc/conditions/drc/tt1-0048a25f5fc7e102-C.xtc. The hbook file is : /nfs/farm/babar/drc/babardrc/OedNtuples/standalone_run_211107_10k.hbook. We found the following noisy channels (row and column numbers starting at 1 by convention):
      Sector row column Z (count number ?)
      0 14 9 132
      0 21 10 57
      0 3 7 44
      1 19 8 189
      1 35 12 179
      2 30 12 39
      3 40 29 40
      5 34 16 45
      5 31 4 34
      7 32 23 32
      10 16 15 116
      10 16 16 56
      10 17 16 79
      11 9 2 120
    20 Nov 2007 LE/JB We have modified the HV values for all channels. Things that were done today starting 1pm:
    • Took one calibration before.
    • Taken COSMICS_L1PASSTHRU run #76740
    • The new files that were loaded into the boards are:
        /nfs/bbr-srv01/u1/babar/boot/apps/drc-hv/burt/nominalITrip_20071120_final.snap
        /nfs/bbr-srv01/u1/babar/boot/apps/drc-hv/burt/nominalV0_20071120_final.snap
        /nfs/bbr-srv01/u1/babar/boot/apps/drc-hv/burt/nominalV1_20071120_final.snap
    • Took three calibrations afterwards.
    • Took cosmics afterwards run #76749
    • We had some curious things: 1) Sector 1 scaler was spiking to 2K counts. 2) two channels tripped after changing the HV V0 values: sector5-38 and sector5-45, this was fixed somehow by reentering the trip threshold by hand into epics.
    19 Nov 2007 LE NTR
    18 Nov 2007 LE NTR
    17 Nov 2007 LE NTR
    16 Nov 2007 LE NTR
    15 Nov 2007 LE A calibration was taken at 10:17 am. It looks fine
    14 Nov 2007 LE A calibration was taken at 14:47. It looks fine. Matthias commented out a line on the drc-mon code which was different of the BABAR standard code. The drc-mon no longer turns off the Wiener crates when it is rebooted.
    13 Nov 2007 LE NTR
    12 Nov 2007 LE
    • Took one calibration at 8am, which looks fine. Then turned down HV because the tunnel plug was going in.
    • Turned HV back on around 2:30pm, then took another calibration on which the 8 first channel of DFB 1 in sector 1 appears to be dead.
    • Took another calibration just after that which looks fine. Jose told me not to worry about it, which will probably make me sleep better.
    11 Nov 2007 LE NTR
    10 Nov 2007 LE NTR
    9 Nov 2007 LE NTR
    8 Nov 2007 LE NTR
    7 Nov 2007 LE NTR
    6 Nov 2007 LE Took a calibration at 14:42 which looks fine.
    5 Nov 2007 LE Cosmics runs were taken with EMC as trigger (runs # 76637, 76638, 76646, 76648, 76649, 76650, 76664, 76665, 76666)
    4 Nov 2007 LE NTR
    3 Nov 2007 LE NTR
    2 Nov 2007 LE Took a calibration at 2:30 pm which looks fine.
    1 Nov 2007 LE Took a calibration at 11:30 am which looks fine.
    31 Oct 2007 LE NTR
    30 Oct 2007 JB Calibrated the DIRC before the doors closed at 9am.
    29 Oct 2007 JB
    • Calibrated the DIRC at 9:30am and everything looks fine.
    • Tested the problem with sector 5 during calibrations at ~10:30am, We swapped the module which feeds the light to sector 5 with sector 4 and the problem moved from sector 5 to 4. First we swapped the fiber in the front took calibrations and then swapped the cable on the back and took calibrations, finally we returned everything to normal. Seems it is related to the LED module.
    28 Oct 2007 LE
  • Turned on the HV on EPICS, took 2 calibrations in a row around 1:30 pm. They look fine. Turned down the HV.
  • Turned on the HV on EPICS, took 2 calibrations in a row around 6 pm. They look fine. Turned down the HV.
  • 27 Oct 2007 LE
  • Turned on the HV on EPICS, took 2 calibrations in a row around 11 am. They look fine. Turned down the HV.
  • Turned on the HV on EPICS, took 2 calibrations in a row around 6 pm. They look fine. Turned down the HV.
  • 26 Oct 2007 LE/JB
  • Nicolas has unmasked 2 DFBs, one in sector 6 slot 11 in sector 0 slot 6, which were giving problems with the monitoring of the voltage line.
  • So as to investigate the problems in Sector 6 slot 13, we swap the 2 boards in slot 12 and 13 to see if the problem would follow this swap and it didn't. It turned out later on, that we kept replacing the wrong DFB, and that this was the DFB in slot 11 and not 13 which had 8 dead channels. After realizing that, we change the right DFB Sector 6 slot 11 (6E --> A2), which fixed the problem.
  • Cosmics CS Run 76592 was taken.
  • 25 Oct 2007 LE/JB
  • we took three calibrations which look fine.
  • DFB : Sector 3, slot 6 (64 --> 3F) 16 channels were dead
  • DFB : Sector 6 slot 13 (A2 --> 69) 8 dead channels. Two hours later, 69 had the same dead channels as A2, so we replaced it (69 --> 6B). The same 8 dead channels showep up again immediatly on 6B. This is currently under investigation.
  • Sector 8 Christmas tree found and disconnected (row 33, column 5)
    We tested DFBs with voltage line monitoring problems, which LAL engineers had dome some component replacement on. We put them in the detector, and see if the voltage line monitoring problems were still showing up.
  • DFBs 67, 6B and 3F are now OK
  • DFBs 66 and 68 still have problems but only on the +12V line. According to the LAL enginneers, there is nothing to worry about, since the +12V is common to the whole crate. So 1 board showing a bad +12 line means that there is only some problem with the monitoring.
  • 24 Oct 2007 LE We replaced a bunch of DFBs and DCCs thank to the hard work of the LAL engineers at SLAC.
  • DFB Sector 0, slot 0 (4F --> 87) 8 channels were dead
  • DFB Sector 0, slot 6 (67 --> 4F) voltage line monitoring problem
  • DFB Sector 1, slot 1 (11 --> 4E) 8 channels were dead
  • DFB Sector 1, slot 13 (7F --> 36) 8 channels were dead
  • DFB Sector 6, slot 6 (73 --> 88) FiFo Full errors
  • DFB Sector 6, slot 11 (6B --> 6E) voltage line monitoring problems
  • DFB Sector 8, slot 0 (80 --> 80) 8 channels were dead
  • DFB Sector 8, slot 7 (8D --> 70) 8 channels were dead
  • DCC Sector 6 (14 --> 9) replaced for testing, 14 has been tested OK, now in spare
  • DCC sector 5 (26 --> 12) replaced for testing, 26 has been tested OK, now in spare
  • 23 Oct 2007 LE NTR
    22 Oct 2007 LE/JB/JS We replaced a HV board although nothing was wrong about it as it was just to train Joe and to get a little more training on this subject for Jose and I (sector 3 channel 5 which is the same we replaced on the the 10/18. Board 342 is going in and 344 is going out). Everything is running fine, so we will let 342 in the HV crate and 344 will be converted into a spare. Note that after we put the board in, we couldn't turn the HV on. Rebooting the IOC solved this problem though.
    21 Oct 2007 LE/JB NTR
    20 Oct 2007 LE/JB NTR
    19 Oct 2007 LE/JB Today we replaced a DFB in sector 3 slot 6 (DFB #64 going in, #36 going out) because it had its second chip broken. Jose disconnected a PMT in sector 5 causing a Christmas tree (row 12 column 3). He checked that ramping up the HV channel for this PMT don't make it trip anymore.
    18 Oct 2007 LE/JB The backward doors were opened on the morning. Matt has put the plastic covers on the DIRC. Jose has disconnected a PMT causing Christmas tree in sector 11 (row 12 column 2). So as to investigate another christmas tree in sector 5, he tried to ramp up the channel for this PMT (channel 6, board 5, HV crate 3) but couldn't. Actually all the channels in this board couldn't be ramped up. Together with Nicolas, we replaced the HV board (#388 replaced by #344) and everything seems to be working fine now. Then, the christmas tree was investigated again, and it turns out that the responsible PMT is in row 12 column 3. We will disconnect it tomorrow.
    17 Oct 2007 LE NTR
    16 Oct 2007 LE Matt has put a temperature probe in the chiller and we now monitor the DIRC chiller water temperature, which can be accessed in Ambient.
    15 Oct 2007 LE NTR
    14 Oct 2007 LE NTR
    12 Oct 2007 LE The problem we had with the automatic mail of the chiller alarms is now fixed, that is to say we now see chiller alarms in the automatic mails, if there are any.
    11 Oct 2007 LE The cron jobs seemed to stop failing, although something might have to be changed to make the unix-admin team happy (apparently our cron jobs are using too much ressource). Anyway, this problem is currently taken care of by unix specialists.
    10 Oct 2007 LE Due to a change in the ssh configuration in IR2, a lot of our cron jobs failed, which caused us to get automatic mails every 5 minutes. This is under investigation by Steffen Luitz.
    9 Oct 2007 LE The pilot ALH was rebooted in IR2 in order to take into account the chiller power alarms.
    8 Oct 2007 LE NTR
    7 Oct 2007 LE NTR
    6 Oct 2007 LE NTR
    5 Oct 2007 LE Although the chiller siam has glitches on a day to day basis, the automatic mail of the chiller alarms remains blank. Nicolas looked at that and it appeared that the shutdown configuration file didn't include this alarm. Now it does but we may have to reboot the pilot alarm handler in IR2, so that this configuration change is taken in account.
    4 Oct 2007 LE
  • The backup chiller siam tripped around 12am, but it was inhibited as usual.
  • We wanted to reset this siam since we noticed that it was in the OK state. While doing this, the button trp-tst of the siam #2 was hit by mistake, which caused the Wiener crates to be turned off. We acknowledgded that , and turned the Wiener crates back on.
  • Moving the water siam in another slot did not affect the glitches of the chiller.
  • 3 Oct 2007 JB/LE Today the power to all of IR2 was turned off.
    • At 7 am Matt disabled the SOB dump valve. And later turned off the water plant.
    • at 7:30am we turned off the Wienner crates. Turned off the Chiller, and then the DRC-MON and DRC-HV crates in the EH.
    • At ~11:30am the power came back except in the EH. Matt turned on the water plant.
    • Matt replaced the degassing pumps,but after about 1hr of being on we got an alarm from the sensor which checks if it is on. The sensor is reading a value of ~50 and should be below 15. Matt has switched to the back-up pump and that is working fine. He will investigate.
    • At 2pm there was power in the EH, so turned back on DRC-MON and DRC-HV. And the Chiller.
    • We turned on the Wieener crates and all DFBs are OK.
    • We turned on the HV crates to get the monitoring back, then we turned them off. When we turned them on we noticed that some channels were ramped up (like the other day).
    • Matt has enabled the Dump valve at ~4pm.
    2 Oct 2007 JB NTR
    1 Oct 2007 JB NTR
    30 Sep 2007 JB NTR
    29 Sep 2007 JB NTR
    28 Sep 2007 JB NTR
    27 Sep 2007 JB NTR
    26 Sep 2007 JB NTR
    25 Sep 2007 JB Jochen modified the online code to try to fix the time vs XTC extend plots in the FM plots. He set the variable to 100 instead of 0 in the code but we dont know what units they are. We took a random trigger run: /nfs/bbr-srv02/calib/drc/conditions/drc/tt1-00479778289e350a-C.xtc . But I dont know how to run the FM plots code.
    24 Sep 2007 JB Today we tested the HV monitoring by turning on the HV crates. It works fine. The only problem we found was that 2 Boards (#3 channels 51->55 and #7 channels 38,39) in sector 2 were ON after turning on the crate. I also masked out the faulty DFB in sector 8 by changing the threshold.
    23 Sep 2007 JB There was a chilled water failure in the EH this evening and the some racks tripped. One of them is the rack that holds drc-mon and the chiller siam.
    • The chiller, water plant and gas system are running fine.
    • The rack which holds the drc-mon and drc-hv did not have power so together with Matt we resett the tripped panel and got power back.First he disabled the dump valve.
    • With monitoring back we found out that the FEE crates were off. So we turned them back on. The FNS (finisar) is red because the roms are off (I believe). Another minor thing that is the is a voltage line on DFB in slot#7 sector 8, but thats probably not worrisome because this is the same slot that kills DFBs.
    • Another thing I could not figure out was that we did not get the HV monitoring back. I tried 1) rebooting the IOC 2) power cycling drc-hv 3)turning on one HV crate and then rebooting the IOC again. But Still the HV monitoring is white. This is not worrisome because the HV crates will stay off. Hopefully tomorrow when everything is back on this monitoring will come back.
    • One last thing that was out of order after this Power trip is on N2 gas flow, sector 6 flow was at 198 and the red alarm was at 190, so we just changed the alarm threshold to 200.
    • I inhibited the BCS siam channel.
    22 Sep 2007 JB
    21 Sep 2007 JB
    20 Sep 2007 JB
    19 Sep 2007 JB
    18 Sep 2007 JB
    17 Sep 2007 JB
    16 Sep 2007 JB
    15 Sep 2007 JB
    14 Sep 2007 JB
    13 Sep 2007 JB
    12 Sep 2007 JB
    11 Sep 2007 RA Matt installed the plastic shields.
    10 Sep 2007 RA NTR
    9 Sep 2007 RA NTR
    8 Sep 2007 RA NTR
    7 Sep 2007 RA NTR
    6 Sep 2007 RA NTR
    5 Sep 2007 RA/JB
    • Found the Xmas tree in sector 11; it is at (2,12) indexed from 1. That is, it appears in row 12, column 2 from the bottom and left. There is a screenshot of the histogram used to reach this conclusion; find it in the babardrc home directory, named sec11_sep5_2007.png.
    • Matt disabled the dump valve and is working on the water system, replacing the UV lamp.
    • Jochen says that our problems with the standalone runs "smell like hardware"; he has emailed the experts. Until they respond we are stuck for this part. We will try the ADC calibration - no, same problem. Trying to turn on the HV and see if that helps. The TDC calibration at least works. Trying the random-trigger run again. Well, it runs but stops at 1.4K with a segfault. Email Chris O'Grady, wait for his response.
    • Finally with the help from Matt Weaver and Jochen we have found out that the reason the code breaks is because the christmas tree is giving off too much light and making very large events which the software cant handle. So what I did is turn off the rest of the DIRC and only turn on Sector5. And now we are able to take the standalone run the xtc file is: /nfs/bbr-srv02/calib/drc/conditions/drc/tt1-004739e7aa482c9a-C.xtc and the resulting hbook file is:/nfs/farm/babar/drc/babardrc/OedNtuples/StanAlone20070905_600Events.hbook . However I was not able to determine which pmt it is because we cannot fully ramp up the HV group. It trips at 950V and therefore the good pmts in the group show zero occupancy.
    • I have also now taken the ADC calibration with histograms and large statistics for the French people so they can decide if the voltages need to be raised. The file is /nfs/bbr-srv02/calib/drc/hbook/calibration_adc_200709051729.hbook , the plots file is in the drc/tmp/ directory witht the same name.
    4 Sep 2007 RA Took standalone run in /nfs/bbr-srv02/calib/drc/conditions/drc/tt1-004735df0b39a2a2-C.xtc. Corresponding hbook file in /nfs/farm/babar/drc/babardrc/histo/testSep4_2007_1.hbook. Changed V0 of channel 6 in sector 5 to 290, took new run in /nfs/bbr-srv02/calib/drc/conditions/drc/tt1-0047362cc9dac10e-C.xtc.Lowered voltages of 11, 13 and 14 to V1 level, so they won't trip. In case of channel 11, it is the scaler, so V1=V0; we lowered it to V0-300, which is 1018. So, when we put it back, we should put it back to 1318.
    3 Sep 2007 RA NTR
    2 Sep 2007 RA NTR
    1 Sep 2007 RA NTR
    31 Aug 2007 RA NTR
    30 Aug 2007 RA NTR
    29 Aug 2007 RA NTR
    28 Aug 2007 RA NTR
    27 Aug 2007 RA NTR
    26 Aug 2007 RA NTR
    25 Aug 2007 RA NTR
    24 Aug 2007 RA NTR
    23 Aug 2007 RA Matt adjusted the water level in the SOB tank.
    22 Aug 2007 RA Un-inhibited (exhibited? hibited?) the chiller SIAM, since it seems to have been behaving nicely. Will inhibit it again if the glitches come back.
    21 Aug 2007 RA NTR
    20 Aug 2007 RA NTR
    19 Aug 2007 RA NTR
    18 Aug 2007 RA Paged around 0030 for SIAM glitches, much as expected. Instructed pilot to run script to inhibit SIAM, and keep an eye on chiller. Later: Jose had the idea of asking the pilot to look at FEE temperatures in a strip chart, so I requested that.
    17 Aug 2007 RA Updated pilot short-term instructions to deal with SIAM glitches. Later: Inhibited the SIAM.
    16 Aug 2007 RA SIAM chiller glitches are back.
    15 Aug 2007 JS Matt disabled the SOB dump valve for the UPS battery replacement.
    14 Aug 2007 JS Power chiller glitches are back. we had 4 at about 10pm. they cleared after some seconds.
    13 Aug 2007 JS/JB Trying to investigate the X-mas trees: sector 5 HV group 6, sector 11 HV group 5 or 10. We have to take a standalone run to find out exactly what are the channels.
    12 Aug 2007 JS NTR
    11 Aug 2007 JS NTR
    10 Aug 2007 JS NTR
    09 Aug 2007 JS NTR
    08 Aug 2007 JS Get paged because of a noisy channel in sector 6. It disappeared after some runs.
    07 Aug 2007 JS Get paged in the morning due to high background, so I asked the pilot to call MCC.
    06 Aug 2007 JS Matt replaced the neutron counter power supplly.
    05 Aug 2007 JS NTR
    04 Aug 2007 JS Get paged because FX status error higher than reference in bin 11. This showed up only in one run (75315), probably due to background.
    03 Aug 2007 JS Again, problem with the neutron counter.
    02 Aug 2007 JS determine a noisy channel to be masked out:
    • Sector DFB TDCchip channel
    • 10 0 1 31
    02 Aug 2007 RA/JB
    • Jerry V'avra reports that he has turned a neutron counter power supply back on.
    • Three calibrations taken this morning and at least 1 or 2 did not succed. Took another one now 1:30 and it succeeded.
    • Turned off channel 6 in sector 5 to kill a Christmas tree there. I set V0 and V1 voltage at 100V to avoid more trips (original value for V1 was 1012). Suspect PMT is the third from the right in row 11, since it is dead and in the middle of the blob in run (eg) 75220.
    • It appears that in sector 11 there might be 1 or 2 Christmas trees. There was a large occupancy in sectors 5 and 6 and when we turned off HV group #10 it disappeared (look at run 75245 and run 75247.
    • Scaler in Sector 10 was still hot and we belived it might be because of HV group#5 in sector 11 so we turned it off and indeed Sec.10 scaler went down.
    • We turned on HV group#10 in sector 11 and large occupancy in sector 5 and 6 didnt come back.
    • we turned on HV group#5 in sector 11 and we see a large occupancy in Sectors 11 and 10 (run#75251). ????
    01 Aug 2007 RA Updated pilot short-term instructions to ignore yellow alarms for neutron counters, since Jerry is working on this.
    31 July 2007 RA Neutron counters' power supply appears to have tripped. Unable to reach Jerry for instructions; since these are not critical to running, instructed pilot to ignore.
    30 July 2007 RA Jose changed the alarm thresholds:
    HIT_RATIO.HIGH 1.5 -> 2.5
    HIT_RATIO.HIHI 2 -> 3
    SCAL_SMALL_SPIKES.HIGH 50 -> 100
    SCAL_SMALL_SPIKES.HIHI 75 -> 200
    29 July 2007 RA Updated DQM short-term instructions to take note of low data in sector 8 bin 0.
    28 July 2007 RA Paged for a problem in configuring ORC with simultaneous alarm from DIRC crate. Probably a dataflow error, but eventually fixed by power cycling TSF crates. Paged again around 1600, another dataflow problem with accompanying DIRC alarm. Fixed by Matt Weaver replacing a board.
    27 July 2007 RA NTR
    26 July 2007 RA NTR
    25 July 2007 RA NTR
    24 July 2007 RA NTR
    23 July 2007 RA NTR
    22 July 2007 RA NTR
    21 July 2007 RA NTR
    20 July 2007 RA NTR
    19 July 2007 RA Matt turned off the dump valve for the UPS battery replacement (which was unsuccessful), then turned it back on again.
    18 July 2007 RA NTR
    16 July 2007 JS NTR
    15 July 2007 JS NTR
    14 July 2007 JS NTR
    13 July 2007 JS/JB Update the white board of the gaz system. Previously:
    • SOB: 150
    • sector 0: > 150
    • sector 1: 100
    • sector 2: 100
    • sector 3: 125
    • sector 4: 100
    • sector 5: 100
    • sector 6: 100
    • sector 7: 100
    • sector 8: 100
    • sector 9: 125
    • sector 10: 100
    • sector 11: 11
    Now:
    • SOB: 150
    • sector 0: > 150
    • sector 1: 100
    • sector 2: 100
    • sector 3: 125
    • sector 4: 100
    • sector 5: 125
    • sector 6: 150
    • sector 7: 125
    • sector 8: 100
    • sector 9: 125
    • sector 10: 100
    • sector 11: 11
    12 July 2007 JS NTR
    11 July 2007 JS Raise again the threshold for the integrated scalers : yellow @ 11, red @ 12
    10 July 2007 JS Unmask 8 channels of DFB 1 in sector 6.
    9 July 2007 JS NTR
    8 July 2007 JS NTR
    7 July 2007 JS Background seen by DIRC scalers is still high, mostly when they push the HER current above 1800 mA. We decided to raise the alarm thresholds:
    • scalers: yellow @ 600kHz, red @ 700 kHz (previously: yellow @ 400 and red @ 600 kHz)
    • rate ratios: yellow @ 3.0, red @ 4.0 (previously: yellow @ 1.5 and red @ 2.5)
    • integrated scalers: yellow @ 9.0, red @ 9.5
    Get paged in the afternoon to be informed that PEP was going to inject Helium for a vacuum leak chase.
    6 July 2007 JS Get paged yesterday night because of persistant red alarm in the DIRC scalers. Told the pilot to ask MCC to reduce the background.
    5 July 2007 JS Since yesterday there is a noisy channel in sector 6, harness 12, HV group 55.
    4 July 2007 JS NTR
    3 July 2007 JS Get paged due to bad timing plots. Tell the shifter that this is due to the bad trickle injection. The background causes a lot of dead time.
    2 July 2007 JS Jose update the DQM checklist
    1 July 2007 JS NTR
    30 June 2007 JS NTR
    29 June 2007 JS NTR
    28 June 2007 JS NTR
    27 June 2007 AP/JB/JS/ES ROD Activities:
    • X-Trees in sector 0 and sector 7 unplugged.
    • Replaced DFB #1 in sector 6 (Board 6E -> 3D).
    • Replaced DCC in sector 5 (Board 9 -> 26).
    • No work done on the crate filters.
    • Before the ROD, one calibration failed. This was due to a bad board in the master timing crate, Chris O'Grady replaced it in the afternoon.
    We took calibrations and cosmics, everythings looks OK. Nicolas un-did the changes he made on epics :
    • Unmask the alarm comming from the finisar of sector 5 with the command:
      caput DRC::FEE:FNS_ST.CALC "A&B&C&D&E&F&G&H&I&J&K&L"
    • Make the DIRC runnable without bypassing sector 5 monitoring:
      caput DRC:SC_5:RUN.CALC "(A=1)&&(B=1)&&(C=0)?1:0"
    • Unmasked sector 5 FEE alarms in the alarm handler
    • Unmasked sector 5 alarms in epics:
      caput DRC::FEE:ST:STAT.CALC "12-(A+B+C+D+E+F+G+H+I+J+K+L)"
      caput DRC::FEE:ST:STAT.INPH "DRC:SC_5:FEE:ST:STAT.VAL PP MS"
    • Unmask Xmas tree HV channel (sector 0, channel 1)
    26 June 2007 JS Get paged in the afternoon because the gaz flow in barbox 6 was higher than 100. I pointed the DQM to the short term instructions.
    25 June 2007 JS/AP/JB Plans in case of ROD on wednesday 27:
    • Replace the DCC in sector 5. Spare to be used: 26.
    • Replace DFB 6 in sector 3 (board ID 36). Spares that can be used: 64.
    • Replace DFB 1 in sector 6 (board ID 6E). Spares that can be used: 3D
    • Replace DFB 0 in sector 8 (board ID 80). Spares that can be used 68.
    • Unplug X-Tree in sector 0 (coordinates row = 2, column = 9 => DFB 8, channel 2, correspond to 5th one from right).
    • Unplug X-Tree in sector 7 (coordinates row = 39, column = 28 => DFB 13, channel 56, correspond to the 3rd from right to left and 3rd from top to bottom).
    24 June 2007 JS NTR
    23 June 2007 JS The HV group 1 in sector 0 tripped again this morning (we didn't get the alarm because it is disabled). Jose put the runnable value at 900V and for the moment it is working like that.
    22 June 2007 JS/JB Yesterday at 8pm, the HV group 1 in sector 0 tripped. This is due to the christmas tree. The HV has been turned off after that. This morning we turn on the HV step by step.
    Nicolas has done several changes in EPICS:
    • Mask the alarm comming from the finisar of sector 5 with the command:
      caput DRC::FEE:FNS_ST.CALC "A&B&C&D&E&F&G&I&J&K&L" (ignoring the input "H" which correspond to sector 5)
    • Make the DIRC runnable again by bypassing sector 5 monitoring:
      caput DRC:SC_5:RUN.CALC "(B=1)?1:0"
    • Mask sector 5 FEE alarms in the alarm handler,
    • Masked sector 5 alarms in epics:
      caput DRC::FEE:ST:STAT.CALC "11-(A+B+C+D+E+F+G+I+J+K+L)"
      caput DRC::FEE:ST:STAT.INPH "DRC:SC_5:FEE:ST:STAT.VAL NPP NMS"
    • Mask Xmas tree HV channel (sector 0, channel 1)

    DFB to be replaced:
    • sector 0, DFB 0 (board ID 4F).
    • 2 in sector 1, DFB 1 (board ID 11) and DFB 13 (board ID 7F).
    • sector 3, DFB 6 (board ID 36).
    • sector 6, DFB 1 (board ID 6E) .
    • sector 8, DFB 0 (board ID 80).
    21 June 2007 AP/JS/JB Activities of the day (ROD):
    • At 7 turn off the HV
    • turn off the FEE crates
    • As the temperature was raising, we turn off the chiller.
    • At 10:30 Mike replaced a connector card at the back of the chiller power siam.
    • At 11 turn on the FEE crate and chiller.
    • Turn on the HV. Need to reboot the HV-IOC because of UNV alarms
    • 11:30 take a calibration, everything looks fine.
    • Around 1pm we got white alarms in sector 5. Rebooting the DIRC MON-IOC didn't fix the problem. We took a calibration and a cosmic run that showed that data is ok, but the monitoring is not working. We're are linving without it for now, and if the ROD get extended until tomorrow (most likely) we'll like to replace the DCC in that sector.
    Plans in case ROD get extended tomorow:
    • Replace the DCC in sector 5. Spare to be used: 26.
    • Replace DFB 6 in sector 3 (board ID 36). Spares that can be used: 64 and 3D.
    • Unplug X-Tree in sector 0 (coordinates row = 2, column = 10 => DFB 9, channel 1).
    20 June 2007 AP NTR
    19 June 2007 AP Jochen masked out the 8 channels identified on 06/18/07.
    18 June 2007 AP New 8 channels to be masked out:
    • Sector DFB TDC chip TDCchipchannel
    • 0 0 0 0
    • 0 0 0 1
    • 0 0 0 2
    • 0 0 0 3
    • 0 0 0 4
    • 0 0 0 5
    • 0 0 0 6
    • 0 0 0 7
    17 June 2007 AP NTR
    16 June 2007 AP NTR
    15 June 2007 AP Chiller Problem Summary: HVAC people accidentally tripped pumps which deliver the supply of cool water to the DIRC chiller. The Temperature of the water in the chiller therefore started to rise (up to 19 degrees) and that tripped the chiller. The FEE crates automatically turned off. After HVAC people came and turned the pumps back on the temperature in the chiller started to go down. The DIRC chiller was then turned back on and then FEE elctronics were turned on. Everything seems to be working fine.
    14 June 2007 AP/JS The temperature sensors in sector 10 have been flaky since January 2007 and with the heat wave of these days they are even more. We are having alot of yellow alarms because of this, so Nicolas raised the inlet temperature yellow alarm threshold for this sector (from 34 to 35 degrees).
    13 June 2007 JB/AP/JS We think that the X-mas pmt in sector 0 is has the following coordinates X=10, Y=2. This corresoponds to DFB#9 ,channel#1
    12 June 2007 BJW NTR
    11 June 2007 JB/JS Determine 8 noisy channels:
    • sector DFB DFBchannel TDCChip TDCChipChannel
    • 6 1 0 0 0
    • 6 1 1 0 1
    • 6 1 2 0 2
    • 6 1 3 0 3
    • 6 1 4 0 4
    • 6 1 5 0 5
    • 6 1 6 0 6
    • 8 7 4 0 4
    10 June 2007 BJW NTR
    9 June 2007 BJW Paged around 11 AM again for FX Status bin 11 being high. Pointed DQM to short term instructions were it shows how to correlate it with ROM 3 and the HV in sector 6. Rewrote the DQM shortterm instructions to hopefully clarify this correlation procedure.
    8 June 2007 BJW Paged around 10:30 PM for FX Status bin 11 being high. Pointed DQM to short term instructions were it shows how to correlate it with ROM 3 and the HV in sector 6.
    7 June 2007 BJW Noticed 8 channel cold hole in sector 3 outer middle (c.f. run 73491).
    6 June 2007 BJW NTR
    5 June 2007 JB NTR
    4 June 2007 JB NTR
    3 June 2007 JB NTR
    2 June 2007 JB NTR
    1 June 2007 JB NTR
    31 May 2007 JB Got paged this morning because of high backgrounds. The background had been creeping up since yesterday evening until finally today it caused alarms. I talked to MCC and they were able to fix that.
    30 May 2007 JB/AP The scripts for inhibiting and uninhibiting the Chiller SIAM are the following:
    ~babardrc/Scripts/inhibitDrcPowerChillerSiamChannel
    ~babardrc/Scripts/unInhibitDrcPowerChillerSiamChannel
    Also the temperature strip chart which monitors the FEE can be restarted this way:
    open a terminal on this machine and type
    cd
    StripTool DircTemperatures.cnf &
    29 May 2007 AP Power chiller SIAM reenabled, FEE temperature strip shart removed from con01 and Pilot and DQM short term intructions updated about this.
    28 May 2007 AP NTR
    27 May 2007 AP/BJW
    • Many glitches in the power chiller SIAM, some of them lasted about 4 min, so Brad inhibited that SIAM channel via Nicolas' new script written this afternoon. Red alarms continued to appear in the ALH however, so Brad concluded Nicolas' script did not work and physically went into EH and inhibited the only DIRC SIAM which was showing a trip. 1/2 hour later the DQM noted that the DIRC occupancy plots look very distorted, with very high occupancy in the HV group 11 for all sectors as well as many other noisy channels. There were no alarms for the DIRC, so in consultation with Emmanuele and Nicolas we turned off HV group 1 in sector where there is currently a christmas tree. We tried going to standby to reconfigure as well, but that did not work either. We were considering taking a calibration when Nicolas realized that Brad inhibited by mistake the wrong SIAM channel, the one which sets our HV to runnable, so we were running for about an hour at low voltage in stable beams. The reason for the hight occupancy for the HV group 11 in all sectors was that the scalers are in that group and they never ramp down.
    • Updated the Pilot and DQM short term instruction telling the pilot to check the FEE temperature strip charts and to page the DIRC commissioner in case of any tendency of the temperature going up.
    • Nicolas put two new alarms in the ALH. The first one tells if more than 5 channels are in the V1 voltage configuration during stable beams, and the other tells if the HV SIAM channel is inhibited during stable beams.
    26 May 2007 AP NTR
    25 May 2007 AP NTR
    24 May 2007 AP A couple of things:
    • Got paged because after a global calibration DIRC didn't come to global mode. I told the pilot to go to the HV control panel and put The DIRC in Auto. That fixed the problem. Maybe this is correlated with some changes that Jim Hamiltom made to the calibration script. Jim is looking in to it now.
    • Just before the 3:45pm meeting we got a red alarm about the Power DIRC chiller SIAM. The Chiller and the SIAMs were ok. Ray checked the scope attached to the chiller power and no glitch observed.
    23 May 2007 AP NTR
    22 May 2007 AP NTR
    21 May 2007 AP NTR
    20 May 2007 AP NTR
    19 May 2007 AP NTR
    18 May 2007 AP Got Paged last night twice. The first one was around 4:30am because the bin 11 in the FX plot was above threshould. It was due to a noisy channel in sector 6, DFB 1, HV group 5. It went away after a couple of runs. The short-term instructions for DQM have been update telling about this noisy channel. The second page was around 5:30am due to a high scaler ratio, but it was in a situation of high background for all the subsystems. So i told the pilot to correlate high scaler rate with high background and complain with MCC.
    17 May 2007 AP NTR
    16 May 2007 AP The channels that Jose identified at May 14thwere masked out by Jochen. He updated too the gain files for the DFB 7 in sector 8 that we replaced at May 1.
    15 May 2007 AP Got paged around 8pm and several e-mails from the DQM because the time hit distribution for module 0 and 3 were scrued up. It was due to noisy channels in sector 0 and sector 6.
    14 May 2007 JB/AP
    • Matt has lowered the gas flow (at 10:40am) through slots 5, 6, 7 back to normal just for a couple of hours in order to check if the humidity in this slots changes.
    • dew point in sector 6 seems to be increasing, all other sectors are constant.
    • I took a standalone run:
      /nfs/bbr-srv02/calib/drc/conditions/drc/tt1-0045291742dd2e86-C.xtc
      /nfs/farm/babar/drc/babardrc/OedNtuples/standalonerun_14May2007_216726events.hbook.
      The results are:
      Sector 0 : DFB 0 chnl 1 with 9MHz
      Sector 0 : DFB 5 chnl 7 with 10MHz
      Sector 6 : DFB 1 chnl 7 with 1.6MHz
    • Mat Enabled the SOB dump valve around 6pm.
    • Took a calibration just after the DIRC shiled was in around 6pm. Everything seem to be fine.
    13 May 2007 ES Sasha came to disconnect the primary sensor from barbox slot 6, we'll stay like this for about a month, or until we have another access to the top of the detector
    also we found another possible noisy channel in sector 6, HV group 5 during cosmic runs, tomorrow we'll take a standalone run
    13 May 2007 AP NTR
    12 May 2007 AP NTR
    11 May 2007 AP
    • BE door opened around 9am.
    • Took calibration (everything fine). Turned HV off.
    • UPS people finished their work without problems.
    • BE door closed around 1pm.
    • Turned HV back on. Took calibration (everything ok).
    • DIRC now included in cosmic DAQ.
    10 May 2007 AP NTR
    9 May 2007 AP
    The pilot notest that since 10am the outside humidity and the SOB_electronics humidity were diverging. We're still investigating.
    8 May 2007 AP
    Pilot made a question about the variation of humidity in the barbox slot 6 (increasing). I told the pilot that humidity was correlated with the external temperature that is increasing too.
    7 May 2007 JB/AP
    - Lowered the low alarm limit of the +12V line of DFB#7 in Sector8. This line is flaky and produces alarms.The limits are now at 1V.
    - Updated DQM short term instructions. Report of N2 flow increase on barboxes 5, 6 and 7.
    6 May 2007 AP NTR
    5 May 2007 AP NTR
    4 May 2007 JB/AP Many things:
    • all sensors were put back and enabled.
    • dump valve still disabled.
    • humdity in slot6 still high.
    • Took a standalone calibration with Adc histograms: hbook/calibration_adchistos_20070504.hbook
    • Backward Doors were closed. Took many standalone calibrations afterwards and everything seems fine.
    • Rom files updated for gains due to dfb change.
    • took cosmics run and looked ok.
    • alejandro has just done a calculation to predict the time it will take to dry slot6 it is 10days!.
    3 May 2007 AP Many things today:
    • The high voltage was turn back on. The Wiener creates 0, 1, 5, 6, 7, 9, 10 and 11 came back in test mode due to no posibility of reach them (no bridge).
    • We took a couple of calibrations and seems that the DBF in just replaced in sector 8, slot 7 is ok. And we took some a cosmic run with DIRC and DCH and everything seems ok.
    • The +12 voltage line of DFB sector 8, slot 7 is reading +7.6 on EPICS. All LEDs of this DFB looks ok. The Alarm has been masked out by changing the threshold on EPICS.
    • Water doesn't leak anymore (no dripping) and the humidity it seems to go down slowly (but is still hight compared to the normal conditions). For the time been no action is schedualed.
    • Now the gas flow in barboxes 5,6 and 7 are at 160cc/min, 160cc/min and 200cc/min, respectively (They were increased in order to help in the drying proccess of the barboxes). Emmanuele changed the threshould of barbox 6 in EPICS in order to mask out the alarm because of this.
    • All the humidity sensors were put back in position but not working. Maybe Mat is coming tomorrow to make them work again.
    2 May 2007 JB
    • Scope was set to monitor the chiller power. It will remain conected while chiller is running in single trigger mode so if there is power glitch it will latch.
    • The water level in the SOB reservoir is lower by 1/2 inch compared to yesterday. Matt says it is OK, the fluctuations are large.
    1 May 2007 JB Many problems today.
    • Rod started 7am, but magnet Q2 shorted and burned at 7:50am.
    • A cooling hose broke in the DIRC tunnel and we had a large water leak probably this shorted the magnet.
    • Matt disabled the SOB water dump valve.
    • All power was turned off.(HV,LV,Chiller), Gas system still running.
    • The water did not get into the Wienner crates :).
    • We replaced the bad DFB #7 (id 6C) in sector 8 with DFB id 8D. Still need to confirm that it works fine because LV is still off.
    • DFB that was taken out was tested at drcstand and 15 dead channels confirmed.
    • Around 4pm the bar humidity sensors started to trip. We confirmed visually that there is water coming out of sector 6 the rate is ~14 drops/min.
    • The gas flow in sector6 barbox has been increased by 10% at 7pm.

    30 April 2007 JB NTR
    29 April 2007 JB NTR
    28 April 2007 JB NTR
    27 April 2007 JB NTR
    26 April 2007 JB Today is ROD. I took the chance to take some calibrations and a random trigger run. The random trigger run is /nfs/bbr-srv02/calib/drc/conditions/drc/tt1-0044d0753a951706-C.xtc. The first calibraion was calibration_tdc_200704261129.hbook with no histograms, the second was calibration_tdc_200704261245.hbook with TDC histograms. No calibration with ADC histograms was taken.
    25 April 2007 JB NTR
    24 April 2007 AP NTR
    23 April 2007 AP Got paged twice in the afternoon because of long periodes of high scaler rates. Told the pilot to be persistent in complaining MCC to do something about it.
    22 April 2007 AP NTR
    21 April 2007 AP NTR
    20 April 2007 AP Got paged last night 4am due to SynchRequest error form crate 0x40000, the crate replaced by Chris O'Grady yesterday. Chris told the pilot to rebooting the partition and that solved the problem temporary. Then, in the morning we had a optical fiber disconnection from the same crate. So Chris replaced for the old one, and everything is Ok.
    19 April 2007 AP Ray put some stickers in the DIRC chiller saying the voltage of power supply (safety reasons). Nicolas put an alarm in the ALH about the power DIRC chiller glitches, with instructions to the pilot to tell the DQM to go check the DIRC chiller.
    18 April 2007 AP NTR
    17 April 2007 AP NTR
    16 April 2007 JB NTR
    15 April 2007 JB A couple of noisy channels ( one in sector 0 and one in sector 6) have been coming and going.
    14 April 2007 JB I see a few glitches in the chiller monitoring plots at ~7pm yesterday, not sure why. Nothing very serious.
    13 April 2007 JB Updated short term instructions. there is a noisy channel in Sector 9 hv group #26.
    12 April 2007 AP NTR
    11 April 2007 AP NTR
    10 April 2007 AP NTR
    9 April 2007 AP NTR
    8 April 2007 AP The DFB #7 in sector #8 is not missing any more after taking a global calibration with the magnet ramped down. But in the history for the calibrations the mean time for the TDC chips 0 and 1, DFB #7, sector #8 have changed since friday. We are still investigating.
    7 April 2007 AP The same problem with the global calibration plots.
    6 April 2007 AP In the occupancy global calibration plot the DFB #7 in sector #8 is missing. Nicolas said that this is a feature of the macro that makes the plot and not a real problem (loking at the calibration history everything looks fine). Working on it for traying to undestand it.
    5 April 2007 AP We have a X-tree than only shows up during taking cosmics. It is in sector 7 HV group 55. Nicolas Added one plot to FM about FX error 11. The new plot helps identifing if an FX error 11 is because high Background or noisy channels. Finally, Jose identified one noisy channel in sector 6, row 16 and column 34 that maybe have to maskedout.
    4 April 2007 BJW NTR
    3 April 2007 BJW NTR
    2 April 2007 BJW NTR
    1 April 2007 BJW NTR
    31 Mar 2007 BJW Paged this morning for high background - scalars in the red. Pilot claimed MCC wasn't being responsive about reducing it. I called MCC and they said they were working on it. Within 15 minutes most sectors were back in the green (with the exception of sector 3). MCC was apparently more responsive later when the scalars had trouble and tracked the background trouble to a timing board, which they replaced.
    30 Mar 2007 BJW Nicolas added ALH logic to catch a misinhibited BCS SIAM state in the future. I added a DQM short term instruction about the 8 channels on the DFB in sector 8 which are sometimes dead because this shows up in the plots.
    29 Mar 2007 BJW Nicolas inhibited the backup chiller SIAM, which we had forgotten to do after the EH power outage Monday.
    28 Mar 2007 BJW NTR
    27 Mar 2007 BJW Jochen tested all the masked channels to verify they still needed to be masked and added the 3 new channels: Sector 0, DFB 5, channel 12, Sector 0, DFB 11, channel 0, Sector 6, DFB 9, channel 0.
    The sector 8 DFB we replaced yesterday had the same 8 channels missing in the test stand, but the one we replaced also is now missing the same 8 channels. Ergo there must be some underlying problem with the DCC or Wiener Crate which is killing the 8 channels on the DFB in this slot.
    26 Mar 2007 BJW Trip in EH shut off our HV and Wiener crates (and other peoples stuph, too). Wiener crates came back fine but IOC had to be rebooted to get HV crate 1 to register. (The other 5 crates came back fine.)
    Backward doors opened for the ROD (for EMC), so we took the opportunity to replace the faulty sector 8 DFB (# 87) and the sector 5 DFB with the overlarge timing window (# 5A).
    25 Mar 2007 BJW ntr
    24 Mar 2007 BJW Paged at 6 AM by DQM for low bins in sector 0 hits per sector. Actually there are now 3 high bins. They are also visible in the previous run's FM plot (71442), but the DQM did not notice them last run. Since the other apects of run 71442's FM plots looked fine I told them to keep on truckin'. Looking at the FM for run 71443, there seems to be a new noisy channel in DFB 0 (and no longer an 8 channel hole).
    23 Mar 2007 BJW FM plot documentation updated to include Nicolas' changes. Paged around 11:30 PM for slightly low HV bin in sector 8 - I assured the DQM that the data would be okay after checking the FM plots.
    22 Mar 2007 BJW Nicolas created an expert pagein JAS to see the FX status plots per ROM and added the corresponding plots to our FM plots script.
    21 Mar 2007 BJW Page about FX status plots last night - due to 3 noisy channels: Sector 0, DFB 5, channel 12, Sector 0, DFB 11, channel 0, Sector 6, DFB 9, channel 0. Short term DQM instructions updated until we mask them during the ROD next week.
    20 Mar 2007 AP NTR
    19 Mar 2007 AP NTR
    18 Mar 2007 AP NTR
    17 Mar 2007 AP Got paged at 18:49 due to bin 11 in the Feature extration exceded its limit. There was under very bad trickle injection. Told the DQM to call MCC and tell them that the Background was high. After a while the problem gots fixed.
    16 Mar 2007 AP NTR
    15 Mar 2007 AP NTR
    14 Mar 2007 AP NTR
    13 Mar 2007 AP On March 13 at 3:30am get paged because one scalar (sector 1) went red and kept reading at frequencies of more than 600kHz for about 40 minutes. All the others scalars were fine. That happened during an ODF-CMP problem and it is maybe related to that.
    12 Mar 2007 AP NTR
    11 Mar 2007 AP NTR
    10 Mar 2007 AP NTR
    09 Mar 2007 AP NTR
    08 Mar 2007 AP NTR
    07 Mar 2007 AP Yesterday afternoon high backgrounds conditions on DIRC, causing asymetric "shoulders" in the tming plots (FM pag. 7). Emmanuele made a test with MCC, and found that the asymetric shape is due to bad trickling injection in LER.
    06 Mar 2007 AP NTR
    05 Mar 2007 AP NTR
    04 Mar 2007 AP As in Friday, at 4:22pm we had high dead-time mainly coming from DIRC. Run restarted and the problem was fixed.
    03 Mar 2007 AP NTR
    02 Mar 2007 AP Matt(ODF) and me (Alejandro) have been paged because at 5pm BaBar was having 80% death time from DIRC not due to Backgrounds. The shifters saw "DLinkB_NoStartBit" damage in the Dataflow Fast Monitoring pages for crate 20 and no Drc data for Sectors 10 and 11. This damage type indicates that one (or more) DFBs is sending no data when the ROM is expecting to read an event from them. There's no indication which DFB in Sector 11 may be a problem. The configure transition for the following run failed in the same sector. Unfortunately, we lost the ROM logfile output for this configuration error. Matt told me that in case that the problem shows up again, we have to tell the pilot to restart the run and if this doesn't work to page him. Matt also gave me a method for finding and masking the problematic DFBs in case we need it to do at the midle of the night. This is in the e-mail that i send to babardrc.
    01 Mar 2007 AP NTR
    28 Feb 2007 AP Last night have been one of very high background, with scalers readings over 400kHz, and with peaks up to 1MHz.
    27 Feb 2007 AP Power suply of FEE crate fixed by Ray. Now we have two in the test stand.
    26 Feb 2007 AP NTR
    25 Feb 2007 AP NTR
    24 Feb 2007 AP NTR
    23 Feb 2007 AP NTR
    22 Feb 2007 AP Been paged by DQM due to mismatch of Data and reference in TDC hits plots. Told him that is a known problem. I have added entry in the DQM Short term instruction about it.
    21 Feb 2007 JB 1)The gas flow in sector 2 is normal today. 2)The DIRC chiller water flow is not as stable as usual; it is fluctuating down to 6.1 GPM. Normally it is at 6.5 and quite stable at this value. However the error allowed according to the instructions there is .7 so it is still Ok.
    20 Feb 2007 JB I just did the DIRC tour (3:32pm) and the gas flow in sector 2 seems to have dropped to 85; normaly it is at 100+-25. When I tap on the flow meter it goes to 100 but then it drops down to 85 again; normally when I tap on it the flow goes to 100 and stays there .
    19 Feb 2007 JB Got paged last night due to a transient alarm DRC:SC_3:FEE:ERR:GLINK. This error might be related to ROM, fiber, or DCC failures.
    18 Feb 2007 JB Nada
    17 Feb 2007 JB Got paged yesterday evening due to high backgrounds. The timing distributions looked distorted, specially in run 70345.
    16 Feb 2007 JB Nada
    15 Feb 2007 JB We saw high backgrounds last night,and there were no alarms in ALH because of INJECTION status, Nicolas will add alarms that will work during INJECTION status.
    14 Feb 2007 JB We switched from the BCS to the Dirc Chiller with no problems.A plastic shield was installed on the Dirc chiller to cover some HV lines.
    13 Feb 2007 JB Today is ROD: We replaced DFB 95 (Sector6) with DFB 46. Also Brad installed a humidity sensor in the west side. The doors are closed and the calibrations look fine.
    12 Feb 2007 JB Due to the cooling problem with BaBar, we had to turn off the HV to all the DIRC. Recovered fine.
    11 Feb 2007 JB Nada
    10 Feb 2007 JB Last nights global calibration showed on hot dfb and one cold dfb in sector0. Also after that HV group #11 in sector 0 became noisy.
    9 Feb 2007 JB Nada
    8 Feb 2007 AP Added a note to the short term instruction about some inefficiency in sector 1, HV group 19, visible in JAS FM. That's understood, so the shifter don't need to make an entry in logbook.
    ES unfortunately it looks like DFB 13 in sector 1 has a group of inefficient channels. to be investigated.
    7 Feb 2007 BJW Added a note to the long teerm instructions about background, "Please get feedback from MCC what they did to lower the scalars and record it in the electronic logbook.", because Jerry expressed curiousity about what exactly they are doing.
    ES in the late afternoon we started having trouble from a DFB slot 7 sector 1 (DFb number 95), all the voltage lines were flaky and we had a consequent temporary dataflow error. later in the evening, that DFB started to be very noisy and we had to mask it out both from configuration and data acquisition. run 70047 looks very noisy (but still good); the following runs are good without that board.
    6 Feb 2007 BJW NTR
    5 Feb 2007 BJW Vesda work caused a trip and shut off our Wiener crates. I got 12 fan speed pages and headed up to IR-2 to supervise turning the DIRC back on, which went fine.Chris O'Grady replaced a ROM board hoping to fix the calibration validation problem. We took a standalone calibration to check the board Jochen masked out in sector 0, harness 5 is correct.
    4 Feb 2007 BJW Paged ~4:30 PM for high background. The number in ALH is the alarm/noalarm binary and not the scalars value which confused the pilot because our long term instructions say a zero scalar value is very bad. I updated the long term instructions to clarify.
    3 Feb 2007 BJW Zilch (NTR)
    2 Feb 2007 BJW Jose noticed in the fastmonitoring that the 8 PMT hole in harness 0 of sector 0 reappeared again last night for runs 69867-69875. Jochen masked sector 0 DFB 5 channel 3, which made that high background harness return to normal. Chris and Matt think they understand the problem with the calibration failing the first time in the validation stage, and it is in the ROMs.
    1 Feb 2007 BJW Nothing happened today.
    31 Jan 2007 BJW Emmanuele and I replaced non-configuring board 9C in sector 9 with 43. We also tested 9C and 99 (which had earlier failed to configure in the DIRC) in drcstand and found 9C failed to configure there as well, but 99 appears okay so it was put back in sector 9. DIRC now passes unmasked configuration.
    30 Jan 2007 BJW Paged around 2:15AM because the DQM JAS timing plots were asymmetric. Starting a new run fixed the problem. Denis looked into the enabled/disabled blinking on our background panels and discovered an incompatability in the severity definitions which he plans align. Nicolas however said that this is normal behavior because the scalers integrate hits for 0.5 seconds every ~4 seconds.
    29 Jan 2007 BJW Paged for scalar alarms in sectors 1,2,4 & 8 around 2:30AM which lasted for about 1/2 hour. At the same time there was also dataflow trouble with a board in sector 9 which Chris O'Grady masked because it wouldn't configure. Investigating the configuration problem further today Emmanuele, Joe and I found sector 9 DFB 10 had a problem with TDC 1 (counting from 0), namely, the timecorr range was ~2 times as big as it should have been. Checking the rest of the DFBs we found similar behavior in TDC 1 of DFB 10 in sector 5. Sector 0 DFB 1 was also found to have TDC 1 problems with two side peaks. We asked to open the west door during the ROD on Wednesday to replace the 2 boards in sectors 5 & 9. I was paged this evening around 9:30 regarding high background in bin 11 of the JAS plots - apparently it was close to .01 so I told them to tell MCC to see if they could steer it down. Gerard (pilot) also wondered why in EPICS our background panels blinked enabled/disabled. I assured him that the alarms propagated correctly to the ALH. Around 10:30 an email went out to babardrc that "Script orcCmlogToAlh not running: restarted!" and Nicolas was the first to notice, adding a cronjob to check every 5 min to see if ~babardrc/Scripts/orcCmlogToAlh is running and restart if needed.
    28 Jan 2007 BJW Noticed in the fastmonitoring of run 69689 on the lower left side (harness 0) of sector 0 the number of "dead" PMTs seems to fulcuate a lot This fluctuation doesn't seem to correlate recorded in the logbook either.
    27 Jan 2007 BJW Talked to DQM (I was day pilot) about noisy sector 0 DFB #6 being due mostly likely to low luminosity and updated the shortterm instructions so future DQMs will also be aware.
    26 Jan 2007 BJW Scalers in the red for more than a few minutes (I was day pilot), talking to MCC they were able to tune on it and lower them.
    25 Jan 2007 BJW Updated the DQM JAS plots for low-lumi datataking. Paged around 11pm for high background in sectors 0, 6, 7 & 8.
    24 Jan 2007 BJW Nichts (NTR)
    23 Jan 2007 JB Nada (NTR)
    22 Jan 2007 JB Nada (NTR)
    21 Jan 2007 JB The fan speed monitoring plots dont show anything sensible today, I checked the fans in epics and in ambient and they are Ok. Probably the program producing the plots is screwed up.
    20 Jan 2007 JB Nada (NTR)
    19 Jan 2007 JB Nicolas turned off most the sectors last night due to very high backgrounds which were observed due to PEP tuning, he only left one HV group on corresponding to the scaler in sector 3. This morning PEP has better beams and we have turned all sectors back on, BaBar will take a couple of hours of colliding data. The background levels seem to be ok now.
    18 Jan 2007 JB Nada (NTR)
    17 Jan 2007 JB The DIRC chiller underwent preventive maintenance early this morning, at 9am we switched from the back-up chiller to the DIRC chiller and no glitches have been observed with the SIAM until now 5pm.
    16 Jan 2007 JB Emmanuele and Ray replaced the faulty chiller SIAM, no problems were encountered. We will switch from back-up chiller tomorrow after maintenance is done on the DIRC chiller.
    15 Jan 2007 JB Nada Para Reportar(NTR).
    14 Jan 2007 JB The fast monitoring plots are working again since last night, apparantly the computer producing them had been turned off. Also the noisy DFB in sector 0 is hot again a shown in the fast monitoring plots starting on 69509 .
    13 Jan 2007 JB The hot DFB in the fast monitoring plots has become normal again. However the fast monitoring plots have become empty as of run 69487, James Hamilton has been notified.
    12 Jan 2007 JB On the fast monitoring plots we noticed that on the page of hits/harness sector 0 has dfb 6 very hot, this also shows on hits/hv group, the first run to have this is 69455.
    11 Jan 2007 JB Nicolas has masked out alarms from DFB # 6 in sector 0 and DFB #11 in sector 6 => the number of alarms should go down drastically in those 2 sectors. ***That's something to keep in mind for when these DFBs get changed to have the change undone.***
    11 Jan 2007 JB Due to the problem with the chiller siam we have switched to the backup chiller since Jan 9 evening.
    9 Jan 2007 JB Matt and Ray checked the chiller power with an oscilloscope and found that when there are chiller SIAM alarms the chiller is ok. There were several persistent alarms this morning around 10am. The SIAM may need to be replaced in the next ROD. At the moment the SIAM has been inhibited.
    8 Jan 2007 BJW Paged in the AM about DCC in sector 1 - same problem as in mid-December. Data still good and the problem disappeared again.
    7 Jan 2007 BJW NTR
    6 Jan 2007 BJW NTR
    5 Jan 2007 BJW Paged about the new chiller numbers - Emmanuele added a note of clarification to the short term instructions.
    4 Jan 2007 BJW Replaced DFB in sector 4, slot 6 which was failing calibration. Nicolas masked two DFBs (sector 0 DFB 6 and sector 6 DFB 11), committed drc-mon and drc-hv changes to CVS, restored running ALH configuration, re-enabled paging scripts, updated the DFB gains in the ROM files, and a partridge in a pear tree.
    3 Jan 2007 BJW NTR
    2 Jan 2007 BJW Emmanuele, Jose & I replaced the DCC in sector 3 and 2 DFBS (#0 & #3) in sector 4. Calibrations look good thereafter.
    1 Jan 2007 BJW NTR
    31 Dec 2006 RA NTR
    30 Dec 2006 RA NTR
    29 Dec 2006 RA NTR
    28 Dec 2006 RA Chiller water levels dropped a bit, chiller went into alarm, Wiener crates turned off. Red alarms from fans. Chiller came up again all right. Later (around 2000) : There it goes again. Emmanuele switched to backup chiller. It seems we definitely have a leak.
    27 Dec 2006 RA NTR
    26 Dec 2006 RA NTR
    25 Dec 2006 RA NTR
    24 Dec 2006 RA NTR
    23 Dec 2006 RA NTR
    22 Dec 2006 RA NTR
    21 Dec 2006 RA NTR
    20 Dec 2006 RA NTR
    19 Dec 2006 RA Wiener crate monitoring shows an average of ~2000 alarms this cycle; usually the average is around 500. Not sure if this is a problem or not.
    18 Dec 2006 ES Today we took a cosmic run. Although we see some noisy electronics channels, we can see that the DIRC has been quiet and taking data smoothly. Unfortunately, after 4 pm, the fast monitoring got stuck so the last run that we can check is 69139.
    17 Dec 2006 RA NTR
    16 Dec 2006 RA Unable to take the evening calibration due to problem with cen-bip. Matthias will fix.
    15 Dec 2006 ES We did not open the doors today because yesterday night sector 1 suddendly turned green. We preferred waiting until the second of January to do only (hopefully) one door opening, replace what has given us trouble in the first weeks. We do not see the emergency of opening today.
    14 Dec 2006 RA Sector 1 has a large problem: Epics is reporting very wrong voltages. Since the calibrations look all right, it seems that this is a monitoring problem, presumably a problem with the DCC. Since the sector has shown this kind of problem before, though, it is possible that the problem lies elsewhere, although we hope not, since DCCs can be replaced. Our plan is to open the doors and install the bridge tomorrow, move the DCC around to see if the problem shows up, and otherwise do diagnostics. At the same time we will replace the problem DFBs in sectors 0 and 4.
    13 Dec 2006 RA Board 67 in sector 0 is giving us bad voltage readings. Not sure of the cause yet. It looks ok in calibrations, though. We will take two calibrations daily in the hope of spotting any further problems. Also, we will check the development of all the other DFB voltages to see if any of them are beginning to drop, since this one showed a gradual decline before it went into alarm territory. Further, the chiller is showing high flow and low pressure; we will have people working on it tomorrow.
    12 Dec 2006 RA The replaced board in sector 4 has eight missing channels. Since it was only put in yesterday, it seems unlikely to be corrosion. We won't replace it until January, and will hope that it's an isolated case. With so many fragile electronics being bumped about, some of them almost have to have problems. We have the regular chiller back.
    11 Dec 2006 BJW Problems with 2 DFBs in sectors 6 & 4 to be addressed by reopening the DIRC. HVAC group did work on the DIRC chiller.
    10 Dec 2006 BJW NTR
    9 Dec 2006 BJW NTR
    8 Dec 2006 BJW Doors closed. The DIRC is fine (tm).
    7 Dec 2006 BJW Reassembled most of the rest of the DIRC.
    6 Dec 2006 BJW Had a meeting to finalize installation schedule for the next two days to get the DIRC completely reconstructed and the shield doors closed before rain (and high humidity) returns on Friday. Emmanuele, Rolf & I put preventatively repaired fantrays back in sectors 4, 5 & 7 (one already back in six since yesterday). The chiller had a flow fault around noon which cut power to the Wiener crates so we reinhibited the chiller SIAM. Nicholas, Phillipe & Anne Marie populated these 4 sectors with the same DFBs they had previously held.
    5 Dec 2006 BJW NTR
    4 Dec 2006 BJW Rolf & Emmanuele removed the other 6 fantrays to recieve preventative maintainence from Ray.
    3 Dec 2006 BJW NTR
    2 Dec 2006 BJW First 10 DFBs back from Lupe's lab. 80% tested fine in the test stand, but the 2 had problems unrelated to the conformal coating. Ergo Lupe was given the green light to do them all.
    1 Dec 2006 BJW The first set of boards went to Lupe's lab to be cleaned, recieve the 3 jumpers, and be conformally coated. Rolf, Emmanuele & I reremoved the metal plates from the DFBs which had tested okay in the test stand. Christophe fixed ~20 DFBs.
    30 Nov 2006 BJW Mtg this morning to decide how to proceed with the electronics. DFBs will all get preventative jumpers on the 3 traces which failed do to corrossion in some. DFBs will all get a protective silicone coatingto prevent future corrossion on the traces (Q-tip applicaple Dow Corning 1-2577). Ray saw 1 trace become corroded in 90% of the fantrays he has fixed (causing the loss of the CANBUS address), so these will also all be preventatively jumpered. Emmanuele & I removed the first set of six and put them in Ray's office (9, 10, 11, 0, 1 & 2). The Wiener crate PSs are more complex, so it was decided to order replacement parts because there was no usual supect trace. On the bridge today we also noticed that the 4 Wiener crates on top of the DIRC (10, 11, 0 & 1) are not screwed in. Unfortunately these require metric screws which Matt is trying to locate for us. Matt also vaccumed out the crates again after we removed the fantrays.
    29 Nov 2006 BJW Power outage in the EH last night ~1am shut off the Wiener crates. Emmanuele restarted those 10 crates this morning. Rolf & I started up the last two fantrays in crates 5 & 8 after replacing the fixed fantray in 8. Shipped a faulty DCC to France for repair. Nicholas & Cristophe tested the last two sectors of DFBs. Biweekly meeting held where plan to repopulate the DIRC was discussed (details still in flux).
    28 Nov 2006 RA Christophe and Nicolas have tested about 60 boards, and found only a few corroded tracks; these will be fixed by adding a jumper. Since these boards were ok at our last calibration, the corrosion must have started quite recently, and been stopped by the cleaning - at least, we hope it's stopped! Due to a power loss, it will be necessary to restart the external-fan procedure. Brad and Emmanuele started with sector 10 at 1000.
    27 Nov 2006 RA (1100) Moved external fan to sector 7; turned sector 6 Wiener crate on. (1400) Moved external fan to sector 8; turned sector 7 on. (1700) Moved external fan to sector 9, turned sector 8 on.
    26 Nov 2006 RA NTR
    25 Nov 2006 RA NTR
    24 Nov 2006 RA NTR
    23 Nov 2006 RA Thanksgiving weekend - we will keep an eye on the Wiener crates in Epics, but won't be able to access any part of IR2, even the outer hall.
    22 Nov 2006 RA Sectors 0, 1, 10 and 11 are running. Nicolas has got the last DFBs back from Naprotek. The metal plates have now been washed twice.
    21 Nov 2006 RA Moved external fan to sector 1; turned sector 0 on.
    20 Nov 2006 RA Got back DCCs from Lupe; will test them over the next few days. Remaining 139 boards were brought to Naprotek. We should get the last ones back by Wednesday.
    19 Nov 2006 RA NTR
    18 Nov 2006 RA NTR
    17 Nov 2006 RA Per Naprotek's request, we took the metal plates off the remaining boards (having gotten 28 of them back, which Nicolas will test). 8 screws per board, times ~140 boards... Lupe will clean the remaining DCCs.
    16 Nov 2006 RA Matt cleaned nine of the sectors; Lupe cleaned a DCC using alcohol and soft-brush, and it now works nicely on the test-stand.
    15 Nov 2006 RA Removed all the DFBs and DCCs from the DIRC for cleaning. Found an SOB leak in sector 8; Matt and Jerry were able to repair it.
    14 Nov 2006 BJW Completed labelling and unscrewed the wires attached to the DFBs in preparation for removal
    13 Nov 2006 BJW Labeled the wires attached to the DFBs in preperation for removal
    12 Nov 2006 BJW NTR
    11 Nov 2006 BJW NTR
    10 Nov 2006 BJW NTR
    9 Nov 2006 BJW NTR
    8 Nov 2006 BJW The consensus at the biweekly meeting is to leave the DIRC off for now until we get more information on the DFB failures from the engineers in France.
    7 Nov 2006 BJW Bridge installed to do work on sectors 1&2. Fan trays and PSs in several sectors removed for repair. DCC in sector 1 removed.
    6 Nov 2006 BJW Rebooting DRC-MON IOC fixed white gas flow numbers in EPICS. DCC trouble in sector 1. Lost EPICS control of crates in sectors 2, 6 & 8 - hardware in fantrays lost the CANBUS address (can still be controlled manually).
    5 Nov 2006 BJW NTR
    4 Nov 2006 BJW Nicholas paged me about trouble in sector 1. Today it was blinking "+12V Overvoltage". All FEE temp sensors white in EPICS. Shut off DIRC for the rest of the weekend. Three gasflow monitors white in EPICS (0,4&8) - not at real problem because screens on top of BaBar show good values.
    3 Nov 2006 BJW Removed failing sector 7 Wiener crate PS. Today it was blinking "+12V Overvoltage". Overvoltage is a possible reason for so many recent DFB failures in this sector. Plan to test the PSs from the other sectors with recently failing DFBs after to talking to Ray Monday.
    2 Nov 2006 BJW Sector 7 Wiener crate failed, blinking "-5 low voltage". Emmanuele powercycled it and it came back up okay, but we will monitor it to see if the PS needs to be replaced.
    30 Oct 2006 RA NTR
    29 Oct 2006 RA NTR
    28 Oct 2006 RA Got a page from 679-4469; nobody at that number knew anything about it.
    27 Oct 2006 RA NTR
    26 Oct 2006 RA NTR
    25 Oct 2006 BJW Replaced 6 DFBs and one HV board.
    24 Oct 2006 BJW NTR
    23 Oct 2006 BJW Got 3 pages driving into SLAC this morning, the worst of which said "SOB dump". Fortunately this was not the case and some replacement work was being done to the power so Matt had already disabled the SOB dump valve. The HV would also be affected so I turned that off as well. Emmanuele came in and the three of us replaced a hard-to-reach DFB in slot 0 of sector 9 since Matt will be at LBL tomorrow. We will leave the HV and Wiener crates off until we complete the rest of the work scheduled for tomorrow.
    22 Oct 2006 BJW NTR
    21 Oct 2006 BJW NTR
    20 Oct 2006 BJW NTR
    19 Oct 2006 BJW NTR
    18 Oct 2006 BJW All Wiener crate fan speeds changed (some up, some down) between 12 am 1 pm yesterday. The changes were only about 10-20 rpm however.
    17 Oct 2006 BJW Twenty HV channels in 6 different sectors tripped at 1:22pm.
    16 Oct 2006 BJW NTR
    15 Oct 2006 BJW NTR
    14 Oct 2006 BJW NTR
    13 Oct 2006 BJW NTR
    12 Oct 2006 BJW NTR
    11 Oct 2006 BJW NTR
    10 Oct 2006 RA NTR
    9 Oct 2006 RA The cause of the damaged DFBs remains mysterious. They were all in sectors 6 to 9, but there is no evidence of any particular hard usage in these areas.
    8 Oct 2006 RA NTR
    7 Oct 2006 RA NTR
    6 Oct 2006 RA Nicolas replaced the sector 6 front-end crate, as it was turning itself off. Several DFBs have been damaged, and were replaced.
    5 Oct 2006 NA Not a good day for our hardware... Found Wiener crate 6 off this morning. After turning it on via epics, Jochen K. reports online problems which are finally tracked down to a faulty DFB in sector 6 (DFB 63, 4th from the top of the crate, replaced by DFB 09). Then, looking at the ALH, I noticed that all parameters from DFB #13 (closest to the DCC) in sector 9 are in error => DFB to be replaced tomorrow. Caveat: how to access safely this crate (big ladder!?)? How to fix the harness? When I went back home I opened epics and saw that WCR #6 is off again. We do have a problem with it => I'll ask Ray and Mike tomorrow. Last but not least, the HV channel #1 in sector 4 tripped again during the night just like yesterday morning => there's likely a Xmas tree here. To be investigated when the front-end hardware is fixed...
    4 Oct 2006 RA Replaced CAEN crate number 3. Board 8 in it was still not working; replaced it. (Out 20004698, in 20004701). On coming back up, we observed 'holes' of about 8 PMTs in sectors 4 and 6, probably due to DFB problems (Sector 4, DFB 0, Sector 6, DFB 13). One high-voltage channel (Sector 4, board 1, channel 31) has the correct voltage, but no current; it is not an epics problem. For now we will not take any action, until we know what is happening.
    3 Oct 2006 RA Emmanuele, Nicolas, Jochen, and I switched the HV crates back on. Crate 3, board 8 does not respond properly; it reports HVMAX. Since this is the same crate that's given trouble before, we decided to replace it as soon as Paul Stiles has been consulted. On ramping up the voltage, Sector 8, channels 45 and 48 reported overcurrents; this disappeared when we ramped down and back up. Also plugged the crate measuring field strength back in - it had been unplugged by accident.
    2 Oct 2006 RA NTR
    1 Oct 2006 RA NTR
    30 Sept 2006 RA NTR
    29 Sept 2006 RA NTR
    28 Sept 2006 RA NTR
    27 Sept 2006 BJW Now monitoring the Wiener crate fans daily.
    26 Sept 2006 BJW Switched all the Wiener crates to run in normal rather than bypass mode with Emmanuele. Sector #3 came up all white when we switched it on, but powercycling the crate itself fixed this. DFB #6 in sector #7 was also faulty so we replaced it (76 -> 0E) and updated the appropriate databases. The Wiener crates are now on. The SIAMs are also inhibited because Walt wanted us to leave the chiller off, and the alh was set to normal configuration for FEE & WCR.
    25 Sept 2006 BJW NTR
    24 Sept 2006 BJW NTR
    23 Sept 2006 BJW Received a junk page for 733-9400 which is apparently not a real phone number.
    22 Sept 2006 BJW NTR
    21 Sept 2006 BJW Paul Stiles had a look at HV crate 3 and came to the conclusion that are troubles with it are just coincidence.
    20 Sept 2006 BJW NTR
    19 Sept 2006 BJW Matt has isolated the problem with the resistivity sensors and they are giving us good numbers again.
    18 Sept 2006 BJW Matt noticed a discrepency in the resistivity readings which were ~ an order of magnitude off.
    17 Sept 2006 BJW NTR
    16 Sept 2006 BJW NTR
    15 Sept 2006 BJW NTR
    14 Sept 2006 BJW NTR
    13 Sept 2006 BJW NTR
    12 Sept 2006 BJW NTR
    11 Sept 2006 BJW NTR
    10 Sept 2006 BJW NTR
    9 Sept 2006 BJW NTR
    8 Sept 2006 BJW NTR
    7 Sept 2006 BJW NTR
    6 Sept 2006 BJW NTR
    31 Aug 2006 RA NTR
    30 Aug 2006 RA NTR
    29 Aug 2006 RA NTR
    28 Aug 2006 RA NTR
    27 Aug 2006 RA NTR
    26 Aug 2006 RA NTR
    25 Aug 2006 RA NTR
    24 Aug 2006 RA NTR
    23 Aug 2006 RA Large amount of spurious HV TRIP alarms.
    22 Aug 2006 RA NTR
    17 Aug 2006 BJW Run 5b comes to an early end due to a fire in IR-4.
    16 Aug 2006 BJW NTR
    15 Aug 2006 BJW Nicholas added ROM occupancy plots for each of the 6 ROMs to the fast monitoring.
    14 Aug 2006 BJW The current in the bucking coil seems to really be falling according to Walt (that VSAM values rather than those from MCC are correct). Thus we have also started monitoring the B-Field inside the DIRC in case a corresponding drift appears there. So far none is observed.
    13 Aug 2006 BJW NTR
    12 Aug 2006 BJW NTR
    11 Aug 2006 BJW I would have been paged about high bin 11 (data truncation) by the swing DQM shifter, but it was me. Basically this was due to the high luminosity (11+). MCC was able to tune it down somewhat, getting it just under .1%, and since the data quality was unaffected (looking at the fast monitoring) Emmanuele said we could live with it.
    10 Aug 2006 BJW NTR
    9 Aug 2006 BJW NTR
    8 Aug 2006 BJW Paged around suppertime about the water in the tank on the DQM checklist being around 1 cm too high. Assured the pilot that we were aware of this and altered the short term instructions to reflect this knowledge.
    7 Aug06 RA NTR
    6 Aug06 RA NTR
    5 Aug06 RA NTR
    4 Aug06 RA NTR
    3 Aug06 RA There is a slightly strange smell when you open the door of the DIRC gas rack. Jerry is investigating.
    2 Aug06 RA NTR
    1 Aug06 RA NTR
    31 Jul06 RA NTR
    30 Jul06 RA NTR
    29 Jul06 RA A power glitch to the degasing pump panel caused two problems : The degasing pump went off, and the pressure monitor that reports the SOB water level went off. Epics thought the SOB had dumped. Jerry was able to restore the power and fix the problem. He requested that if this happens again, we should contact him, since he will then want to replace the pump.
    28 Jul06 RA NTR
    27 Jul06 RA Paged by IR-2 around 0500. Pilot reports 'DIRC daq problems'. Cleared up by itself in about fifteen minutes.
    Later : Chris traced the problem to a power glitch, which caused all the fans to stop and also locked up the DFBs. So when the pilot reconfigured, the DFBs did not communicate properly. On the second reconfiguration things worked.
    26 Jul06 RA NTR
    25 Jul06 JB    -Emmanuele contacted the personn responsible for the Chiller, so that we could try to cool more the Wiener crates (we have been already very close to red alarms for sectors 2 and 4).
       -short term instructions for DQM shifters have been updated to explain we know there is a noisy PMT in sector 6 causing high occupancies on several plots. We just ask to warn us if deadtime appears because of the DIRC
    24 Jul06 JB    We continued having high occupencies for sector 6 (but it comes and goes). We'll keep things like that until the generated deadtime remains fine(Jochen knows how to mask the faulty PMT in case of Emergency)..
       -A global calibration failed once, because of the DIRC. Jochen was informed of it.
    23 Jul06 JB    -Lots of yellow alarms due to inlet temperature of Wiener crates (very hot weather). The pilot masked these alarms (SWING shift) for sectors 2,4(threshold already changed up to 34 degrees),3,7,8,11 (where thresholds were still at 35 degrees).
       -During SWING shift, high occupancy seen in sector 6 (cosmics run 66949 ending at 15h39 fine; following processed cosmics run beginning at 17h54 showing problems). Problem not seen by DIRC crew before the day after (see next entry).
    22 Jul06 JB NTR
    21 Jul06 JB NTR
    20 Jul06 JB NTR
    19 Jul06 JB Jerry confirmed the high water level in the SOB tank is due to the hot weather.
    18 Jul06 JB    -Looked at the "inlet temperature"'s yellow alarm for Wiener crate in sector 2 (see yesterday's entry): the alarm's threshold at 33 degrees has been reached several times-> new threshold set at 34 degrees+unmask the alarm on ALH. Changed also corresponding thresholds for sector 4 and 5 as alarms appeared during DAY shift.. If it happens again, the pilot was instructed to mask the alarm and send an email (logbook entry).
       -found out there are 3 temperatures scanning to monitor crates temperature: inlet temperature(=air temp. at the entrance), ps temp.(=air temperature around the power supply), bin tem. (temp. of air after it cools the system). We expect: Tinlet < Tbin . This is true for DCH and EMC Wiener crates, but not for DIRC crates. Several tests where tried to understand:
          *Canbus adresses' comparaison between DIRC and other systems shows we didn't inverted the labels of inlet and bin temperatures.
          *studies of data during shutdown (FEE on, but no cooling by water chiller) shows cooling water is not responsable for the observation.
          *studies of the crate on the test band wasn't relevant (no differences between the 3 temperatures read directly on the crate), obviously because the crate wasn' t hot enough. Possible explanation: the 3 extra fan added on crates, and the position of temp. sensor could explain the observation. Maybe studies should be done during shutdown (comparaison between Epics readout and Wiener's crate screen output).
       -discovered that the water level in the SOB tank is slighly above the upper the max. level allowed (green bar taped on the tank). Contacted Jerry and Matt, but it's probably linked with hot weather and dilatation of water.
    17 Jul06 JB During DAY shift, there were recurrent yellow alarms with the tempreature of Wiener crate in sector #2. The pilot masked the alarm. This alarm began to appear on Friday 14th, but became more and more frequent (appearing around 6pm during hot days).
    16 Jul06 JB NTR
    15 Jul06 JB -High background spikes during OWL shift-around 5:30am- (seen by the pilot via high HV current in sector 10 and 11)
    -During UPS work, Matt disable/enabled the dump valve.
    14 Jul06 JB    -Matt moved the degasing pump back to the UPS system (was put on normal power when there were problems on the UPS) at 8am. Maybe it will partially fix the glitches problem with the readout of the degassing pump variable. However, as ths UPS is having trouble again, and will need some work tomorrow, Matts switch back to normal power at 3 pm.
       -Nicolas changed daily fan monitoring plots. We keep the usual scale, but there will be arrows for values that go above or under this scale.
    13 Jul06 JB Increased manually the N2 flux in sector 2 and 4 as they were around 15cc/min under their nominal values.
    12 Jul06 JB NTR
    11 Jul06 JB talking with Philippe about the DIRC analogic chips, for data acquisition, we found out that corresponding thresholds need to be set when changing a DFB. Apparently, no documentation exists about it... to be continued...
    10 Jul06 JB Jerry checked the situation with the degassing pump (we had glitches 3 times over the week-end). We will continue like that (cf entry of 07 July), knowing that the 52V readouts for the degassing pump in ambient is not a hardware problem of the degassing pump (probably a problem with the card transmitting values read). As we keep a lower alarm for this variable, we will still know if the degassing is having troubles.
    09 Jul06 JB NTR
    08 Jul06 JB NTR
    07 Jul06 JB    *Today it appears we don't see very well the front-end crates fans having a speed going to 0 because of the switch off of front-end crates yesterday. In fact it is probably due to the scale (on daily the point-averaged on 2 minutes- is going under the minimum valueof the plot; whereas we can see it on monthly plots, where values are averaged on 15 minutes).
       *We talked with Paul Stiles today to understand better what happened with the faulty boards we gave to him, coming from crate in slot 3 (2 removed after the power outage of June 4th, 1 on June 30th). 2 are already repaired (both used as board #3), 1 is still under fix. The main conclusions are: no common point between the 3 failures(especially for the 2 faulty boards from board 3); no reason to think the crate is responsable for the boards' failure. Consequently, we will keep the 'spare' crate in slot 3 untill the August shutdown, then Paul will investigate this crate.
       *Nicolas added long term instruction about the green alarms DRC:HV:UNDERVOLTAGE, OVERVOLTAGE, TRIP (see yesterday's entry).
       *New alarm "degasing pump off" (got an automatic page). This time (cf Monday 4th), it wasn't single glitches, but things look very simailar. Seing at ambiant, we got permanant value for degasing pump stucked 52 (usualy, it's arount 10, and the red alarm threshold is at 15). Went to IR2 to see the hardware: everything was fine. Obviously, with have a readout problem; Nicolas changed the red alarm threshold up to 60 (obviously the max value is 52), so that we won't have the alarm again. Jerry was paged
    06 Jul06 JB    *During DAY shift, around 11:15, ROMs crashes. The pilot followed long term instruction and restarted a new run without paging or sending email. Thanks to Matt, and logbook entries, we dicovered that in fact all 12 DIRC front-end crates went off at the same time for about 20 seconds; this is what created the ROMs crash. The reason why all Wiener crates went off remains not well understood. Maybe the contactor, which interrupts FEE crates power in case there is a dip or when the chiller has a problem (seen via the SIAM), interrupted power for a very short time (the crates are supposed to latch off, and then we have to manually turn back them on; maybe if it was short enough, the latching mechanisms or EPICS monitoring didn' t catch it). As we could expected the run 6366 associated with theROMs crash is showing problem on feature extraction (lots of entries in unusual FX status bins).
       * the pilot shifter send an email to know what means the green alarm labelled DRC:HV:UNDERVOLTAGE, OVERVOLTAGE, TRIP. As it is not the first time the pilot shifter have questions about it we will add instruction for pilot shifter (just for reminding: these alarms are due to a lose of connexion between epics and the HV crates; Nicolas preventing the Alh from showing 'white alarms' when epics doesn' t see crates, but the is no way to suppress the green alarms appearing when epics see the transition alarm->no alarm during the readout following a lose of communication with the crates).
    05 Jul06 JB    -Jerry disabled the SOB dump valve as there was maintenance work on the UPS system.
       -When I did the daily check tour at IR2 (aroun 4:15pm), I saw the "vacuum reads" of degasing pump was around 18Hg, whereas it should stay between 20 and 30Hg. In fact, it's due to some work Jerry did, and he forgot to close a little valve. Everything was fine again for the SWING DQM checklist at 8pm.
       -the run taking while the board #3 of sector 4 (crate in slot 3) was off have been DQG-ed OK by Eli and by the PID expert (see entry on 30 Jun06). So next time a similar problem occurs, the faulty board wouldn't need to be replaced immediately. This information will be added in the ops. manual.
    04 Jul06 JB I received 3 automatic emails from IR2 to warn that the degasing pump was off. In fact, it was 3 very close glitches in the redeout of the degasing pump variables (looking at ambient: 3 values around 51, when usual value is around 11). Nicolas added a new protection to prevent us from new future gliches. The pilot had to restart ALH to make it take into account the change.
    03 Jul06 JB Nicolas is officialy replacing Georges as DIRC co-system manager.
    02 Jul06 JB I was contacted (around 1 pm) by the pilot shifter because of frequent DIRC scalers yellow alarms followed by red alarms linked with integration of the time the scalers are above threshold. Obviously, it was just due to background (which kept increasing during the afternoon), so PEPII had to improve the quality of beams. The situation became normal again at 7pm (scalers always below yellow threshold).
    30 Jun06 JB    * I was paged at 8pm because of an HV board failure. It appeared that the board #3 of crate in slot #3 was dead -persistent "UNV"- (this board, powering the 8 outermost channels of sector 4 was already changed after UPS failure 0n 06/03). As we weren't sure the quality will remain absolutely fine (data not first processed by PC farm yet), we decided to change the board as soon as possible (the other solution was to wait for a beam lose, but: *the beams looked very stable; *we would have caused downtime anyway if we were called to intervene after a beam lose). The intervention generated 17 minutes of downtime.
        IMPORTANT NOTE: it appeared after processing that data were ok. Consequently, if this situation happens again, we know we can wait a beam lose before changing a board powering the outermost channels of a DIRC sector.
       *Bradley noticed there is a slight double hump on page 5 in the TDC hit per module for run66192. This is probably linked with the logbook entry at 07:30 noticing MCC was increasing lumi very quicly during the run -> superpostion of plots for high and low lumi during the same run.
    29 Jun06 JB
       *before UPS work, Jerry disabled (at 10am) the SOB dump valve, to prevent any problem from occuring (valve re-enabled at 3:45pm).
       *received an email from DQM saying there is one entry in bin 5 in accumulation plot of Feature Extraction. However, the problem is better known since June 6th: there is a bug in the plot which always existed but wasn' t seen as it was starting plotting when you had at least 2 entries in any bin. However, there is still explanation for the existence of one entry in bin #5 ('TcUnderflow'). Jochen will look into it when Niels will be back.
    28 Jun06 JB NTR
    27 Jun06 JB NTR
    26 Jun06 RA UPS replacement went forward at 10:00. Two problems occurred : First, DRC-MON did not reboot properly; Ray traced this to the fan crate of the VME, which we replaced. We also had the same problem as last time we lost a UPS, with the gas system not reporting properly; GMB status showed a red zero. Matt fixed this with a reboot of the GMB, making sure the dump valve was disabled first.
    25 Jun06 RA NTR
    24 Jun06 RA IR-2 paged me because bin 11 in FX plot was too high. Not a DIRC problem, PEP was having difficulty injecting - very short run, only 44 seconds.
    23 Jun06 RA The flow in the DIRC water chiller has been around 4.4 all week; nominal is 4.8 +- 0.7. Water level seems normal; not clear what is causing this. Will keep an eye on it. Nicolas implemented Jerry's new formula.
    Discovered that fan monitoring plots were not being updated properly; it seems the commissioners have failed to look at the dates on the x axis. The 'monthly' plots were showing April... Nicolas fixed the scripts. Lesson : Do not just look at the time development, also check what time the plots are for!
    22 Jun06 RA Jerry believes the discrepancy yesterday is due to offset drift, ie a change in the analog-to-digital conversion (I think?) and has suggested a change in the conversion formula.
    21 Jun06 RA Nicolas and Jerry between them discovered a discrepancy between the Honeywell box temperature, and the epics variable. Investigation continues.
    20 Jun06 RA Nicolas adjusted the scaler-alarm histograms in the daily background monitoring, to take into account the new alarm values. The difference is very clear, percentage of shift in yellow alarm goes from 10 (Swing 19th) to 2 (Owl 20th).
    19 Jun06 RA We now understand the FastMon problem described for the 16th; this happens when a run ends in beamloss.
    18 Jun06 RA Pilot paged me with red alarms in FEE, proxy not runnable. Problem seems to have been related to missing timing information from PEP; cleared itself up after a while. No data time lost as we were taking cosmics.
    17 Jun06 RA NTR
    16 Jun06 RA FastMon for run 65740 shows a strange bump near 0 for the TDC-hits-per-module plot. We have had this once before, the 26th of April. Both then and now, the strange run was followed by a ROM crash; this may be coincidence. Nicolas is talking to Chris to understand this better.
    The SOB degassing pump is reporting intermittent yellow alarms; the flow is dropping to 6.6 or so for one data point, then recovering. We emailed Jerry and Matt, and will monitor; right now we think it's spurious.
    15 Jun06 RA Global calibration got stuck; Nicolas fixed it.
    14 Jun06 EL Rolf is taking over for the commissioning.
    13 Jun06 EL NTR
    12 Jun06 EL NTR
    11 Jun06 EL NTR
    10 Jun06 EL Pilot phoned for alarm on scalers and small scaler spike fraction. I told him to call MCC and to tell them to tune the beams. Then he phoned me for a yellow alarm on the degasing pump SIAM. It was fake, and restarting the ALH cleared the problem. I asked the DQM to cross check that everything was fine on the water plant. He did not notice anything special.
    9 Jun06 EL NTR
    8 Jun06 EL NTR
    7 Jun06 EL Emmanuele gave the monthly report at the Operations meeting. Bill wants us to clarify the situation on Online at the Collaboration Meeting.
    6 Jun06 EL NTR
    5 Jun06 EL Brought the 2 HV boards that died on saturday during the power outage to Paul Stiles.
    4 Jun06 EL At 5:am, I check the calibration taken after the power outage, at 2:45am. The plots 1 & 2 looked strange in the bottom of sector 4, and the other plots were fine. I thought it was just due to the fact we had changed the boards. I sent an email to Nicolas and Emmanuele to let them know, and phoned IR2 to ask to take another calibration when the opportunity arises, but without interfering with data taking. I tried to check the Fast Mon plots, but although some runs had data, no event had been processed: due to the outage, the post-processing did not restart. I sent an email to Steffen Luitz and to Jim Hamilton asking them to fix it. In the morning, the DQM sent an email mentioning strange features in sector 4. It appeared that the problem was due to the new boards: after changing them, we should have reset the value of the voltages before putting back the DIRC into the partition. Nicolas called Emmanuele who went to IR2 to fix this. Then Katherine called me because the pilot made a confusion between me and Emmanuele, and she could not understand what was happening. Emmanuele phoned me to explain me what the problem was. I went to IR2 to find Emmanuele who was setting those voltage values. Data for the runs that were taken with the bad voltage values will have to be checked to see if it is usable or not.
    3 Jun06 EL Called by the pilot at 3:00pm: red alarms on the DIRC scalers. It was just due to background, I told him to call MCC to tune the beams. Called by the pilot at 4:30 for a major computing outage. Request that all subsystems come to IR2 because all the racks are down. Arriving at IR2, the gas system, the FEE, the degasing pump and the HV were off. The chiller and the rest of the water system was on. We paged Jerry and Matt by phone, because nothing of computing was working: contact database, Epics, logbook were not accessible. Matt came to disable the dump valve. It appeared that the outage was not a computing problem but a trip of the UPS system. All the systems on UPS are down. The UPS sytem was bypassed, and then subsystems started to recover. First power came back in the EH, and then on the gas+water system. Jerry came to IR2, checked the status of the gas system and enabled the valve as everything was OK. The degasing pump came back by itself. When computing recovered Epics, we ramped up the FEE and the HV. But one board of crate 3 tripped, and one was red and never ramped up. We changed both, and put the broken at the second floor, waiting monday to bring them to Paul Stiles. Then we tried to take a standalone run, but it was not working. We realized that as we had to ramp up the FEE, then the ROMs had to be rebooted. So we did this, and took the standalone run, which was fine. After, we checked the data on the cosmic run that was on-going, and it looked fine. We left IR2.
    2 Jun06 EL NTR
    1 Jun06 EL NTR
    31 May06 EL NTR
    30 May06 EL NTR
    29 May06 EL Paged at 4:30 pm for a SOB water leak alarm. In Epics, all the SIAMs of the gas system were white. The IOC and VME crate panel were mostly white, with voltages red and at 0 value. When in IR2, I looked at the gas rack. Indeed, all the SIAMs were off. The VME crate had a message Fan failure on the display. Jerry was paged but he was in Prague. We paged Matt, who told us to take no action until he arrives and disable the dump valve. When he got there and inhibited the dump, he removed the VME crate, and changed the fans. We had some trouble to find the spare fans. He had to take some that belonged to a crate from Jerry. Matt fond some fans with machanical defects (they could translate along their axis), and also a board slot of plastic that may have obstucted the path of the fans. Once the fans were replaced, he put the crate back, reconnected the canbus and everything in the Alarm Handler and in Epics was back to normal. We waited a bit before enabling the dump valve again, so as to be sure we had reached a steady state, and that everything was really back to normal. Then Matt enabled the valve, about 3 hours after the page. In the meanwhile, PEP had troubles with power supplies and was not able to deliver, so although it took us three hours to recover, we did not cause any downtime.
    28 May06 EL NTR
    27 May06 EL NTR
    26 May06 EL NTR
    25 May06 JB Walt changed the scale for the DIRC occupancy in the Epics pannel.
    Emmanuele updated the pilot short term instructions according to the new thresholds we set yesterday.
    Nicolas Saved changes in the CVS repository of the drc-mon package the new set of thresholds.
    I was paged because of an entry of the "feature extraction" plots for DQM: there were entries in bin number 5 for frequency plot. However, there is no entry in the accumulation plot... The best explanation is that it is an artefact.
    24 May06 JB    * During the operation meeting, Walt pointed out we should have red alarm thresholds for the scalers that correspond to the threshold above which we could have induced problems; we should monitor backgound just with yellow alarms. He also noticed that the item illustrating the occupancy of the DIRC, in the Background section of Epics, was always full and proposed to changed it. It's linked with the fact that the scale is not relevant anymore.
       * After having talked with Joe, we found out it was normal that the indicator in this pannel for the occupancy increased, and that it would be fine to let Walt change the scale from 10% to 20% (it seemed the current occupancy value was around 13).We also changed the red alarms thresholds for all sectors: they are all at 600kHz now. We saved the new setting in the proper epics configuration file to prevent this to be erased when the IOC gets rebooted.
       *There were some discussion inside DIRC team (especially with Jerry, Nicolas and Joe), about how to deal with hig rated seen by the DIRC (Jerry talked about erecting more shielding).
    23 May06 JB We had alarms today again, linked with high rates in DIRC scalers. Yellow alarms theresholds for all sectors were raisen from 300kHz to 400kHz, except for channel 3 and 4 (both at 450kHz). Consequently most of yellow alarms have the same threshold as the red ones now (400kHz), exept sectors 3 and 4 with red alarm at 500kHz. We plan to put red alarm thresholds 100kHz above the yellow ones when the situation is stabilized, and plan just to monitor background with the new settings. The pilot short term instructions has also been updated so that the pilot will be aware he may have red alarms linked this DIRC scalers, and he should just send an email to babardrc.
    22 May06 JB PEP2 is back to high luminosity. Consequently, the background is higher too. We had red alarms for sector 3 due to "smallspikesfractions". Emmanule rises the red alarm threshold for sector 3 (from 300kHz to 450kHz), hoping this will prevent us from having too much new alarms. According to Joe there won' t be problems with the DIRC hardware (PMTs) or PID untill we keep steady rates under 600kHz, maybe even a little more (the only thing to fear is to have big background spikes; above 600kHz).
    21 May06 JB NTR
    20 May06 JB NTR
    19 May06 JB Nicolas changed the defaut mask for the SOB water level so that the problem with Alhdiff should hopefuly be fixed.
    18 May06 JB NTR
    17 May06 JB We will have to change the DIRC cellphone today. Nicolas is leaving for France.During the owl shift Janis worried about a difference for the SOB water level (DRC:WTR:VSM:CALC0), displayed when using Alhdiff. However, this is a known problem, linked with a bug in Epics (there are long terms instructions for pilot already). Jochen is continuing working and training on the online part of the DIRC.
    16 May06 JB Jochen worked on DIRC calibration the whole day (between 9am and 8pm). we had to ramped up manually DIRC HV at the begining on the morning to let Jochen work (we are in a ROD, so all subdetectors were at V0).
    15 May06 JB ADC calibration status report from Nicolas and Jochen: after a successful calibration, an hbook file has been produced +a postcript file. However, there are troubles to understand it. During the checktoor, the flow of N2 in sector 2 was found at 120cc/min instead of 100cc/min; it was set back to the nominal value. No change appeared in Ambiant. A new section was added in the ops. manual to know how to recover the DIRC after a power outage.
    14 May06 JB Problem on cooler water main line's temperature during swing shift, which generated a power outage in the Electronic Hut. on UPS.Matt came in to bypass the SOB dump valve while we were turning on drc-mon -- just in case... After two reboots of drc-mon and one of drc-hv the DIRC came back healthy and seems not to have suffered from the power cut. FM plots + global calibration data look fine.
    13 May06 JB NTR
    12 May06 JB Nicolas added guidances on how bypassing the injectable flag and the ORC status (I had to do it in the trouble of HV crate we had yesterday, and there was no documentation yet).
    11 May06 JB during day shift, HV crate #3 generated white alarm. Changing the 2 fuses of this crate fixed the problem. Runs 64688,64690 and 6492 seem to have problems ahs they have strange feature for arrival time. It is due to backgroung and it appears that after L3 trigger, these problem disappear and the data seem ok. Consequently, nothing to worry about...
    10 May06 JB during day shift, pilot asked questions about some variables hosted by the DIRC (red alarms). We found out that it was in fact one of the Jerry's variable. Nicolas added a link in ops manual pointing to the list of Jerry's variable, and also one for water system variables.
    09 May06 JB Background decreased since 04 May06 -> threshold of alarms related to background set back to what it was before this day.
    08 May06 JB NTR
    07 May06 JB NTR
    06 May06 JB Jochen did a long DIRC Calibration for Marco.
    05 May06 JB Matt worked on the water system this morning, with the assistance of Emmanuele et Nicolas. Jochen did online training for some hours this afternoon.
    04 May06 JB The "telalert database" has been updated (not as easy as expected; the system should be improve to make further change of pager easier). Jochen finished studying DIRC calibration. High background in IR2 when taking data (due to very short scrubbing after downtime: 20 minutes) -> Nicolas changed alarms thresholds (scaler rate in sector 3; scaler rate ratios; neutron counters)
    03 May06 JB Jochen studied DIRC calibration (tried several ADC-Cycles files today and will continue tomorrow moring). After Jochen was done for the day, Nicolas rebooted drc-hv (in the future, Matthias' s change -equivalent to suppressing the alarm- should prevent the drc-hv CPU alarm from triggering spurious beeps in the ALH). Nicolas added pages in the ops. manual about DIRC testband and finisar issues. It appears the new pager is not reachable via telalert -i babardrc -m "..."; it's still linked the old pager we changed because of spamming trouble.
    03 May06 RA Nicolas added an alarm for the circulating pump. It will tested before the ops meeting. Also our spurious CPU alarm is back; it will be removed.
    02 May06 RA We (NA, ES, EL, and RA) replaced the DCC board in Sector 5 as planned; this solved the problem.
    01 May 06 RA Finistar module in sector 5 does not connect after a DRC-MON reboot. We have attempted to reboot, move cables around, and put in older versions of IOC software. We believe that the problem lies with the FEE. Fortunately we are in ROD, so we will open the doors tomorrow morning and replace the module.
    30 Apr06 RA NTR
    29 Apr06 RA NTR
    28 Apr06 RA NTR
    27 Apr06 RA NTR
    26 Apr06 RA NTR
    25 Apr06 NA Summary of Dominique Breton & Christophe Beigbeder work on the DFBs:
       * faulty DFBs: #6A, #71, #0D, #7E have been repaired (should be good spares). No problem found on DFB #50 ⇒ should be kept as 2nd order spare and tested again in the future. The problem on DFB #57 was seen but not identified ⇒ shipped back to France for more tests.
       * spare DFBs: all were tested OK apart DFB #78 which was fixed.
       * Hardware: everything has been stored again in the DIRC cabinet, in particular the crucial boards used to test the DFBs. The Macintosh used for the tests is getting older ⇒ a spare one may be sent from Orsay.
       * Problem: ageing effects have been identified on some capacitors. The effect is to move upwards some channel thresholds ⇒ reduction of the number of photons collected. Could this explain (part of) the decrease of the number of associated photons per track? Hard to say: random effect (seen only on few boards); may depend on the operating conditions of the board (thresholds different when the board is just turned ON); no idea on the status of the boards in the detector. To be discussed (in particular when I'm in Orsay end of next month).
    24 Apr06 RA FastMon processing for run 62164 ran twice, which caused a problem with the file. Nicolas will work on this.
    23 Apr06 RA NTR
    22 Apr06 RA NTR
    21 Apr06 RA Matt McCulloch did maintenance work on the water system.
    20 Apr06 RA NTR
    19 Apr06 EL NTR
    18 Apr06 EL NTR
    17 Apr06 EL Run 63954 and 63955 have huge background. It can be seen in the fast monitoring plots.
    16 Apr06 EL NTR
    15 Apr06 EL NTR
    14 Apr06 EL NTR
    13 Apr06 EL Increased the scalers red alarm threshold of sector 3 (Changed from 300000 to 350000) due to poor background conditions (scaler ratio above 1.5). We will have to change it again when background is reduced.
    12 Apr06 EL NTR
    11 Apr06 EL After the new patches installed last night by Steffen, the post-processing of the Fast monitoring did not restart. I only noticed it today. We sent an email to Steffen and Jim: Steffen fixed it immediatly. The fast monitoring is back, and the plots are fine.
    10 Apr06 EL NTR
    09 Apr06 EL Mike Kelsey sent an email in detector HN to know what was this problem of mask alarm when doing the ALHdif, concerning the SOB water level. Nicolas replied to his mail.
    08 Apr06 EL NTR. Received just one page from IR2 informing me that the magnets were about to be ramped up.
    07 Apr06 EL NTR
    06 Apr06 EL Global calibration taken while DIRC was in RED status, as the magnets were down but the bucking coil was still on. The constants loaded after this calibration were wrong. We took another calibration to check that the calibration was bad because of the presence of the magnetic fiels: these two calibrations were consistent. Then we took two successive calibrations just before data taking (and after magnets were ramped up) to have the proper constants loaded in the database.
    05 Apr06 EL NTR. Emmanuel takes over.
    04 Apr06 ES Nicolas rebooted HV-IOC because there was a wrong synchronization due to the reboot of almost all the IOCs made by Matthias two days ago.
    03 Apr06 ES Nicolas set higher scaler rate ratio' threshold to avoid alarms due to high background. He set the DQM reference plots back to collisions. Emmanuel and I went to pick up the spare HV crate in Paul Styles' workshop.
    02 Apr06 ES NTR
    01 Apr06 ES The standalone calibrations work well now (work done by Jochen and Niels)
    31 Mar06 ES NTR
    30 Mar06 ES NTR
    29 Mar06 ES NTR
    28 Mar06 ES NTR
    27 Mar06 ES NTR
    26 Mar06 ES NTR
    25 Mar06 ES NTR
    24 Mar06 ES NTR
    23 Mar06 ES Nicolas turned back on the HV crates and, surprise! one of them failed. According to the Ops. man. it is likely to be a failure of the internal 5V power supply, therefore we replace it with the spare we have in IR-2. The Ops. man is very detailed and well written, but it misses one last step, which is to set the caenet number to the current crate once we turn it on, otherwise we can't control it via EPICS. After some time, we figure it out and we double check calling Philippe. Now it is working and the broken power supply is in Paul Styles' office.
    22 Mar06 ES Turned off HV crates for the VESDA test tomorrow morning
    21 Mar06 JB NTR
    20 Mar06 JB NTR
    17 Mar06 JB NTR
    16 Mar06 JB Ray found a VME power supply for the testband in IR2 (missing since at least 9 Mar 06, when the disparition was discovered).
    15 Mar06 JB Jochen was added on the whiteboard to be warned when there is a downtime period, so that he could try to fix the troubles that appeared yesterday(failing standalone calibration).
    14 Mar06 JB *The anomaly with N2 flow in sector 10 and 0 disappeared today (and the humidity is still ok)-stable situation at midnight-: the flows in the 2 sectors remain stable at 197cc/min (1 year ago, we had 195cc/min in sector 10, 197cc/min in sector 0- so, we are close to the former configuration). However, what happened is still not clear (no answer to a sent email).
    *Jochen Kaminski worked on the DIRC while the LINAC was down. The standalone calibration he tried to do failed (obviously, it's due to changes since the October shutdown as it worked at that time). It is planned to try to fix it the next downtime period.
    *During the IR2 daily tour, the flow in sector #2 was found a little low (185cc/min instead of (100+/-25)cc/min), so we increased it by hand to reach the nominal value (humidity didn't change with this new setting).
    13 Mar06 JB *This morning Minor Alarm with N2 flow at the exit of barboxes: in sector 0 and 10 the flow went under the threshold of 190cc/min (189.9cc/min); However, humidity remains stable and at the expected level. The lower threshold was moved to 185cc/min, but we'll keep an eye on that channel.
    *In the evening, the flow in sector 0 and 10 continues to go down, reaching the new threshold whereas humidity remains stable. We move the threshold to 180cc/min to prevent new alarms, but we'll keep an eye on the channel.
    09 Mar06 JB Minor change in Epics: new parameterization of scaler rate (use of new set of coefficients). The rate will be closer to 1.
    08 Mar06 JB I (Jerome) take over. A VME power supply has disappeared from testband in IR2. Ray trys to find another one.
    01 Mar06 EL NTR
    28 Feb06 EL 2 emails received around 11:pm. There was a problem with the fast monitoring post processing on two runs. It was due to the Oracle outage.
    27 Feb06 EL NTR
    26 Feb06 EL NTR
    25 Feb06 EL Calibration problems went away by themselves.
    24 Feb06 EL New JAS plot are available for the DQM. They now display references.
    Yellow alarm on the SOB water level. It disappered very quickly, so I was not paged.
    Paged around 10:00 pm for a problem of global calibration. The DIRC completes the calibration but when validating the results, it gets stucked. Another calibration was performed. The result is the same. As a calibrtion has been taken earlier in the day, and since there is no beam, we decide with Nicolas to let it like that until tomorrow.
    23 Feb06 EL Paged at 8:10 am for the problem of the ALH on DRC::WTR:VSM:CALC0 which has a T while it should not. This happens when the pilot makes his ALH dif. Told the pilot that it was known and under investigation by Matthias. We do not know if it appeared earlier this night as other pilot do not seem to have completed the ALH dif.
    22 Feb06 EL 1:00 am Turned HV OFF with EPICS and physically to allow EMC people to work on electronics. Matt put the covers on the PMT. Nicolas disabled the alarms for the HV and for the scalers in the ALH
    21 Feb06 EL NTR
    20 Feb06 EL NTR
    19 Feb06 EL NTR
    18 Feb06 EL NTR
    17 Feb06 EL NTR
    16 Feb06 EL NTR
    15 Feb06 EL NTR
    14 Feb06 EL NTR (except spam again at 7:20 am)
    13 Feb06 EL NTR
    12 Feb06 EL New SPAM page at 6:30 am
    11 Feb06 EL NTR
    10 Feb06 EL First page for me, around 9am, while on shift: it was SPAM!!!Trying to sell sofware.
    9 Feb06 EL NTR
    8 Feb06 EL NTR
    * Feb06 RA Emmanuel takes over!
    7 Feb06 RA Paged at 0200 : Fake SOB dump. Matt went in and checked water levels, which were fine. Nicolas and Emmanuel traced the alarm to a bit-flipped readout on the VSAM channel that generates this alarm.
    6 Feb06 RA NTR
    5 Feb06 RA NTR
    4 Feb06 RA NTR
    3 Feb06 RA NTR
    2 Feb06 RA Paged at 0620 : drc-hv lost contact, white alarm. Called Nicolas, who called Matthias, who rebooted the IOC, which fixed the problem.
    1 Feb06 RA NTR
    31 Jan06 RA NTR
    30 Jan06 RA NTR
    29 Jan06 RA NTR
    28 Jan06 RA Paged about drc-mon yellow alarm, 86% CPU use. Told the pilot to reboot it; this was a mistake, one should not reboot for yellow alarms, and also Matthias should be told. Lost about 6 minutes of data. Correct procedure would have been to call Nicolas.
    27 Jan06 RA NTR.
    26 Jan06 RA Got a page about the SOB water level; Matt was working on it, says it is taken care of.
    25 Jan06 RA Updated DrcCommissioner to release 18.6.3
    24 Jan06 RA NTR
    23 Jan06 RA NTR
    22 Jan06 RA NTR
    21 Jan06 RA NTR
    20 Jan06 RA NTR
    19 Jan06 RA NA trained RA and EL in the procedure for switching to the backup chiller. After this we un-upgraded the IOC, as the new RTEM board was having some difficulty communicating in some channels -> white alarms.
    Shifter paged me about a very noisy PMT in sector 9 -> I panicked briefly thinking we had a Christmas tree, forgetting about short-term instructions! It is the usual culprit, showing up quite a lot since we are taking cosmics.
    18 Jan06 RA Replaced the DFB responsible for the 8 inefficient PMTs in sector 5. Very difficult access! The PMTs came back as we expected and now seem to be quite normal. 0.1% increase in DIRC efficiency -> much rejoicing.
    17 Jan06 RA Accidental dump of the SOB water while replacing an IOC. Fortunately no major damage was done. However, we have learned always to physically disable the dump when dealing with something that can communicate with the dump logic. 'Communicate' in this case means 'has a wire connected to', there is no need for a logical communication. Transient voltages can get into all kinds of places.
    16 Jan06 RA Our timing shift is back in run 63326, among others. It is a small shift but across the board -> not random.
    15 Jan06 RA NTR
    14 Jan06 RA NTR
    13 Jan06 RA Saw an odd timing shift by a few nanoseconds in run 61222. It disappears in the next run.
    12 Jan06 RA NTR
    11 Jan06 NA Matthias used a 2-hour downtime opportunity to reboot drc-hv after he fixed the ntp client.
    10 Jan06 NA Matthias installed permanently the ntp client on drc-hv (see 04 Jan06 entry) but it isn't working well (i.e. synchronizing the IOC clock) so far. I also updated DrcCommissioner as a new online release was tested (installed and then backed out!) today in IR2.
    I've also cleaned the C&F Manual (adding Emmanuel's details; removing Kalanand).
    09 Jan06 NA drc-hv lost connection with the HV crates for about 1 minute during the day shift. Several white alarm messages showed up in the ALH and in the CMLOG before things came back to normal when the reading process went in timeout.
    I discovered that the normalization factor used to estimate the DIRC occupancy from the FM data was off by a factor 6. Matt fixed this in epics.
    I've also modified the DIRC archiver setup to have 4 new epics variables stored in Ambient: DRC::HV:UNDERVOLTAGE/OVERVOLTAGE/TRIP/NC.
    06 Jan06 NA Got an e-mail from the DQM complaining (not too loudly) about old-fashioned timing references. That's on Marc's todo list and on mine as well.
    04 Jan06 NA We couldn't do any of the activities we planned for the ROD: to access the sector 5 front-end crate, one also needs BE opened by ~1 meter which wasn't possible due to time constraints (the 3 other doors were opened and had to be closed by 7 pm); the new connector designed by Gibson to make nice the connections between the CANBUS lines and drc-mon was not complete.
    On the other hand, Matthias rebooted a few times drc-hv because he installed an additional software to keep the IOC synchronized (a 30 seconds delay appeared on 12/29/2005 and was catched by Andy Salnikov). This software is not installed by default on RTEMS.
    Monthly Report
    03 Jan06 NA Webpage cleaned and setup for 2006! Happy New Year!
    30 Dec05 NA Dummy DQM page.
    12-29 Dec05 KM NTR.
    11 Dec05 RA Nothing to report.
    10 Dec05 RA Very large spikes in scaler rate ratios. Seems to be due to general high background level, rather than a DIRC-specific problem. Will keep an eye on it.
    09 Dec05 RA Nothing to report.
    08 Dec05 RA Nothing to report.
    07 Dec05 RA Nothing to report.
    06 Dec05 RA Nothing to report.
    05 Dec05 RA Nothing to report.
    04 Dec05 RA Nothing to report.
    03 Dec05 RA Nothing to report.
    02 Dec05 RA Nothing to report.
    01 Dec05 RA Nothing to report.
    30 Nov05 RA Nothing to report.
    29 Nov05 RA Shift log reports that a global calibration failed, with all subsystems hanging at the validation stage. But DIRC calibration plots look ok, and have the correct time-stamp. Not sure what causes the discrepancy. (Later) Talked to Matt Weaver; apparently this was an issue with a brief network outage, causing the jobs to appear to hang, even though the files were actually written.
    28 Nov05 RA Nothing to report.
    22 Nov05 NA Raised the alarm levels of the 3rd neutron counter hit ratio: HIGH 2.0 -> 4.0; HIHI 3.0 -> 5.0. This alarm was beeping continuously.
    21 Nov05 NA + KM We added few new DRC HV variables ( DRC::HV:UNDERVOLTAGE, DRC::HV:OVERVOLTAGE, DRC::HV:TRIP, DRC:HV:NC and DRC::HV:PS:STAT) in the ALH (DRC -> HV -> General). They should monitor the ramping features seen during the weekend. If the problem shows up again we will try to investigate more (debug added in the state machine as well).
    20 Nov05 KM + NA We tried to make V0 <-> V1 transitions a few times and all the boards behaved properly. So the problem doesn't occur everytime one goes from V0 to V1. Nicolas and Matthias tried to debug the problem but couldn't reproduce it. We will investigate it further.
    19 Nov05 KM + NA Got paged by swing shifter "DRC_SC_4 HVPS alarm". The red alarm lasted only for a few seconds. It turns out that one HV board (#1) in sector 10 remained at voltage V0 even after DIRC was ramped-down by the pilot. The epics status showed all the sectors at V1 though. AT that time, DIRC was in automatic mode and in injectable state. All other boards were at V1 (= V0 - 300 V) as expected.We are investigating the cause behind this.
    11 Nov05 NA Added Rolf details in the C&F. Reboot drc-soft (nominal scaler currents tuned).
    16 Nov05 NA+Jerry+Mat Today, Matt, Jerry and Nicolas did the SOB water level stability test. They Ramped down the HV and turned off the circulating pump at 8:32 am. The water level was around 5.38 meter at that time. After an initial drop of ~3-4 cm, the water went down slowly at a rate of ~8 cm/h and crossed the yellow alarm level at 9:21 am. At that point, the decrease rate was slowly going down. Around 10 am it increased again before reaching a plateau around 10:20 am. The water level stayed stable there at ~5.22 cm, i.e. two centimeters (!) above the red alarm. The variation in the level decrease rate was attributed to the geometry of the SOB. In this hypothesis, the plateau should correspond to the moment when the water reached the top of the main SOB volume -- in that area, one needs to drop a large volume of water to see a change in the water level. Matt measured several times the water level in the SOB tank and found that this level was increasing at roughly 2 liters/hour. To understand the origin of the leak (somewhat unexpected), he moved a couple of valves (one of these action triggered the automated page Kalanand got around noon) and found that the leak was coming from the PHmeter area. Finally he turned the circulation pump ON again and after a few minutes (the time for parameters -- water temperature, resistivity, PH -- to come back to nominal values) he made the water circulate again. The recovery was almost immediate. The test was quite successful: we now know better the behaviour of the water level with the circulating pump off.
    10 Nov05 NA+MMcC Nice label glued on the water SOB tank to show shifters the nominal range of water. Instructions are to page DIRC if the water gets above or below this range.
    08 Nov05 NA Got an e-mail from last swing shift as he found that the water level in the SOB tank was slightly above normal level. Matt checked this morning and found nothing worrying -- he said that variations in the tank level can occur after the doors get closed/opened or when the SOB filling procedure is running (~ 10 minutes / day in total).
    Rebooted drc-hv to load the IOC monitoring code prepared by Matthias on RTEMS.
    Modified few variables in drc-mon: decrease scaler rate low/lolo alarms (set too high for cosmics in July); make smarter the 'HV in local mode' alarm: it will show up only if BaBar is in automatic mode and DIRC in local mode -- and if we're not taking any calibration.
    07 Nov05 NA Several calibrations were taken this morning and they all were OK. I thought the problem had come back yesterday morning but in fact it turns out that there was a problem with the Odf timing -- shifted backward by ~ 4 days! So the calibration taken yesterday morning was compared with an old one which had the problem => the comparison was obviously bad.
    04 Nov05 DRC crew We worked the whole day on the calibration problem. It was finally tracked down to a loose connection between the sector 10 Front-End and the light generator crate. Unplugging and plugging again the cable on the crate side fixed the problem -- fortunately because the doors were closed!
    03 Nov05 NA Cosmics data taking restarted this morning after the DCH got filled. DIRC HV were turned ON (after tags were put on the plastic covers to let people working in the hall know about it) to check that the DIRC was fine. Cosmics data looked OK but there was a problem with the calibration in sector 10 -- see the posting. Investigation started immediately atfer the problem was discovered -- Matt gave me an hand until the late afternoon.
    02 Nov05 NA+KM Tested the GMB with the help of Jong and Vitaly (from EMC): it was put in the place of the GMB used to monitor the EMC water chiller but didn't work. Jong thinks that some components are missing on the board... Once he would have fixed the problems he currently has on an EMC GMB, it would bring it with ours to Mark Freytag so that he can check the architecture of the board.
    drc-hv crashed (at the same time that all other IOCs running on RTEMS!) in the afternoon due to some network problem (low level RTEMS code). Steffen and Matthias understood what happened and they think they have solved the problem.
    Rebooted drc-mon several times to have the new fan speed alarms working (one per sector, combining the 6 speeds monitored in epics).
    31 Oct05 NA Received the DCC for trigger. Changed DCC in sector #4 (DCC #5, in sector #10 at the beginning of the shutdown) as the readout flipped again. Put back the DCC which was originally in this sector. E-mailed Antoine to let him know the problem was confirmed on this DCC.
    31 Oct05 NA + Ray Test on fan tray in sector #6 -- see this hypernews for details.
    27 Oct05 NA Strange behavior of HV channel #37 in sector #5: the two first time I ramped to V0, the channel didn't keep the voltage but the third attempt was successful and the channel remained stable in the next couple of hours. Will keep an eye on it...
    26 Oct05 NA drc-hv is now running on RTEMS! The last problem was related to the channel access list -- by default RTEMS couldn't change the SIAM status whereas it was working with VxWorks. This was fixed by Matthias and Steffen.
    There was a VESDA alarm in the hall which cut the power off for the chillers and in the E.H. When things came back to normal, one VSAM had died in drc-mon. Changing it was a bit tricky -- one needs to make sure that the cables are all well-plugged in the VSAMs: the connections are a bit loose => working on one VSAM is likely to disconnect the other ones! I paged Matt while I was trying to fix the VSAM as I was getting stupid monitoring values on the DIRC water epics pannel. I fixed the problem just when Matt was arriving.
    24 Oct05 RA+NA+KM DCCs tested: 2 found bad (#21 and #31) => will be sent back to France for tests as one component needs to be changed on each of them to improve their reliability.
    22 Oct05 NA Problem with drc-hv (RTEMS-based board let connected yesterday after the tests): the IOC had a temporary (!?) glitch of unknown origin. Although the crates were physically off, the IOC thought that all channels tripped which generated 56*12 automated messages and... the associated pages! Fortunately, only few of them reached the pager... I renamed the script paging automatically in case of HV trip to avoid similar behaviours in the next days and e-mailed Matthias to let him know about this weird feature.
    21 Oct05 NA + MW Kept on working on the drc-hv ugrade (VxWorks => RTEMS) with Matthias. Communication problems between the IOC and the crates / boards have been fixed and tests are promising (booting and ramping up times significantly decreased) but one cannot talk reliabily with the tripped SIAM which is inhibited (V1) or not (V0). More tests next week.
    20 Oct05 NA + CT Panel 'background' added in epics: it contains the 'scalers' panel _plus_ some other panels displaying different background monitoring variables. The layout of these panels has also been improved: the 12 monitors are now drawn in a way which mimic the view of the DIRC from the backward side of BaBar.
    20 Oct05 NA + MW 2nd set of tests of the new drc-hv board. Yesterday's problems have been solved but new ones appeared: the board is not communicating properly with the CAEN hardware. Matthias will try to fix them.
    20 Oct05 NA + MMcC The readout of the N2 flows in epics was crazy since the end of yesterday's network outage (induced by the UPS power outage). Power-cycling the GMB board in Rack 05 (top of the detector) fixed the problem.
    19 Oct05 NA + MW Started testing a new IOC board for drc-hv. It is running on RTEMS instead of VxWorks => the CAEN driver must be modified to support this change. The new board is running at 1 GHz instead of 66 MHz which should fasten the booting and ramping phases. The first test shows problem at the initial phase of booting. More work expected on tomorrow.
    19 Oct05 Ops. crew We finally got the bridge this morning. We removed the last 4 fan trays from the detector. After the fans were changed, 3 of them (sectors #10, #0 and #1) were put back in whereas another tray was put in sector #11: Matt replaced the fans of this crate (taken out from sector #2 in June) few weeks ago to train himself. We also replaced the DCC in sector #10 whose readout was flaky since the door was opened (DCC #5 out, DCC #15 in).
    12 Oct05 NA+KM+RA We unplugged the Xmas trees during the swing shift. The 2 candidate PMTs were indeed the faulty ones: {sector #3, channel #29, row #22, column #9} and {sector #5, channel #6, row #7, column #14}. We also tested the 4 HV boards we recently got from Paul Stiles: boards 00315, 00319, 00324 and 00327. As they behaved in a satisfactory way we let them in the HV crates.
    11 Oct05 Ops. crew The 8 fan trays are back in the DIRC with new fans.
    10 Oct05 NA+MMcC Matt and I have pulled out 8 / 12 fan trays (sectors 2 -> 9). The 4 remaining ones require the bridge.
    10 Oct05 NA 3rd DCC (number to be checked) given to Selina as she wasn't successful with the two first ones. She had again the same problems => I've e-mailed Philippe & Antoine to get advices as I'm pretty sure that the problem is not the boards themselves.
    10 Oct05 NA DCC #15 given to Selina for tests -- she also has DCC #31 which seems not to be working. HV physically turned off (crate key in my pocket) to allow a safe opening of the BW doors.
    Note: rebooting drc-hv _after_ the crates are off is not a good idea!
    09 Oct05 NA The post-processing of the fastmonitoring data is now working well => it will replace the old-fashioned cron jobs which have been running for years.
    04 Oct05 NA New variable added in the ALH: DRC::FASTMON:FITTEDMEANTDCTIME gives the mean TDC time computed through a fit g+p0 of the corresponding histogram. Alarm limits on this variable should also us to detect two things: a very high bkg level (no visible peak on top of bkg) or a 16.7 ns-shift in the trigger timing.
    Noisy PMT in sector #9 is back and it's noisier than 1-2 weeks ago. Will keep an eye on it...
    03 Oct05 NA Background conditions improved quite a lot (although PEP broke the peak lumi record!) => I've put back the nominal thresholds at the beginning of the swing shift. The instructions are to page me if conditions degrade.
    02 Oct05 NA The background got higher during the swing shift as PEP was pushing the currents up. Therefore, we got alarms for the scaler rate in sector #3 and in other bkg monitoring variables. As DIRC safety was not a concern, I've raised some thresholds to make the shifters' life easier.
    02 Oct05 NA+KM One entry in bin #5 of the FX status plot for two runs. The reason for these is unknown -- Niels will try to investigate it with the help of Chris.
    > 30 Sep05 NA Paul Stiles sent me an e-mail: he found a component improperly soldered in the power supply of channel #0 which was likely the reason why the current was oscillating. He fixed that. I'll get the board back on Monday.
    28 Sep05 DRC crew Safety meeting at 05:30 am with RC and Frank O'Neill (Safety officer). Minor corrections to the JHAM. Door opened only around 7:50 am... Access during 55': NA, KM, Matt, Ray and Frank. Problem is quickly identified: the power supply was dead. Matt replaced it with the spare which was running in the spare FE crate and the sector came back to life! PS (cleared by OHP) will be examined by Ray. Current spare status: 1 new PS put in the spare crate; 1 PS broken. The total access time was 55'.
    27 Sep05 NA Communication lost with FE sector 9 at 02:33 am. Unable to resurrect the crate through epics or by turning off and on the chiller (turning it off opens the relay on top of the detector which shuts down all FE crates). Therefore I requested the backward east door to be opened. As the crew is not available today (!!!) this is postponed until tomorrow morning. We have 3 potentiel culprids: 1 non-bypassable fan; the fuse of the power supply; the power supply itself.
    26 Sep05 NA Working on the Drc Fast Monitoring Post Processing since last week. Things are pretty OK and the new setup will be patched in IR2 by Jim to be tested parallel to the current cron job.
    20 Sep05 NA+KM Replaced HV board #00337 (instable current readout in channel #0) with HV board #00327. The power glitch around 14:45 made the SOB circulating pump to stop. This was noticed at 16:10 when we got a yellow alarm on the SOB level. We were still in IR2 => we restarted the pump quickly by following the guidance from the ops. manual.
    19 Sep05 NA Got an e-mail from Rene (HVAC): he received 3 stirrers. 2 will be put in the DIRC chiller at the next opportunity (perhaps during tomorrow's MD) and the third one will be kept as spare for the future.
    17 Sep05 NA Transient red alarms on DRC:SC_10:HV:PS:IMON_32 since 09/14. The current readout jumps to very high values for a few seconds (scanning frequency of the record = 0.1 Hz) before coming back to nominal. Similar variations (although of much smaller amplitudes) are visible on the VMON channel. The channel connection on the HV board plug looks visually OK. We'll check it by hand next time we have the opportunity to turn off the crate and if the problem remains we'll swap the board with a spare one.
    16 Sep05 KM+NA Since last night (09/16 owl shift) there is a noisy PMT in sector #9: bin #10 in the occupancy per harness plot and bin #0 in the occupancy per HV group plot are slightly higher (at most a factor 2) than the others nearby. Mat did some maintenance work on DIRC water system.
    14 Sep05 NA+KA We found a good candidate for the Xmas tree in sector #3 and channel #29. That's the PMT in row #22 and column #9, see this det. ops. H.N. for details.
    13 Sep05 NA One of the two stirrers agitating the water in the chiller failed this morning. It melted the fuse(s) to which it is connected which cut the power on the chiller. I got paged around 7:10 am and went to IR2. I switched from the chiller to the BCS so that we could restart taking data. Then I sent a web request to CEF and Rene from HVAC came in around 9 am. He found the problem, removed the faulty stirrer and changed the fuse(s). Then the chiller restarted working and I switched back from the BCS to the chiller. Rene should order 3 stirrer spares: 2 to be put in the chiller, 1 to be kept as spare for future problems. I have also update the chiller guidance in the ops. manual (link added to a recent version of the chiller manual) and I have made some small modifications of drc-mon/db/siam.db to create new variables which check the inhibit status of a channel before claiming it has tripped. Vincent Poireau (DCH & RC) provided me the piece of code needed for this change.
    12 Sep05 NA New Xmas tree in sector #3, channel #29. It appeared at the end of the Day shift. The corresponding HV channel is now off, the alarm is masked in the ALH and the short term instructions for pilots and DQMs have been updated. We have a request on the white board: 1 hour without data taking to run a standalone run to try to guess which PMT is the faulty one.
    09 Sep05 NA The DIRC saw 2 indirect consequences of the power cut during the owl shift. 1) Spew files were not produced during few hours because bbr-farm02 had a problem; 2) the trigger timing was shifter by 1 tick (16.7 ns). Chris called me to confirm the problem (visible on the FM plots which had restarted to be produced). He rebooted the master crate and the problem disappeared.
    08 Sep05 NA Got back the three HV boards repaired by P. Stiles. Board #00315 (the one which failed on July 10) had a broken component preventing the HV channels to be powered. The two other ones (#00319 and #00324) had some settings which limited the current or the voltage in some channels. These have been corrected by Paul and I using the CAEN interface with the HV boards.
    07 Sep05 NA+Matt Good fans appear to be very... good! Matt changed a full tray in 5 hours.
    06 Sep05 NA The good new fans arrived!
    04 Sep05 KM Paged by day DQM-shifter "DIRC water chiller temp fluctuation greater that +1". I looked into the chiller read-out. The temperature looked pretty stable and fluctuations were within the threshold ( 15 +/- 1). There was no 'flow error' and temperature doesn't touch 'low' or 'high' value. Will keep monitoring this.
    31 Aug05 NA Matt Weaver and I swapped 2 ROMs. If the GLINK problem is due to the board, it should move from Sector #11 to Sector #9 (from ROM 2-2 to 2-1). Matthias also rebooted drc-soft to pickup the last changes I've made in the code. The main difference is that alarms should now only beep when PEP is in 'Stable Beams' mode.
    28 Aug05 NA Another occurence of the GLINK problem (63 seconds this time instead of 5 in the previous case) => e-mailed cpo and Matt.
    26 Aug05 NA Kalanand + Emmanuele 'joke' driven by the day pilot... See this thread for details.
    25 Aug05 NA Transient GLINK alarms from sector 11 during swing 08/25 and owl 08/26 => contacted cpo. He has already see such problem in 2003 but it didn't show up again since that period. We may swap the corresponding ROM with another one to see if the problem follows the board.
    24 Aug05 NA Monthly report given at the IR2 meeting.
    Yellow scaler alarms are beeping again now in the ALH as the bkg conditions have significantly improved in the last weeks.
    Matthias has finally setup the DIRC software IOC and so the background monitoring are now running! After a few hours of running it seems obvious that the scaler and neutron ratios should trigger alarms only when we're in stable beams mode -- injection periods are randomly noisy!
    23 Aug05 NA Got paged at 9 am as the RC wanted to know our policy about Global Calibrations: the beams hadn't been lost for almost three days. Delaying a global calibration by few days is not a problem for us. Anyway, beams got lost before noon and so all systems were calibrated through ORC. I've also moved the pilot and DQM 'long term' instructions from the 'short term instructions' webpages to dedicated ones. Finally, it seems that Wiener sent us the wrong fans => we should get 100 new ones by the end of next week, i.e. still on time for the shutdown.
    22 Aug05 ES NTR
    21 Aug05 ES NTR
    20 Aug05 ES NTR
    19 Aug05 ES NTR
    18 Aug05 ES NTR
    17 Aug05 ES Kalanand and I (with an extra help of Nicolas) will constantly monitor more carefully the spew files, in order to find quickly eventual timing problems.
    16 Aug05 NA Investigated timing shifts seen by SVT on their fastmonitoring plot since run 57156 (08/15 owl shift). The shift is also present in the DRC spew files but it wasn't detected as it made the timing fits fail -- which also frequently occurs when things are normal. From my reply to SVT on det. ops. HN, Chris deduced that the trigger timing was shifted by 1 tick of the 15 MHz clock. He rebooted the master crate before Run 57211 and the shift disappeared.
    16 Aug05 NA Kept on working on the drc-soft IOC package: package created in CVS, several bugs corrected + tuning done on standalone. Waiting for Matthias to setup the proper framework on the central software IOC to have the code running.
    15 Aug05 ES NTR
    14 Aug05 ES NTR
    13 Aug05 ES NTR
    12 Aug05 ES NTR
    11 Aug05 NA+ES NTR
    10 Aug05 ES NTR
    09 Aug05 NA+ES Rene (HVAC) checked the gauge: it's working fine. He didn't find any obvious problem on the chiller. He said that the important is that the flow is fine which is the case. So we'll let things as they are now and monitor the pressure -- if it would decrease we would change the pump which should take 1-2 hours once we would have bought a spare. DQMs have been asked to write the gauge output in the checklist.
    09 Aug05 NA I've rebooted drc-gas to update the epics version from 3.14.6 to 3.14.7.
    08 Aug05 NA I've made a request to ask HVAC people to check our chiller (low pressure / all other parameters OK). Rene came at 1.40 pm and decided to change the pressure gauge tomorrow during the MD.
    05 Aug05 NA+KM We've met Matt and Dave in IR2 to look at a fan tray. They've tried to open it to estimate the amount of work required to change all of them. It seems this will take longer than expected. We'll discuss this during the next meeting.
    05 Aug05 NA I got some corrections from Matt on the SOB degasing section: I'll implement them right now.
    04 Aug05 NA I've update the ops. manual section dealing with the SOB circulating pump and I've created a new one for the SOB degasing pump.
    03 Aug05 NA I discussed with Matthias about the new epics variables. He thinks that the IOCs should not be touched now in the middle of the Run as they are very stable. So it was decided to compute all the background variables on a soft IOC -- code running on a Linux or sun machine instead of a Power PC. The advantages are several: faster floating operations, simpler tests (that IOC can be rebooted as often as requested even during data taking!). I'll work on this soft IOC asap.
    03 Aug05 NA Matt and I were automatically paged around 4.30 am because the degasing pump stopped working. It turns out that the new outlet installed for safety reason few months ago had tripped. We reseted it (there is a button on the outlet) and were able to restart the pump which was OK.
    02 Aug05 NA Used the long MD to work on the new epics variables for background monitoring. The new version of drc-mon didn't work perfectly and so I decided to put back in the IOC the one which was running before I started working on it.
    28 July05 NA+KM+ES We've been training for Control Access!
    27 July05 NA I've updated the perl script creating the .alh files for HV (drc-hv IOC). Please use it from now to include permanent changes in the ALH.
    27 July05 NA I've tested a new version of drc-mon: changes of the background variable names, scaler state machine improved and check that the gate is received by the scaler board (indirect check: once a minute we check that some scaler rates changed). The tests were successful and so I let the new version in the IOC.
    26 July05 NA This morning we went three times non runnable for periods of 20 seconds (2 IMON/VMON samplings): twice when the channel went from 0 to 10 V (automatic recovery after a trip) and once when I tried to turn the channel ON the first time. So, to go non runnable some voltage needs to increase which is logic. What is more puzzling is that there were three other periods of time during which the voltages were raising and for which the DIRC remained runnable...
    26 July05 NA Fixed the broken script drc-hv/script/hvtrip.csh which was supposed to send an e-mail to babardrc when there was a channel trip. Now it works and it sends a page to the oncall DIRC expert as well.
    26 July05 NA+ES Got paged this morning because HV channel #6 in sector #5 tripped. It turns out this is a Xmas tree. Emmanuele and I found a very good candidate for the faulty PMT: row 7, column 14. We'll unplug it when we have access...
    26 July05 NA I've loaded in IR2 a new version of the JAS fastmonitoring: each set of plots has now its dedicated page and the DQM goes from one page to the other by clicking on html links at the bottom of the page. This should ensure that he/she looks at all plots all the time...
    22 July05 NA+KM+ES We brought the 3 faulty HV boards in Paul's Stiles office before the IR2 meeting.
    22 July05 NA Set low and lowlow alarm values (@ 100 Hz) for scalers. This would ease finding HV problems on the boards powering the scalers (1/4th of all boards).
    21 July05 NA New feature added in alh: automatic page and e-mail when the average fan speed gets in red alarm in one sector.
    21 July05 CT CVS commit of the epics changes + new versions of drc-mon and drc-hv put in production.
    21 July05 NA+NvB Got paged because a global calibration got stucked in the validation phase. After investigation, it turns out that there was a zombie process trying to access the DB which was hence locked. So the new calibration constants couldn't be loaded in -- on the ORC, the validation step covers both the validation and the writing to the DB. After the process was killed, two standalone and one global calibrations were successfully taken.
    20 July05 NA+CT Tested some new features in drc-mon and drc-hv. Backed up the previous version of the ioc codes to avoid any problem when data taking restarts. Epics work will keep on tomorrow.
    20 July05 NA+KM+ES The 6 HV boards repaired by P. Stiles in spring have been tested. 4 of them are working fine and have been let in the detector -- hence we have 4 more tested spares. The 2 other ones will be brought back for repair with the one which broke on July 10.
    16 July05 KM The problem of distortion in DIRC timing plots and trigger-line visited us again ( this time also seen in SVT timestamp plots). See the logbook entry of June 29 for more details. Seems like a trigger problem or something to do with beam condition ....
    15 July05 KM More news on the DRC pump alarm problem - in light of Jerry's observation that it is merely a readout problem which can happen with any VSAM , Matthias implemented a check for the absolute value of the DRC::PUMP variable. We will also think about implementing a software delay in response time (to, say, 5 minutes) for alarm if this happens again.
    14 July05 KM Jerry and Ray have been trying to find the source of the fake DIRC pump alarm. Christophe is in SLAC to look into EPICS problems.
    13 July05 KM Intermittent false alarm "Degassing Pump OFF". Jerry & myself went to check many times - the pump is functioning alright and we couldn't see any problem.It happens because "DRC::PUMP" is fluctuating. It should be reading +10.5 but goes to -10.5 from time to time...Please see the plot .
    12 July05 KM Reset the default ("burt") HV values.
    11 July05 KM NTR
    10 July05 KM+JO We lost one HV board ( board #5) in sector 1 this morning. We replaced this board (HV board 5 in Sector 1 (HV carte 0)) promptly. Following are the details:
                                                                                                    
    Old (faulty) board : Serial No: 00315, SLAC ID: 20004679
    -------------------
    New board          : Serial No: 00343, SLAC ID: 20004671
    -------------------
    
    Downtime due to this replacement was negligible, as PEP lost the HER and we worked during the injection. To restart datataking promply, we set the HV values by hand. The "burt" setting will be done at next opportunity ( already put a message on white board). Meanwhile, the effected runs 55975-55980 were marked flawd. Now everything is working alright. DIRC is Fine !
    09 July05 KM NTR
    08 July05 KM NTR
    07 July05 KM NTR
    06 July05 KM NTR
    05 July05 KM NTR
    04 July05 KM NTR
    03 July05 KM NTR
    02 July05 KM NTR
    01 July05 KM NTR
    30 Jun05 KM+NA Changed the threshold for number of entries in bin #11 of FX plot from 5000 (previous) to 10000 (new).
    29 Jun05 KM+NA+JO During the owl shift we were paged because our JAS timing plots were not showing the expected peaks for run 55662 and also bin #11 of FX plot had >10000 entries (threshold : 5000 enties). We could not find any obvious reasons for this problem apart from distorted trigger lines (page 14 of FastMon spew plots) and in fact the problem disappeared in the next run. So we think it has something to do with trigger configuration. For details and ensuing discussion please see the hypernews posting .
    25 Jun05 KM Put DIRC back in normal data-taking mode (auto configuration).
    23 Jun05 KM Put DIRC in safe mode (local + V1 configuration) in anticipation of a rocky beam condition.
    21 Jun05 NA+KM+ES We've unplugged the PMT responsible for the Xmas tree: sector 1, row #37 and column #20. Then we've also replaced the fan tray in sector 2: this took us more time than expected as there was no procedure in the manual. It will be updated next week to describe this operation in details.
    20 Jun05 NA+KM+ES Backward doors opened this morning. We've changed the 3 faulty DFBs:
    • (sector 4; slot #3), DFB 6A replaced by DFB 05
    • (sector 5; slot #7), DFB 57 replaced by DFB BE
    • (sector 7; slot #1), DFB 71 replaced by DFB 3E
    When we've turned things on, we found a Xmas tree in sector 1 in the outer part of the sector (most likely HV group 49).
    Niels unmasked the 4 PMTs in sector 7 which were masked on 05/02.
    18 Jun05 NA I have increased the beep severity of the scalar alarms from yellow to red. Pilot short term instructions have been updated accordingly. See entry in the Det. Ops. H.N. for details.
    14 Jun05 NA The 8 PMTs in sector 5 which were less efficient are now completely unefficient. This seems to mean that the faulty DFB component completely stopped working.
    13 Jun05 NA After a long struggle with JAS, I have commited a new version of the DRC Fast Monitoring JAS pages, hopefully nicer and clearer than the previous one.
    09 Jun05 NA Yesterday evening during the swing shift I found that 3 scalers were in yellow alarm whereas the background was OK. It turns out that I had set by mistake the 3 thresholds at 30000 instead of 300000. I fixed that immediately.
    08 Jun05 NA PEP requested a few hour access this afternoon. So I rebooted drc-mon after having included in the running version most of the changes made yesterday. The only difference is that we decided to keep the scaler yellow alarm thresholds at 300 kHz uniformly.
    07 Jun05 NA I've been working most of the day shift on drc-mon. I've tested a new version of the IOC with several changes (sector-dependent scaler alarms, possibility to disable the beep for the yellow scaler alarm, possibility to bypass (or not) by default the fan readout in the FEE, correct variables for sectors 8-11 to scalers.dl, ALH guidances changed accordingly) but it is not in production yet. I've also made a minor change in the drc-hv ALH: the yellow alarm occuring when DRC is in local control is now in a directory called 'General' which helps the pilot to locate it.
    03 Jun05 NA I've rebooted drc-mon during the swing shift to change the low and lolow alarm thresholds for the average fan speed in the Wiener crate. The mean speed (computed on 6 fans over 9) is very stable and its expected value is 3120 rpm. Hence, I've put the yellow alarm at 3050 rpm and the red alarm at 2950 rpm. We'll see if these levels are accurate. By default, the alarm for the sector 2 is disabled as we currently have one broken fan there.
    01 Jun05 NA Info from cpo: the few events taken without sector 2 on Sunday (runs 54810 -> 54813) are associated with ODF damage bits and are thus not processed. So these runs are flagged good for the DQ.
    31 May05 KM Today morning (when BaBar was not taking data), I did some some diagnosis on the faulty fan (fan #2) of sector 2. After switching FEE crate OFF and then putting back ON again, the faulty fan came back to life. However it is too unstable to be trusted to run smoothly (see here and comparison plot for a different fan). So I have continued with bypass status, via EPICS, for the failure status record. Right now (11:55 am) the faulty fan is running (but I don't know for how long ! ).
    Also, to prepare for the possible fan replacement in this sector (#2), I setup a 'fan test' this morning with 9 fans in a spare fan tray - they will be running for a couple of days.
    30 May05 KM Continued monitoring temperatures and fan speeds in sector 2. Fortunately nothing unusual today.
    29 May05 KM+JO Unfortunately today we had first DIRC-induced downtime in run 5. At 15:10 DIRC went non-runnable - the FEE crate on sector 2 was shut down as a consequence of a fan failure (fan #2). Luckily this is one of the six fans for which we can bypass, via EPICS, the failure status record. So for the time being this crate is running with 8 fans on, this prevents us from requesting an emergency access during this memorial-day weekend. Jose pointed out that this is the second fan failure in recent times (the previous one was sometime in mid-december). I (KM) am monitoring temperatures and fan speeds in sector 2.Also Added the following request to the whiteboard (in IR-2 control room): "Please page DRC expert during weekday (8am - 6pm) if there is at least one hour without beam". This will allow us to do some diagnosis of the falty fan.
    Since this is the second fan failure in recent times and the fans have reached their expected lifetime,we plan to change the 9 fans of this crate as soon as we get access.
    25 May05 KM Monitored SOB water-level whole day. The value looks very stable, fluctuating in a very narrow range 5.41 - 5.44. Nothing abnormal.
    24 May05 KM Neils did some test on DIRC partition during ROD.
    23 May05 KM There was a false alarm regarding "low water level in DIRC SOB". Matt did some check and he couldn't find anything wrong. Looking into epics and ambien database every thing looks alright.
    21 May05 NA+KM DIRC switched on in the morning: no hardware problem, minor issues with the online software as usual... A ~9.10 pm a VESDA alarm cut the power in the detector. So I (NA) had to come back to IR2 to ressurect the DIRC a second time. A breaker on top of the E.H. was opened which prevented the FEE to be powered. It took some times to understand that was the problem but everything was fine when the breaker was fixed.
    18 May05 NA Power outage at 7.51 am => SLAC back to stone age for 3 days... Meanwhile, Matt changed the green power light.
    17 May05 NA The day DQM shifter noticed that the green power light of the 'Water degassing' panel is off although things are working well. I've e-mailed Matt about this.
    16 May05 NA I was paged by the swing pilot as a test run for SVT people couldn't start because of a SynchRequest error in a DRC crate. The pilot was asked to retry starting the run and everything worked well at the second attempt. My guess is that the ROMs were in a bad state at the first attempt and that rebooting them (which is automatically done when a transition fails for a subsystem) fixed the problem.
    11 May05 NA Monthly report given at the IR2 meeting.
    10 May05 NA I discussed with Stefan about fast monitoring improvement. He provided me various useful information I'll try to compile in a webpage. During the night he improved the layout of the JAS FX_status histogram which should stop the flow of DQM spurious e-mails from last week. I also got the 4 last HV boards in repair such as the crate. To be tested...
    07 May05 NA Update the oncall planning until mid-July.
    03 May05 NA The scalar spikes problem has been fixed this evening with the help of Steffen and Matthias: we've added a lemo cable between the gate generator and the scalar module through a reset signal is sent each time when the gate appears. In this way, even if the state machine does not see the gate (which is the reason why we got the spikes), the counter will be reseted anyway. A spike will be replaced by two consecutive readouts having the same values in all channels -- the refresh is only done when the gate is detected. This seems a nice solution as the scalar output is expected to be flat during normal conditions. I've e-mailed Ray so that he can make our temporary cable permanent.
    02 May05 NA Decided to mask 4 new PMTs in sector 7 (likely to be a problem on the DFB which will be changed when possible). Found one strange structure (4 PMTs) in sector 11 during a cosmics run: present but much less visible during data taking => will monitor its evolution in the next days. The reboot of drc-mon last night put in production my error messages. The plan is to monitor the IOC while looking at the log to see when spikes occur. I also prepared 3 directories for our IOC codes compiled with epics 3.14.7 instead of 3.14.6.
    30 Apr05 NA Standalone run taken during a long PEP downtime: it confirms that the two PMTs have been masked. It also shows that the 4 noisy PMTs in sector 7 have a 4 MHz rate! The standalone LED calibration didn't work again ('infinite loop' like during the last attempt).
    29 Apr05 NA+NvB 2 PMTs masked: (sector 4, row 1, column 1) and (sector 6, row 1, column 7). DQM short term instructions updated as we have 4 noisy PMTs in sector 7. Origin not yet understood => nothing will be done during the weekend.
    27 Apr05 NA Still working on the scaler spikes [true for every day of the week!]: various bugs already fixed in epics but they didn't fix the spikes.
    26 Apr05 NA+KM Still working on the scaler spikes. The gate seems OK and its information appears to be well propagated to the corresponding epics variable (which can be enabled/disabled). After a long discussion with Matthias and Vincent, we think that the problem may be the reset of the counters which is triggered by epics: either the driver doesn't do its job properly or the board itself is faulty (both boards in fact at the other one shows exactly the same feature). More news on tomorrow as we'll certainly have some opportunity to reboot drc-mon during the MD. BTW: studying the scaler state machine showed other problems which will be fixed by rebooting drc-mon.
    24 Apr05 NA Paged again: this time the downtime was as long as expected and so I had time to work on the DIRC. The scaler module was changed but this didn't solve the problem (=> more investigation needed during the week); the standalone calibration didn't work (see posting to babardrc for details); rebooting drc-mon was OK.
    23 Apr05 NA Paged because no beam expected during ~ 2 hours. Arrived in IR2 ~15 minutes later to discover that the injection just restarted. Apologies from pilot and liaison shifters: MCC fixed quickly the problem. Stayed ~1 hour in IR2: permanent injection problems but never bad enough to require some repair -- there was always a small current in PEP! Left IR2 without having done anything.
    22 Apr05 NA Strange behavior of scalers (and neutron counters) found by Matt Weaver during his liaison shift: values ~ double from time to time. Indeed, later checks showed that such spikes have 'always' been there -- at least 1 year ago they were. A first test excluded a gate stability problem from the IFR generator. Then, soft. tests with Jose and Matthias showed that the epics input values (directly coming from the hardware) have these spikes as well. Hence, it was decided to change the scaler module at the next opportunity: hopefully it will solve the problem.
    21 Apr05 NA drc-mon to be rebooted when possible: Sherry changed db/vavra.db and I slightly modified the high alarm thresholds for N2 flows in all sectors expect #0 and #10 where the electronics flowmeters are saturating (miscalibration).
    20 Apr05 NA 8 PMTs found less efficient in Sector 5. They form one chip of a DFB which is likely to be faulty. One standalone calibration (hbooks are currently not filled during global calibrations) + one standalone run needed to confirm the diagnostic.
    19 Apr05 NA Included drc-hv/burt/ in CVS. Fixed the path of an epics setup which was pointing to an old-fashioned file.
    17 Apr05 KM+NA One more good physics run taken: 53736. The lumi is higher than on Saturday and so the occupancy plots are finally almost flat.
    16 Apr05 NA First data taken: Run 5 is starting! Two good physics runs taken: 53732, 53733. The lumi is so small that they are 'hybrid' between cosmics and physics data. Otherwise they look fine.
    15 Apr05 NA E-mail sent to Bill W. to warn him about the DRC plan for the October shutdown (period confirmed): 108 fans to be changed by a SLAC tech. He acknowledged on Monday.
    13 Apr05 NA Tests done with Matt and Jerry on the N2 flowmeters. Found that epics flowmeters reach a plateau at 200 cc/min, a value below (!) the high level alarm (220). I also rebooted drc-mon to set new low level alarms for N2 flows: low 80 -> 70; lowlow 60 -> 50
    12 Apr05 NA DIRC backup pager (6508469804) given to Kazuko so that it can be cancelled.
    11 Apr05 NA I rewrote more clearly the normal values for the 12+1 N2 flowmeters -- some figures were almost gone with the time. Fixed the yesterday swing DQM logbook entry.
    08 Apr05 KM NTR
    07 Apr05 KM Yesterday's mysterious pager is understood: during the DQM training, someone pushed on the expert pager button whereas there was an old message in...
    06 Apr05 KM I got the the following automatic pager from EPICS: "Permeate Flow and Concentrate Flow @ 0 GPM for the DIRC Water System ROU". Bob and I went to inspect meters measuring DIRC Water flow and we couldn't find anything wrong anywhere. So it turned out to be a false alarm. I will talk to EPICS expert to avoid (minimize) such alarms in future. Bob adds: "Yes, this in fact is beyond a false alarm. The system (RO water) has no provision to read these values into EPICS, nor are they instrumented to be read out into anything else. If they were read out, the normal value would be "0 GPM" unless the RO water system were adding water to the DIRC system. The DIRC DI water system looks fine."
    05 Apr05 KM NTR
    02-04 Apr05 KM 4/2 Day shift DQM shifter raised two checklist issues: 1.) "Gas flow in sector #2 reads 85 instead of 100" => But as far as I can see from Apr 01, 2005 onwards in ambient, gas flow in sector #2 was always 100.5 (constant).
    2.) "Water tank outlet reads ~2 instead of 3 gpm" => This is perfectly within the allowed range. A month or so ago I asked this question to Bob Reif and he wrote the following reply (on March 07, 2005): "The level you see is the middle sensor, not the lower sensor, at which point the water system will top off automatically and bring the level back up to the upper sensor level. The water system looses water through evaporation in the areas that are purged with nitrogen and this slow loss is normal. The water return tank has lost about 1/4 inch of liquid in the last 3 weeks. Upon truly reaching the lower level sensor, the circulation pump will shut down to protect itself from operating with no water inside."
    I checked the water tank this morning (04/04) : Right now water is above the lower level and it is filling up.
    01 Apr05 NA recompiled the epics code for the DIRC IOCs and rebooted them (Jim H. fixed a bug in the pvget epics command).
    31 Mar05 NA added a third scaler for Jerry in epics (thanks to Matthias).
    30 Mar05 KM Niels is done with his tests on the DRC online.
    The 108 fans we ordered to replace those of the FEE crates just arrived. They will be changed during the next shutdown whenever it occurs.
    29 Mar05 KM+NA Last night DQM shifter paged me (KM) to ask why the timing in fast monitoring plots were shifted by ~ -40 ns. We, at the moment, think it is related to the fact that we are reconstructing cosmic rays coming from outside BaBar as couple of tracks coming from interaction region.So it looks like a cosmic run feature, however we have raised this question to DIRC software experts and we're waiting for their respone.
    28 Mar05 KM Paged from the swing shift DQM shifter last night about non-zero entry in one of the bins of FX-status histogram on Fast-mon page which means some problems with data taking. In this case it seems that a DFB is not fully synchronized: Niels is looking into it.
    24 Mar05 KM Paged from the DQM shifter about transient noisy channels in harness (sector 4). Already instructions in the FastMonitoring were updated a few days ago (03/10) to better describe these events in response to exactly the same message from DQM shifter.
    21 Mar05 NA Problems with the water chiller: the water level started to decrease quickly during the afternoon, triggering a potential leak. In fact, the DIRC chiller dump valve was accidentally opened! As this is already the second time such problem appears, I'll update the chiller instructions to describe this problem and the way to fix it.
    18-20 Mar05 NA IFR-mon does not crash anymore => our scalers are working fine.
    15 Mar05 NA Tried to recover in one day more than one year of epics modifications done outside the CVS framework... The 3 CVS packages drc-gas/hv/mon are now obsolete and have been replaced by epics-drc-gas/hv/mon. They contain a merge of the code in the old CVS repository, of the recent modifications done by Christophe and of the new framework introduced by Matthias and Sherry during the epics version migration. Most of the job done today, few tunings left to be done tomorrow... Monthly report (pdf) given at the 3.45 pm meeting
    14 Mar05 NA Another crash of the IFR gate generator... Discussion with Matthias and Sherry about DIRC IOCs: Christophe's changes were not propagated because they were not in CVS. The CVS archives of the 3 packages (drc-mon, drc-hv, drc-gas) are more than 1 year old => 3 new packages created: epics-drc-mon, epics-drc-hv, epics-drc-gas. We'll try to include Christophe changes in Matthias' version tomorrow.
    13 Mar05 NA Nick fixed the gate generator problem around 3 pm. Matthias and him will try to solve these recurrent failures during the week.
    12 Mar05 NA Like yesterday night, I found that the scalers are not working. Yesterday it was a problem on the IFR gate generator; I assume the same thing occured tonight. I sent a e-mail to Nick and warned the pilot (no bkg, no problem!). During my shift, I monitored the drc-hv IOC CPU load during V0 <-> V1 transitions (attempt to understand why the DIRC ramping in global mode looks so slow). The load is quite constant in the range 25-30%. I also noticed that the recent changes made by Christophe in epics disappeared during the version upgrade => I e-mailed Christophe and Matthias.
    11 Mar05 NA NTR (checklist OK).
    10 Mar05 NA H.N. from the owl DQM shifter about transient noisy channels in harness and/or HV plots. Instructions in the FastMonitoring updated to better describe these events.
    09 Mar05 NA+JO DIRC turned off during the 30' power outage (1 pm -> 1.30 pm) and turned on after it ended. Matthias paged me around 5.15 pm to request the permission to reboot drc-mon to get rid of a duplicated db record. A last reboot occured just before 8 pm. The pilot didn't put back the DIRC in the partition until about 11 pm when I called IR2. It seems that the pilot didn't understand well the ORC information: he thought the DIRC couldn't be put in but doing this and simply beginning the run worked.
    08 Mar05 KM 1.) Calibration problem was solved yesterday. Jim says "We had to update the releases (online and offline), then there was a feature in one of the scripts that prevented this from working properly. Said feature is a hardcoded release number which needs to be removed.When you guys have finished your tests I will try to make the scripts release independent again."
    2.) DIRC fast monitor is NOT working. Stefan Spanier is looking into it.
    07 Mar05 KM+JO Jim Libby is working on calibration problems again. We switched on DIRC on his request. Matthias continues to work on upgrading DIRC IOCs. Checklist ok.
    05 Mar05 KM checklist ok.
    04 Mar05 KM+Jose 1.) new Epics version being installed in IOCs by Matthias: so far, drc-gas has been upgraded and seems to be working fine.
    2.) Jim Libby is working on calibration problems. We switched on DIRC on his request & switched off at the end of the day. Jim will continue his test on Monday.
    3.) Matthias migrated drc-mon to the new version of EPICS. Jose turned the FEE off at 4pm but for some reasons ( probably due to configuration in the new version) the crates were powered ON a bit after 7pm. Jose noticed it just before midnight. Since Chiller & HV were off, we couldn't afford to have FEE on and running because its temperature was going up (went past the alarm level). So Jose had to come to IR-2, switch on the chiller and let the temperature go down. After that he switched off FEE. We need to closely monitor this & avoid its recurrence.
    03 Mar05 NA+JO+KM The backward right door was opened this morning to fix the Xmas tree. We didn't succeed in turning ON the HV channel to which that PMT was belonging: when we tried the first time, it tripped. At the second attempt, it tripped again, immediately followed by several boards in neighboor sectors. We followed the safety procedures defined in the corresponding non-routine JHAM and disconnected the faulty PMT after having identified it. The PMT was in sector 2, harness 3, channel 24 (column 05, row 22). Finally, we took a 15 minute-long standalone run with the HV ON to check that everything was fine again after the fix.
    02 Mar05 KM+JO+NA DIRC restarted after the fix of the power in IR2 south panel. After the fix of two problems on drc-mon, a standalone was taken and a Xmas tree was discovered in sector 2.
    01 Mar05 KM checklist ok
    28 Fev05 KM checklist ok
    25 Fev05 KM checklist ok
    17 Fev05 KM+JO Switched off crate #7 in anticipation of power cut (before noon) & switched on back (at 2 pm) after power was back. Everything OK.
    16 Fev05 KM+JO Attempt to do a LED calibration failed. Jim is investigating the problem.
    15 Fev05 NA Message from Bob: "The switch over to the new outlets at the DIRC water system have been completed with the exception of the single UPS plug powering the SOB water level monitor. This one will be done as soon as the electrician's supervisor completes the flash hazard calculation for this circuit."
    14 Fev05 NA+JO Another attempt: same failure. The problem seems to be related to the DB. Jim needs to talk to Andy about it.
    11 Fev05 NA+JO Attempt to do a LED calibration failed. Jim is investigating the problem.
    11 Fev05 NA+PG+JO The SOB water circulation pump didn't restart after the short power cut of this morning. After a manual reset on the panel, it restarted.
    11 Fev05 NA The plot for yesterday's standalone data taking can be found here. A PMT in sector 4 was very noisy in the first hour (~ 1 MHz) but then its rate strongly decreased below 1 Hz. The noisy PMT in sector 5 is still there: constant rate around 150 kHz over the 6 runs.
    10 Fev05 NA Standalone data taking between 10.40 am and 5 pm: about 9 million events collected. Updates of the Operations manual: standalone run section improved + new section dealing with the N2 flow tuning.
    09 Fev05 NA+KM Successful test of the switch between the BCS and the DIRC chiller.
    03 Fev05 NA In the afternoon, Bob installed a second stage on the N2 regulator. In this way, we're better decoupled from fluctuations in the N2 tank. Since this fix, flows are stable again (maybe more than ever!). In addition, Christophe created stripcharts for the N2 flows in the 12 sectors, the inlet pressure and the humidity sensors.
    02 Fev05 NA+JO DIRC turned on around 5 p.m. Two runs (52542 and 52543) were taken with DIRC and B-field both on before the DIRC was switched off again due to the current limitation in IR2.
    28 Jan05 NA N2 pressure level decreased again in the late afternoon -- flow spikes close or above 7.05.
    26 Jan05 NA N2 pressure level decreased at 7.45 pm (6.96 -> 6.16).
    26 Jan05 NA Spikes in the N2 flow still present -- although no more increasing. Origin still unknown: don't come from the BaBar N2 tank (almost empty yesterday, filled this morning around 6.30 am); no PEP-II activity.
    25 Jan05 NA Power test at IR2: the DIRC completely on (chiller, HV, FEE) needs 25 A. Large (~10%) variations in the N2 pressure in short timescales (few tens of seconds) with growing amplitudes since 1 am. Bob will look at them tomorrow morning. N2 pressure level decreased by hand around 7.30 pm as peak values were above the 7.05 limit.
    24 Jan05 NA Compared this standalone run with the one of December 3rd 2004. All PMTs show roughly the same rate in both runs.
    23 Jan05 NA Analysed standalone runs taken 2 days ago. The standalone run plot shows a noisy (~100 kHz) PMT in sector 5. This PMT was already noisy during December standalone runs (~30 kHz). On the other hand, the very noisy PMT in sector 6 is quiet now.
    22 Jan05 NA Increased the N2 pressure level from 5.74 to 6.30.
    21 Jan05 NA+KM+JO DIRC on for 5 hours. HV board #360 successfully tested (let in the detector); calibration OK; 3 million events taken in standalone run. DIRC turned off after the end of the day shift. New version of the DIRC fast monitoring installed in IR2 (with the help of Jim H.).
    20 Jan05 NA DFB 7E tested in the test crate: synchro problems => deeper investigation next week with the Macintosh software.
    18 Jan05 JO+KM+NA 1.) Increased the N2 pressure level from 5.6 to 6.4.
    2.) we requested in the detector operation's meeting that we would like to switch on DIRC for a few hours for cosmic run to check that everything is ok.
    15 Jan05 KM+JO There was an unexpected SLAC-wide power outage this morning at 10:15. We got a "Degassing Pump Off" alarm. The power came back around noon.
    13 Jan05 NA+KM+JO 1.) We recently noticed that the EPICS alarm values on the N2 pressure record DRC:CAN2:ST29:ALLVAL28, as well as the ones from the N2 flow per sector DRC:CAN2:ST29:ALLVAL16 to 27, are not being propagated to the alarm handler. This means that a problem there -and we have had a few these last days- could remain unseen for a few hours, until someone checks out on EPICS (or until the navigator does its checklist).
    2.) Bob worked on changing the filters.
    3.) We decided to keep drc off during the cosmics run, until the power line is repaired.
    12 Jan05 NA+KM+JO DRC switched on at 1 pm and off at 5 pm during the power test. Info from Philippe: the DIRC power is 30 amp.
    11 Jan05 NA Increased the N2 pressure level from 5.8 to 6.3.
    07 Jan05 KM Increased the N2 pressure level from 4.8 to 6.3.
    06 Jan05 KM Again: Lowered the N2 pressure level from 7.1 to 6.3.
    04 Jan05 NA Again: Lowered the N2 pressure level from 7.1 to 6.5.
    03 Jan05 PG Again: Lowered the N2 pressure level.
    31 Dec04 PG Lowered the N2 pressure level (was at 7.4)
    27 Dec04 JV Again VESDA fake alarms. DRC-MON was rebooted and produced several fake SOB water level alarms.
    26 Dec04 JV VESDA fake alarms. DRC-MON was rebooted and produced several fake SOB water level alarms.
    13 Dec04 PG Fan tray replaced. Problem fixed.
    11 Dec04 PG In the fan-tray of FEE crate 7, one fan died: this automatically switched off the crate. . As it is one of the 3 Wiener-modified group, the "fan failure" cannot be bypassed. Removed DIRC from the state machine, will request access on monday to replace it.
    04 Dec04 JO Calibration cron job failed, because the disk was full. Nicolas made some space cleanup.
    03 Dec04 JO BW door was opened at our request (EMC needed it too). We replaced the DCC from sector 3. It now has DCC##22 on, and working fine. We'll send the faulty DCC back to LLR to check. Global Calibration failed twice, but succeeded afterwards.
    02 Dec04 JO Turned the DIRC on (HV, chiller, FEE). Reset the alh configuration to normal mode. Problem: the FEE crate from sector 3 shows faulty values on power supply and temperature. It is probably a readout problem, as calibration and random triggers look ok (calibration plot, standalone run plot). There is also a very noisy channel (row=3.and.column=6.and.sector=6), with rate ~MHz.
    Nov04 JO NTR
    29 Oct04 JO Turned off the DIRC (HV, chiller, FE electronics). Set the alarm configuration to shutdown mode. A short status report was given (link here).
    20 Oct04 JO 8 FEE crates went off ca. 15h30. Pilot says it was correlated with several AC failures.
    14 Oct04 JO Tested all HV boards on crate 6; problem remains. Replaced crate 6, problem fixed.
    13 Oct04 JO HV values unstable on sectors 10-11: fluctuate from V0 to V1 and back. Will investigate tomorrow...
    11 Oct04 JO The playback FastMon was reenabled.
    09 Oct04 JO BaBar takes cosmics. Persistent white alarm on DRC:SC_3:FEE:FNS:TXO. Fixed by power-cycling the wiener crate.
    08 Oct04 JO Chiller drained when switched on. Call HVAC, they found the problem: one valve below V32 was in the wrong position.
    07 Oct04 JO Bob refilled the SOB. More here.
    30 Sep04 JO Bob found a water leak on sector 8. More here.
    22 Sep04 JO Attempted at testing the gate generator spare; could only perform 2 of the 6 tests suggested by CAEN (interfering with IFR/LST activities)
    14 Sep04 JO Power cabling replaced. Modified remaining DCC boards (S01, S08, S09).
    13 Sep04 JO Modified 3 DCC boards (S00, S10, S11). Replaced 1 DFB (S07, B11)
    10 Sep04 JO Fan tray cleanup: upper half finished. LED calibration runs taken.
    09 Sep04 JO Tested all spare DCCs: 5 fine, 1 repaired, 1 sent back to LLR
    08 Sep04 JO Fan tray cleanup: lower half done. Several standalone runs taken.
    07 Sep04 JO Start shutdown activities
    01 Aug04 KY NTR
    31 Jul04 KY End of Run4!!!
    30 Jul04 KY NTR
    29 Jul04 KY NTR
    28 Jul04 KY NTR
    27 Jul04 KY Jim masked the noisy channel in sector 9
    26 Jul04 KY NTR
    25 Jul04 KY NTR
    24 Jul04 KY NTR
    23 Jul04 KY New noisy channel in sector 9, harness 10. Also the water leel of the tank is reading about 1cm high.
    22 Jul04 KY NTR
    21 Jul04 KY NTR
    20 Jul04 KY NTR
    19 Jul04 KY NTR
    18 Jul04 QZ got paged again on the HV problem when it ramp down.
    17 Jul04 QZ got paged around 11:00pm, HV sector 10 channecl 32 over current.
    16 Jul04 QZ NTR
    15 Jul04 QZ NTR
    14 Jul04 JO NTR
    13 Jul04 JO (Boda Franek) ORC calibration control now checks for Dirc HV
    12 Jul04 JO NTR
    11 Jul04 JO NTR
    10 Jul04 JO NTR
    09 Jul04 KY
    08 Jul04 KY
    07 Jul04 KY
    06 Jul04 KY
    05 Jul04 KY NTR
    04 Jul04 KY NTR
    03 Jul04 KY NTR
    02 Jul04 KY paged because of entries in bin 11 (> 100)
    01 Jul04 KY NTR
    30 Jun04 KY NTR
    29 Jun04 KY NTR
    27 Jun04 KY NTR
    26 Jun04 KY NTR
    25 Jun04 KY NTR
    24 Jun04 KY NTR
    23 Jun04 KY NTR
    22 Jun04 KY NTR
    21 Jun04 KY NTR
    20 Jun04 QZ NTR
    19 Jun04 QZ NTR
    18 Jun04 QZ NTR
    17 Jun04 QZ NTR
    16 Jun04 QZ NTR
    15 Jun04 QZ NTR
    14 Jun04 QZ NTR
    13 Jun04 QZ NTR
    12 Jun04 QZ NTR
    11 Jun04 QZ NTR
    10 Jun04 QZ NTR
    09 Jun04 QZ NTR
    08 Jun04 QZ NTR
    07 Jun04 QZ NTR
    06 Jun04 QZ NTR
    05 Jun04 QZ NTR
    03 Jun04 QZ computing people move dirc Achiver to a virtual achiver, IOC mon is rebooted by them.
    02 Jun04 QZ dirc monthly report (ps, pdf)
    01 Jun04 QZ NTR
    31 May04 QZ NTR
    30 May04 QZ NTR
    29 May04 QZ NTR
    28 May04 QZ NTR
    27 May04 QZ NTR
    26 May04 QZ test the new gate generator, it works fine at teststand, but does not work in caen. put in a new scalar card in .
    25 May04 QZ NTR
    24 May04 QZ NTR
    23 May04 QZ NTR
    22 May04 QZ NTR
    21 May04 QZ NTR
    20 May04 QZ NTR
    19 May04 QZ NTR
    18 May04 QZ NTR
    17 May04 QZ NTR
    16 May04 QZ NTR
    15 May04 QZ NTR
    14 May04 QZ NTR
    13 May04 QZ NTR
    12 May04 QZ NTR
    11 May04 JO NTR
    10 May04 JO New EPICS record: DRC::WTR:VSM:CALC40 reads out a time-integrated PMT signal installed by Jerry (details here)
    09 May04 JO NTR
    08 May04 JO NTR
    07 May04 JO NTR
    06 May04 JO NTR
    05 May04 JO NTR
    04 May04 JO Paged beacuse of FX bin 11.....
    03 May04 JO Bob replaced power light LED on the reverse osmosis.
    02 May04 JO NTR
    01 May04 JO Power failure in region 4. The 476Mhz clock was down, making DRC non runnable. Solved with power back on.
    30 Apr04 JO Paged because of a few entries in FX plot bin 11. Asked Stefan to add a comment on this in the JAS page.
    29 Apr04 JO NTR
    28 Apr04 JO Jim masked out the noisy channel for runs after 47588. Coordinates are sector 7, column 14 row 7. New key produced: 3437
    27 Apr04 JO The "usual noisy" pmt on sector 7 is again sparking. Looks like time to mask it has come.
    26 Apr04 JO NTR
    25 Apr04 JO NTR
    24 Apr04 JO NTR
    23 Apr04 JO NTR
    22 Apr04 JO Bob will look for a repacement for the RO power light (a neon, in fact).
    21 Apr04 JO The power light LED on the reverse osmosis unit was reported as now dead, but is still flashing. Thomas informed Jerry and Bob. There was again a GLINK alarm (snapshot), but no outofsync, no ODF damage was seen. Is it the EPICS value that gets crazy?
    20 Apr04 JO DQM reported a spike on sector 7. It's the "usual noisy" one. DQM short-term and HV ramping instructions were updated.
    19 Apr04 JO Paged because of bin 11 in FX histos.
    18 Apr04 JO NTR
    17 Apr04 JO NTR
    16 Apr04 JO Philippe checked the missing chip codes on the gate generator. Info sent to CAEN.
    15 Apr04 JO NTR
    14 Apr04 JO NTR
    13 Apr04 JO There was message in the logbook, concerning a DRC:SC_6:FEE:ERR:GLINK red alarm at 20h12; lasted 4 seconds, no track of any problem in the ambient at that time... should we worry?
    12 Apr04 JO NTR
    11 Apr04 JO NTR
    10 Apr04 JO Paged because of bin 11 in FX histos, and a noisy channel on sector 7.
    09 Apr04 JO DQM reports water system power light is blinking. Informed Bob.
    08 Apr04 JO Checked with Philippe the Xilinx chip codes on the V462 gate generator. Took a sandalone run; looks fine. Thomas added a record to EPICS for the Helium sensor (see his posting here). Kept on taking more standalone runs as downtime continued...
    07 Apr04 JO The inlet pressure was slightly high this morning. I lowered it.
    06 Apr04 JO request to be called in case of >1hour downtime to check page generator chip-type and code; merged with similar IFR request
    05 Apr04 JO NTR
    04 Apr04 JO NTR
    03 Apr04 JO NTR
    02 Apr04 JO NTR
    01 Apr04 JO Thomas intalled the new version of the state machine to fix the ramping in case of beam loss or global calib.
    31 Mar04 JO NTR
    30 Mar04 QZ NTR
    29 Mar04 QZ NTR
    28 Mar04 QZ NTR
    27 Mar04 QZ NTR
    26 Mar04 QZ NTR
    25 Mar04 QZ sub-system montly report(ppt)
    24 Mar04 QZ Today was a day for ROD, Jerry did some work on Helium sensor, neutron shieding and PMTs status checking; José and I unplugged the HV cable on the PMT that caused the christmas tree in Sector #11.
    23 Mar04 QZ Thomas put the HV ramping down and up code in the state machine, it worked fine.
    22 Mar04 QZ NTR
    21 Mar04 QZ NTR
    20 Mar04 QZ NTR
    19 Mar04 QZ NTR
    18 Mar04 QZ NTR
    17 Mar04 QZ NTR
    16 Mar04 TH Jim masked a noisy PMT in sector 10, Harness 13 Channel 2
    16 Mar04 KY 100% deadtime from DIRC...Jim removed some debugging code to solve the 100% deadtime over last few days..may not solve problem.
    15 Mar04 KY Front-end crate in sector 10 turned off while in injection mode...suspicion is that pilot rebooted drc-mon instead of dch-mon and there is a known problem where this crate turns off after reboot
    14 Mar04 KY 100% deadtime from DIRC
    13 Mar04 KY NTR
    12 Mar04 KY Code to ramp up/down the HV has been installed
    11 Mar04 KY paged for about 50 out-of-order errors for run 45971
    10 Mar04 KY Jim has put in a work around for the phi asymmetry problem and it appears to work..the sequence is reboot ROMs, configure, mask TDC's, wait 10s, unmask TDC's, then configure again.
    09 Mar04 KY Jim and Chris have reproduced the phi asymmetry problem on the test stand
    08 Mar04 KY NTR
    07 Mar04 KY NTR
    06 Mar04 KY NTR
    05 Mar04 KY NTR
    04 Mar04 KY NTR
    03 Mar04 KY NTR
    02 Mar04 QZ NTR
    01 Mar04 QZ NTR
    29 Feb 04 QZ NTR
    28 Feb 04 QZ NTR
    27 Feb 04 QZ NTR
    26 Feb 04 QZ NTR
    25 Feb 04 QZ Jim Libby gave a talk on phi sysmmetry at IR2.
    24 Feb 04 QZ NTR
    23 Feb 04 QZ NTR
    22 Feb 04 QZ several seconds 100% dead time
    21 Feb 04 QZ several seconds 100% dead time
    20 Feb 04 QZ NTR
    19 Feb 04 QZ comics runs have some spikes in sector0 and sector 11, I lowed the votage for sect11, HV group #1 from 1145 to 1000 and sect0 HV group #5 from 1145 to 1000.
    18 Feb 04 QZ BCS testing
    17 Feb 04 QZ NTR
    16 Feb 04 QZ NTR
    15 Feb 04 QZ This early morning, DIRC 100% dead time due to known(09 Feb) ODF hardware problem, lost about 2 minutes data(see shift log file)
    14 Feb 04 QZ NTR
    13 Feb 04 QZ NTR
    12 Feb 04 QZ NTR
    11 Feb 04 QZ in pc1 farm, runs 46333,46334 have dips in sector 03,04 at "nb of assoc hits per event vs. sector" plot, a "christmas" tree found at sector 11 HV group #0, voltage on this group lowered to 1000.
    10 Feb 04 QZ NTR
    09 Feb 04 QZ This early morning around 4:30, 100% dead time due to ODF problem.
    08 Feb 04 QZ NTR
    07 Feb 04 QZ people from Data Quality group pointed out the runs 44162 and 44111 have dips in Nb associated hits per event vs. sector, the reason is still unkown. Phillipe also made request at 3:30pm meeting that the PEP people reduce the machine background.
    06 Feb 04 QZ NTR
    05 Feb 04 QZ NTR
    04 Feb 04 QZ NTR
    03 Feb 04 KY NTR
    02 Feb 04 KY NTR
    01 Feb 04 KY NTR
    31 Jan 04 KY NTR
    30 Jan 04 KY NTR
    29 Jan 04 KY I gave the status report
    28 Jan 04 KY Today there was a power glitch just before 12 noon lasting for about 2 seconds. Upon coming in the almost all the electronics in the EH was off and the water pumps were not running. I restarted the water pumps immediately, but had to wait to turn on the EH hut because chilled water was not available. When the ROMs were turned on the GLINK was still down, so they were restarted and the GLINK was restored. Everything else restarted normally.
    27 Jan 04 KY NTR
    26 Jan 04 KY NTR
    25 Jan 04 KY NTR
    24 Jan 04 KY NTR
    23 Jan 04 KY NTR
    22 Jan 04 KY NTR
    21 Jan 04 KY On the ROD Feburary 18th we would like the BW doors open
    20 Jan 04 KY NTR
    19 Jan 04 KY NTR
    18 Jan 04 KY NTR
    17 Jan 04 KY NTR
    16 Jan 04 KY Something is wrong in sec. 10 ch. 32 which makes it operate at maximum operating voltage (2300 mA) for an undertermined amount of time before randomly deciding to act normal again. The alarm limit has been raised from 2300 to 2350 mA.
    15 Jan 04 KY Philippe installed a backplane in the SIAM that will allow us to ramp up/down the DIRC. There was an aomalous timing spike in the arrival time spectrum at around -80ns that was shown to due to sampling triggers in the inihibit window.
    14 Jan 04 KY NTR
    13 Jan 04 KY NTR
    12 Jan 04 KY HV alarms from sec.10 ch. 32 (currents at around 2400 above at alarm limit of 2300 lasting for about 20 seconds at a time every 4 or 5 minutes)
    11 Jan 04 KY HV alarms from sec.10 ch. 32 (same as above)
    10 Jan 04 KY paged because high backgrounds caused JAS to lose the peak in the arrival time spectrum
    09 Jan 04 KY NTR
    08 Jan 04 KY NTR
    07 Jan 04 KY High backgrounds causing Stefan's fast monitoring to lose the arrival time peak
    06 Jan 04 KY NTR
    05 Jan 04 PG HV crate 1 replaced (the one that tripped a few times last year). During the change, a HV board was damaged: HV crate 1 board#3 (325 <--> 392). I replaced another DFB: Sector 10 DFB#2 (A2 <--> 2C). The DFB in sector 11 that I replaced was actually good.
    04 Jan 04 PG A bad HV board replaced: HV crate 4 board#8 (388 <--> 356).
    03 Jan 04 PG I turned the DIRC ON (HVs, Chiller and FEE crates). I had to add a few buckets of water into the chiller. 2 new Xmas tree, both in sector 2: HV#47-R=35-C=6, and HV#48-R=35-C=24. I replaced 2 DFBs: Sector 1 DFB#6 (0D <--> B3), and Sector 11 DFBV#10 (BE <--> A9).
    22 Dec 03 QZ NTR
    21 Dec 03 QZ NTR
    20 Dec 03 QZ NTR
    19 Dec 03 GT NTR
    18 Dec 03 GT After some tests to ramp down the HV crate, all the HV currents had EPICS alarm thresholds to 2200 muA (before few channels were at 2300 mu). We had EPICS alarms during injection and Steve increased the HV thresholds to 2300 muA for these few HV channels.
    summary of the HV channels having thresholds to 2300 muA instead of 2200 muA : (sector,hv), (0,12), (1,4), (2,3), (3,28), (5,38), (5,45), (8,26), (8,31), (8,38), (8,44)
    17 Dec 03 GT NTR
    16 Dec 03 GT Paged for 10,000 in bin 10 and 100 in bin 25. bin 25 may be fixed by replacing a DFB during the Xmas shutdown. Add a note on that in the DQM short term instructions.
    15 Dec 03 GT HV crate #1 tripped. Need to be restarted manualy.
    50,000 entries in bin 10 during run 43182 (trickle run). The injection markers were not there...
    14 Dec 03 GT NTR
    13 Dec 03 GT NTR
    12 Dec 03 GT NTR
    11 Dec 03 GT During the MD, the bucking coil tripped which provocated an alarm on the B field panel. It is now fixed and the pilot could not give me more information.
    Increase again the gaz system pressure
    10 Dec 03 GT NTR
    09 Dec 03 GT NTR
    08 Dec 03 GT NTR
    07 Dec 03 GT NTR
    06 Dec 03 GT NTR
    05 Dec 03 GT NTR
    04 Dec 03 GT NTR
    03 Dec 03 GT Increase the gaz system pressure from 5.6 to 6.4
    02 Dec 03 GT NTR
    01 Dec 03 GT Increase the gaz system pressure
    30 Nov 03 GT NTR
    29 Nov 03 GT NTR
    28 Nov 03 GT NTR
    27 Nov 03 GT Thomas fixed a broken link in the DIRC short term instruction.
    26 Nov 03 GT NTR
    25 Nov 03 GT NTR
    24 Nov 03 GT NTR
    23 Nov 03 GT NTR
    22 Nov 03 GT NTR
    21 Nov 03 GT NTR
    20 Nov 03 GT NTR
    19 Nov 03 GT I've just done a list of all the HV channels having mean values of their current > 2100 muA : (sect,hv,current) (0,12,2168), (1,4,2125), (2,3,2109), (3,28,2122), (5,45,2114), (5,38,2142). I just compared these values with the corresponding values in May, I saw no changement except for the one in sector 1 hv 4. The value of the current for the last channel was 1900 muA before the 25Aug03 and 2110 muA after and the HV value shows no changement for this day.
    Philippe increased the threshold in the EPICS alarm (.db files) from 2200 muA to 2300 muA for these HV channels : (sector,hv), (1,4), (3,28), (5,38), (8,26), (8,31), (8,38), (8,44).
    18 Nov 03 GT NTR
    17 Nov 03 GT around 8 pm, the pilot paged me for over-currents in (sect 3, hv 28) and (sect 5, hv 38). Between 3 pm and 8 pm, it happened 4 and 5 times. These HV channels have mean values for their current over 2150 muA (threshold 2200 muA).
    16 Nov 03 GT There was a power outage around 8:45 am. The gaz system (on UPS) and the chiller were still alive during the outage. It ended around 11:15am. After the outage, the circulation pump did not restart. I just tryied the ON/OFF button and Bob explained me a small procedure to restart the pump (I added this procedure in the operation page). He told me that the automatic paging worked only when the power came back (it was the same for me). Maybe 3 minutes later, I restarted drc-hv and drc-mon IOCs. For information, the minimal value I saw for the water level in the epics panel in the SOB was 4.8 (meters?). Then after restarting all the FEE and HV, there was a problem with a finisar of a DCC in the crate 3 : fixed by a power cycle of the crate.
    15 Nov 03 GT I've been paged for 4 new errors in the damage bit word 24 or 26 (I'm still not sure of the numbers). Jim knows the situation and will try to understand that.
    14 Nov 03 GT NTR
    13 Nov 03 GT they started the trickle injection around 4 pm. They want to stop when 0.5 fb-1 data have been collected (First trickle run : 42149). Fast Monitoring plots look OK. Conclusion : Nothing to report for the DIRC
    12 Nov 03 GT NTR
    11 Nov 03 KY NTR
    10 Nov 03 KY NTR
    09 Nov 03 KY NTR
    08 Nov 03 KY NTR
    07 Nov 03 KY for the past three weeks there have been no HV alarms in the alarm handler..this has been due to a CVS problem. Thomas has provided a temporary solution, but the CVS problem still remains.
    06 Nov 03 KY NTR
    05 Nov 03 KY NTR
    04 Nov 03 KY NTR
    03 Nov 03 KY HV Crate #1 tripped this morning
    02 Nov 03 KY NTR
    01 Nov 03 KY NTR
    31 Oct 03 KY NTR
    30 Oct 03 KY Thomas gave the status report
    29 Oct 03 KY NTR
    28 Oct 03 QZ NTR
    27 Oct 03 QZ NTR
    26 Oct 03 QZ NTR
    25 Oct 03 QZ NTR
    24 Oct 03 QZ NTR
    23 Oct 03 QZ NTR
    22 Oct 03 QZ global calibration was failed yesterday afternoon due to a register readback problem, it now was fixed. scaler rate was modified to the same today, not increased, the detail and the decision for scaler rate will be in next week's dirc monthly report, 300K Hz for yellow warning and 500K Hz for red alarm.
    21 Oct 03 QZ DRC failed global calibration due to read damage from node 3 crate 0x80000(DFB 14 in Sector 4 failed to configure properly)
    20 Oct 03 QZ NTR
    19 Oct 03 QZ NTR
    18 Oct 03 QZ NTR
    17 Oct 03 QZ NTR
    16 Oct 03 QZ A new spike was found at Sector 4, Harness 2/ HV group 4 in the Owl shift of Oct. 16 and Swing shift of Oct. 15
    15 Oct 03 QZ A spike was found at Sector 10, Harness 13/ HV group 11, From Run 41142 to 41150
    14 Oct 03 QZ NTR
    13 Oct 03 QZ NTR
    12 Oct 03 QZ NTR
    11 Oct 03 QZ NTR
    10 Oct 03 QZ add HV crate no connection and scale zero rate as a new condion to the DIRC runnable flag; High background due to the beam injection last about 30 minutes around 2pm, scale rate was up to 1.7MHz.
    09 Oct 03 QZ Three things: a) Runnable flag modified, Chiller were added on this flag, if it tripped, DIRC becomes not runnable; b) automatic e-mail procedure were tested for several alarms(HV channel, degasing pump off); c) standalone runs were took, no big problems; work was done by PG and TH, thanks!
    08 Oct 03 QZ NTR
    07 Oct 03 BLH NTR
    06 Oct 03 BLH NTR
    05 Oct 03 BLH at 22:44 HV crate #1 went off. the drc did not go non-runnable and so the pilot began a new run. we are investigating why this happened. the first time the crate was turned back on about half of the boards remained off.
    04 Oct 03 BLH NTR
    03 Oct 03 BLH NTR
    02 Oct 03 BLH NTR
    01 Oct 03 BLH successfully removed 2 xmass tree PMTs - sector 0 row 7 column 1 and sector 5 row 38 column 22
    30 Sept 03 BLH NTR
    29 Sept 03 BLH NTR
    28 Sept 03 BLH NTR
    27 Sept 03 BLH NTR
    26 Sept 03 BLH New christmas tree sector 0 group 10 - tripped 12 HV channels.
    25 Sept 03 BLH ambient database plots show that wiener crate temperatures are tracking the temp in the detector hall.
    24 Sept 03 BLH fake degassing pump off email sent(lets blame it on the brownout). BCS siam fixed by walt. Dirc::configurable now checks to see that FEE are on.
    23 Sept 03 BLH NTR
    22 Sept 03 BLH still having inlet temp readings of 34.
    21 Sept 03 BLH changed all yellow wiener inlet temp alarms to 35. increased chiller flow from 4.85 to 5.3.
    20 Sept 03 BLH NTR
    19 Sept 03 BLH changed crate 4 yellow temp alarm to 34 degrees. outside temp is 79 degrees. maybe we should put the chiller flow back to a higher level.
    18 Sept 03 BLH looked for sector 5 christmas tree, but couldnt raise voltage high enough to get a signal in the bad group.
    17 Sept 03 BLH DIRC monthly report DIRC
    16 Sept 03 KY Raised the alarm limit in sector 10, channel 32 to 2400 because of alarms for about five minutes after injection
    15 Sept 03 KY NTR
    14 Sept 03 KY NTR
    13 Sept 03 KY NTR
    12 Sept 03 KY Crate #3 tripped due to HV channel #49 in sector 5. I turned the voltage to 10V. Also, other channels turned of accidently dur to bad PMT + bad background. There was power outage today (two of them). After the first one we had trouble restarting the frontend, but finally managed to do so.
    11 Sept 03 KY NTR
    10 Sept 03 KY The TDC spectrum is okay with a trigger mask 30FFFFF. The first events of run 4 were today and the OPR plots show no assoicated hits with tracks. This problem is under investigation.
    09 Sept 03 KY The fast monitoring shows a TDC spectrum which is shifted by about 300 ns which a trigger mask 34FFFFF. The early signal was determined to be due to early EMC triggers
    08 Sept 03 KY NTR
    07 Sept 03 KY NTR
    06 Sept 03 KY NTR
    05 Sept 03 KY NTR
    04 Sept 03 KY FEE Crate #3 shows high temperature and fluctuations..raised yellow alarm to 34 degrees
    03 Sept 03 KY Dirc-Mon IOC showed white for 16 channels. It was a coding problem which Thomas fixed.
    02 Sept 03 KY only sector 8 shows evidence of a Christmas tree taken by a standalone run. We disconnected the PMT in sector 8, row 33, col 11. Also, the voltage in sector 10, channel 32 was raised to allow for montoring of this channel.
    01 Sept 03 KY there are several spikes in the DQM plots (sectors 6,7,8). We will understand more tomorrow.
    31 August 03 KY NTR
    30 August 03 KY there is an OVC in sector 10, channel 32 that spikes about every five minutes and last for exactly twenty seconds. I decreased the voltage in that group to 10V.
    29 August 03 KY Today there was a vacuum leak in the confined space. They began to use helium until they were stopped at around 12 noon (not sure when they started). They didn't start again until Jerry okayed the way the helium was appied to the beampipe. Also, there is an issue about cabling. Perhaps the circuit breaker that feeds into the DFBs is too hot.
    28 August 03 KY NTR
    27 August 03 KY VESDA check, power on and off all electronics in EH
    26 August 03 KY NTR
    25 August 03 KY couldn't get the crates to key on properly, Thomas had to xyplex into the crates (all of them) to turn them on
    24 August 03 KY NTR
    23 August 03 KY NTR
    22 August 03 KY NTR
    21 August 03 KY in HV crate #5 replaced four HV boards with spare boards, since the spare boards were working fine, we left them in there
    20 August 03 KY NTR
    19 August 03 BH received fake SOB dump and low water level pages. not sure why ...
    18 August 03 BH NTR
    17 August 03 BH NTR
    16 August 03 BH NTR
    15 August 03 BH after turning chiller on, noticed that the flow was zero even though there was normal pressure and the pump seemed to be running. flowwheel wasnt turning. replaced.
    14 August 03 BH NTR
    13 August 03 BH changed scalar response time to ~ 2 sec
    12 August 03 BH NTR
    11 August 03 BH NTR
    10 August 03 BH NTR
    09 August 03 BH NTR
    08 August 03 BH of the six spare DCCs, one has a CanBus problem and three others have a problem with a loose socket connection. we are thinking about souldering the loose connections. right now, we only have 2 good spares.
    07 August 03 BH NTR
    06 August 03 BH NTR
    05 August 03 BH FEE fans cleaned
    04 August 03 BH South wall power turned off for ~ 1hr. gas system run off of battery.
    03 August 03 BH NTR
    02 August 03 BH NTR
    01 August 03 BH HV board #0 in HV crate 5(sector 10) replaced
    31 June 03 BH NTR
    30 June 03 BH NTR
    29 July 03 GT The DIRC has been switched on to do calibration runs and test DFBs after the reprogramming of the chips by Dominique.
    28 July 03 GT EPICS is usable. The power outage switched off the IOCs. Philippe rebooted them today and EPICS seems to be all right now.
    27 July 03 GT EPICS is still down.
    26 July 03 GT A power outage has happened without effect on the DIRC but EPICS is down. This hypernews describes the problem and relative operations going to be done.
    25 July 03 GT The power cables of DFBs has been changed. The chiller has been switched off.
    24 July 03 GT NTR
    23 July 03 GT The BCS was tested without probem.
    22 July 03 GT The chiller has been switched on.
    21 July 03 GT NTR
    20 July 03 GT NTR
    19 July 03 GT NTR
    18 July 03 GT NTR
    17 July 03 GT Dominique and Philippe finished reprogramming DFBs chips (see hypernews for more information)
    16 July 03 GT NTR
    29 June 03 QZ NTR
    28 June 03 QZ in case of voiding impact of Helium on PM, Philippe turned off the HV.
    27 June 03 QZ NTR
    26 June 03 KY DIRC status report given here
    25 June 03 KY at around midnight the crate #6 died and didn't come back. There was an access today in which it was determined that the fan was broken in the crate. Therefore, it was replaced and now works.
    24 June 03 KY Wiener crate #6 tripped once early this morning, reset it and rebooted the ROMs. A few hours later it tripped again. However, this time I was unable to reboot due to the fact that it kept tripping. This went on for about half an hour (from 7:30to 8:00). Then, after about half an hours rest, Thomas was able to get it running.
    23 June 03 KY Drc-Mon IOC needed to be rebooted
    22 June 03 KY NTR
    21 June 03 KY NTR
    20 June 03 KY NTR
    19 June 03 KY gave DIRC status of trickle injection at ops meeting
    18 June 03 KY NTR
    17 June 03 KY NTR
    16 June 03 KY NTR
    15 June 03 KY NTR
    14 June 03 KY NTR
    13 June 03 KY paged about bin 26 in JAS...ElemOverFlow about 3 entries
    12 June 03 KY trickle injection starts, DRC is fine
    11 June 03 KY NTR
    10 June 03 BH Scalar alarm rates changed. Sector 2 and 4 - red 300 yellow 250. Sector 3 red 400 yellow 300.
    9 June 03 BH thomas added guidance to alarm handler.
    8 June 03 BH NTR
    7 June 03 BH NTR
    6 June 03 BH NTR
    5 June 03 BH NTR
    4 June 03 BH NTR
    3 June 03 BH Chiller is now running at flow=5.6gpm and pressure 35. No one changed anything. Decided to keep it this way. New limits are flow=5.5+/-.6 and pressure 40+/- 15. Changed all HV fuses to 20 amps. Before sunday they were Crate #1-10 Crate #2-20 Crate #3-10 Crate #4-20 Crate #5-10 Crate #6-10
    2 June 03 BH DIRC Chiller had overpressure trip during the time it took for the central chiller to switch from maunal to auto.
    1 June 03 BH ~7:09 HV crate #1 blows a fuse. ~14:17 HV crate #3 blows a fuse. This is the one we just replaced. (maybe the SVT people used a different type of fuse??) After central chiller outage, we were delayed because our chiller refuses to turn on if temp is >7 degrees over limit. We had to temporarily reset the limit
    31 May 03 BH NTR
    30 May 03 BH 21:00 HV crate #3 dies. Turned back on - worked for a few minutes then died again. Took out 2 boards and it seemed to run. Borrowed a crate from the SVT people (but it was labeled "Polytechnique) appears to be same problem we had may 15th. LCD wasnt working -> problem with 5v power
    29 May 03 BH NTR
    28 May 03 BH changed gas pressure from 5.5 to 5.8(during MD). Dayle moved DIRC epics code to CVS.
    27 May 03 KY another red alarm from sector 10, ch. 32
    26 May 03 KY on swing shift there was an OVC from sector 10, ch. 32 (same as May 23) Imon = 2203..lasting for exactly 20 seconds
    25 May 03 KY OPR plots from most recent runs not there
    24 May 03 KY NTR
    23 May 03 KY on owl shift there was a OVC from sector 10, ch. 32 that occurred twice around 5:30, once with Imon = 3117 and another with Imon = 3107..both lasting for exactly 20 seconds
    22 May 03 KY NTR
    21 May 03 KY nada to report
    20 May 03 GV noisy PMT in sector 10, hv_group 11, harness 13 came back
    19 May 03 GV NTR
    18 May 03 GV NTR
    17 May 03 GV Repeated HV alarm for sector 0, hv_group 12. The alarm level for this channel had been reset to the default value, 2200 muA, after the power outage, and not to 2500 muA, to which value it was changed after the HV increase. This is now fixed in the IOC. It has to be changed in the database for the next IOC reboot.
    16 May 03 GV NTR
    15 May 03 GV power outage in the EH: water system not affected; low voltages came back OK; however high voltage power supply for crate 3 (sectors 4 and 5) did not come back and had to be replaced by the spare. There was a T0 jump of 2 ns in sector 8 in the first online calibration after the power outage.
    14 May 03 GV new noisy PMT in sector 10, hv_group 11, harness 13: intermittent problem
    13 May 03 GV NTR
    12 May 03 GV NTR
    11 May 03 GV adjust the bar box manifold pressure in rack 38 (from 7.00 to 6.60)
    10 May 03 GV NTR
    09 May 03 GV There was a fake "degazing pump off" automatic page early this morning following a reboot of cen-mon. It appears that every reboot of cen-mon will trigger the automatic page with the current script.
    08 May 03 GV NTR
    07 May 03 GV NTR
    06 May 03 GV increase scaler alarm and warning thresholds to 250 kHz and 200 kHz respectively on all sectors (they were 180 kHz and 150 kHz on most sectors); also add a new script to send an automatic page to Bob Reif and a mail to babardrc when the degasing pump goes OFF.
    05 May 03 GV new T0 jump: sector 8 is back to its previous value
    04 May 03 GV high background during the week-end, many "out of synch"
    03 May 03 GV first T0 jump since fiber swap, sector 8 is affected; so the problem follows the fiber and is due to the LED or driver part
    02 May 03 GV NTR
    01 May 03 GV DIRC status report given today at ops meeting
    30 Apr 03 GV NTR
    28 Apr 03 BH NTR
    27 Apr 03 BH NTR
    26 Apr 03 BH NTR
    25 Apr 03 BH NTR
    24 Apr 03 BH NTR
    23 Apr 03 BH DIRC Epics screens changed
    22 Apr 03 GV switch LED fibers used for the online calibration: sector 7 with sector 9 (those were switched by mistake during the ROD on March 27) and sector 8 with sector 10 (to study of the T0 jumps observed in sector 10 follow the fiber)
    21 Apr 03 QZ last night, DRC white alarm which is due to cooling problems in the EMC forward barrel power supply rack.
    20 Apr 03 QZ NTR
    19 Apr 03 QZ NTR
    18 Apr 03 PG I lowered the HV of Sector 0 HV_group 12 (which was giving transient alarms during injection). I lowered back to its nominal value: 1436 V --> 1359 V. The current decreased from 2169 muamps down to 2052 muamps. We should not get any more alarms.
    17 Apr 03 QZ Intermittent alarm last night. add one buckle of water to chiller.
    16 Apr 03 GV Intermittent alarm (Imon>2200) during injection for sector 0, hv_group 12 and sector 3, hv_group 28. These two hv_groups have both high HV: 1436 and 1404 V.
    15 Apr 03 GV NTR
    14 Apr 03 GV New HV settings where loaded: the gain loss correction is now done hv_group by hv_group
    13 Apr 03 GV NTR
    12 Apr 03 GV NTR
    11 Apr 03 TH Changed key in order to fix DAC settings for the four recently changed DFBs.
    10 Apr 03 GV NTR
    09 Apr 03 GV NTR
    08 Apr 03 GV NTR
    07 Apr 03 GV NTR
    06 Apr 03 GV NTR
    05 Apr 03 GV DRC-MON was rebooted
    04 Apr 03 GV IOCs rebooted to include minor epics change; BBR-DEV00 died and was replaced
    03 Apr 03 GV NTR
    02 Apr 03 GV HV raised by 43V on all PMTs
    01 Apr 03 QZ NTR
    31 Mar 03 QZ NTR
    30 Mar 03 QZ NTR
    29 Mar 03 QZ NTR
    28 Mar 03 TH Here is a detailed recap of the ROD activities yesterday. Thanks to all that participated (in random order: PG, AKY, QZ, BH, VS, JV, Georges Vasileiadis, Karl Bouldin, Svend Jensen, Sam Zalog + BCS and hall crew)
    • The following PMT have been unplugged:
      • Sector 0 Row 36 Col 21 Harness 11 Channel 56 HV group 49
      • Sector 3 Row 24 Col 11 Harness 6 Channel 38 HV group 24
      • Sector 5 Row 22 Col 16 Harness 9 Channel 34 HV group 28
      • Sector 10 Row 22 Col 11 Harness 6 Channel 34 HV group 24
      • Sector 10 Row 14 Col 8 Harness 6 Channel 16 HV group 15
      The HV group #1 in sector 11 was ok, so the intermittently noisy PMT could not be disconnected. Just for completeness: During the recent week, the following two channels were masked:
      • Sector 6 Row 1 Col 12 Harness 11 Channel 0 HV group 0
      • Sector 7 Row 6 Col 14 Harness 11 Channel 6 HV group 0
    • The following DFBs have been replaced:
      • Sector 3 DFB 2 (from serial 3E to serial 0B) because of OutOfOrder errors
      • Sector 10 DFB 9 (from serial A9 to serial 16) because of configuration readback errors
      • Sector 10 DFB 13 (from serial AD to serial 5E) because of configuration readback errors
      • Sector 11 DFB 3 (from serial B3 to serial 06) because of configuration readback errors
    • Installed a new water sensor in the water reservoir of the DIRC chiller. The new sensor is connected to the chiller electronics and will report as Level Warn alarm to channel 2(from 1) of SIAM 0 in the DRC-MON crate (right most slot). The original water level switch will report to channel 1 of a new SIAM in the slot left of the original SIAM. The original SIAM has an address setting of A20,A21 set. The new of A8,A20,A21 set.
    • The new backup cooling system (BCS) has been connected to channel 3 of SIAM 0. It will normally be in error state and has to be inhibited. When switching from the DIRC chiller to the BCS, channel 1 and 2 have to be inhibited and the inhibit of channel 3 has to be removed.
    • A picoAmeter has been installed on base of PMT Sector 4 Row 14 Column 0 Harness 0 Channel 12 HV group 10 in order to measure the current seen by the PMT during injection.
    • New power supply cabling for all Wiener Front End crates has been prepared and partially installed. The installation could not be finished and we are still using the old lines.
    • The Backup Cooling System has been checked. It is possible to get the required pressure but it is not clear what the flow is. The flow meter in the return line gave a value of 40 gpm instead of the normal 4.8 gpm. It is very likely that this is due to the not-yet calibrated flow sensor readout.
    • Several TDC calibration and ADC calibrations at different voltages have been taken. Preliminary results have been posted to the DIRC Hypernews by Vasia.
    • Installed Steve's new DRC-HV and DRC-MON software including alarm handler to handle the new signals and to improve automatic messages.
    • Did a visual check for water leaks. None could be found.

    Things to do for the next ROD or long MD:
    • Add EPICS controls for new signals.
    • Finish the power cableing (needs ROD).
    • Add a sensor to monitor the SOB water system pump and connect a VSAM channel in CEN-MON. Adding corresponding scripts and Alarm handler/EPICS changces.
    • Check new configuration readback problem in DFB 8(?) Sector 10 (needs ROD).
    • Check noisy channels in DFB 3 Sector 11 (needs ROD).
    • Exchange HV create for sectors 0 and 1 with spare.
    • Change DRC-MON for HV ramping and to tests.
    28 Mar 03 TH Raised HV group #43, #47 in sector 11 to nominal voltage again as it seems to be electronics noise problem.
    28 Mar 03 QZ Lowered HV group #43, #47 in sector 11 200 volts.
    27 Mar 03 QZ
  • replaced 4 DFBs -- 3 with configuration readback problem, 1 with out-of- order problem -- each of these 4 are now OK
  • have a new configuration readback problem in sector 10 -- this was found after the door had been closed (and could not be opened again)
  • unplugged 5 PMTs and now can turn on their HV groups again -- a sixth PMT has been exhibiting intermittent problems but was OK today and so it was not clear which PMT needed to be unplugged (however, it is far on the outside and is thus not a particularly bad problem)
  • checked for water leaks -- none were found
  • installed a new water level sensor -- now there is a low water level alarm -- it is already in the ALH but needs to put into EPICS
  • installed an picoAmeter on a PMT to monitor currents drawn during injection in order to determine if HV needs to be ramped down during injection
  • did ADC calibrations at nominal voltage, nominal+20V and nominal+40V -- this was done in order to understand how much voltage increase is needed to offset the gain losses seen over the lifetime of the DRC
  • tests for the BCS were OK -- however, the flow rate is a factor of 8 too high and needs to be properly calibrated
  • were not able to finish installation of new power cabling because of the backward east door bracing breaking -- the new power cables are lying on top of the BW door
  • 26 Mar 03 BH NTR
    25 Mar 03 BH sec 7 HV 0 occupancy was 2X normal. switched off HV and restarted JAS, but occupancy peak did not go away. Conclusion - noise is caused by electronics and is not affected by HV.
    24 Mar 03 BH NTR
    23 Mar 03 BH NTR
    22 Mar 03 BH NTR
    21 Mar 03 BH decided problem in Sec 6 HV 0 is due to electronics noise. HV turned back on. Thomas will mask the channel.
    20 Mar 03 BH new Xmas tree in sector 10 HV #15
    19 Mar 03 BH adjust the bar box manifold pressure in rack 38 (from 7.15 to 6.5). added 3 liters to the DIRC chiller
    18 Mar 03 GV lower high voltage of sector 0, hv_group 41 by 200V; hv_group 49 (close to 41) in sector 0 also noisy, this is probably due to the same bad PMT
    17 Mar 03 GV new noisy PMT in sector 0, hv_group 41; adjust the bar box manifold pressure in rack 38 (from 5.84 to 6.29)
    16 Mar 03 GV NTR
    15 Mar 03 GV usual noisy PMT in sector 7, hv_group 0
    14 Mar 03 GV NTR
    13 Mar 03 GV adjust the bar box manifold pressure in rack 38 (from 5.9 to 6.2)
    12 Mar 03 GV NTR
    11 Mar 03 GV during the night, HV crate #1 tripped, this did not pause the run
    10 Mar 03 TH Philippe and Georges noticed some new dead channels in the HV. I tried to raise the voltages for all of those with the following result:
    • Sector 5 HV group 28: When switching on, it immediately trips. It also trips the neighboring group 27. Looks like a new X-mas tree, leave it off.
    • Sector 5 HV group 48: Got it resurrected.
    • Sector 11 HB group 1: As soon as it goes to 500V, it trips. However, it does not cause any problems in the neighboring PMTs. Putting it to lower voltages and keeping it there seems to give the PMT a chance to settle, as the currents go down.
    09 Mar 03 GV in sector 11, hv_group 1 is at 10V
    08 Mar 03 GV site-wide power outage: restart chiller; restart SOB pump; swith off degazing system pump which died, Bob came to change it, will order new spare on Monday; when cooling came back to the electronic house, reboot IOCs, switch on low and high voltages.
    07 Mar 03 GV NTR
    06 Mar 03 GV noisy PMT in sector 6, harness 11, hv_group 0: HV lowered by 20V (already lowered by 80V on Feb 27)
    05 Mar 03 GV NTR
    04 Mar 03 QZ NTR
    03 Mar 03 QZ NTR
    02 Mar 03 QZ NTR
    01 Mar 03 QZ last night DRC white alarm due to IOC disconnection, Philippe reset it.
    28 Feb 03 QZ NTR
    27 Feb 03 QZ 1.DIRC failed the new code global calibration, now still used the old code; 2.DIRC FEE alarm was due to DAQ crates high temperature; 3.TDC hits in sector 6 harness 11 got worse, Thomas lowed the high voltage. 4. Data flow problems, most happened at Sector 3 DFB 2, it is an old problem, Vasia suggested to replace this DFB.
    26 Feb 03 QZ We looked into the Water Chiller for leakage and the sensors, we needed one more water level sensor to be installed.
    25 Feb 03 QZ NTR
    24 Feb 03 QZ last night, Drc Chiller power failure alarm, chiller water level was near warn level, refilled water.
    23 Feb 03 QZ NTR
    22 Feb 03 QZ NTR
    21 Feb 03 QZ NTR
    20 Feb 03 QZ NTR
    19 Feb 03 QZ OutOfOrder error in Sector 3 dfb 2
    17 Feb 03 BH Occupancy in HV 0 SEC 6 reaches 4 times normal. Channel turned off from ~3:30-13:00. Still appears to be coming from one PMT only.
    16 Feb 03 BH i noticed the occupancy in HV group 0, sector 6 was about 2 times normal. the hit map in the spew file looks like it is all coming from 1 PMT. i left a note telling the DQM to keep an eye on it.
    15 Feb 03 BH High voltage for sectors 0 & 1 went off. lost 1 hr of data taking. shifter did not notice that something was wrong because everything was green on the DIRC detector control panel. we need to change this panel and perhaps even send a dataflow error to stop the run automatically whenever the crates go off.
    14 Feb 03 BH NTR
    13 Feb 03 BH NTR
    12 Feb 03 BH NTR
    11 Feb 03 BH HV group 1 in sector 11 was turned back on today. it seems to be working fine.
    10 Feb 03 BH NTR
    09 Feb 03 BH it looks like the chiller water level drops ~1/2 cm per week. i will refill it and start taking weekly measurements
    08 Feb 03 BH NTR
    07 Feb 03 BH NTR
    06 Feb 03 BH DIRC status report for operations meeting Free PDF file! Highlights- 1) Out of order problem fixed. 2) Changing DCC#10 did not stop T0 jumps.
    05 Feb 03 BH NTR
    04 Feb 03 BH NTR
    03 Feb 03 BH HV group #24 in sector 3 tripped. Scalar rate was 160K at 400V with no beams. Noise in sector 7 is from new PMTs - harness 11 #5 and #6, not from previously disconnected PMT #0. maybe DFB #11 is bad
    31 Jan 03 BH NTR
    30 Jan 03 BH NTR
    29 Jan 03 BH NTR
    28 Jan 03 BH NTR
    27 Jan 03 BH found 3 DFBs in sector 10 and 11 which consistely force other DFBs to report configuration data on the wrong(late) clock tick - plan to replace during next access
    26 Jan 03 BH New HV trip in sector 10 #24. made a christmas tree when set to 900V
    25 Jan 03 TH The HV group has tripped again. Raising it to even 600V makes it trip. Left it at 10V, something for the next ROD.
    24 Jan 03 TH found one tripped HV group (#1 in sector 11). Tried to raise the voltage but found over current at 700V. After trying to get as high as possible starting at 600V, suddenly the problem went away and I could raise voltages to 1100V.
    24 Jan 03 BH replaced DFB#8 sector 8 because of out of order errors - replaced DCC sector 10 because of 1 ns jumps in T0 calibrations - disconnected 4 PMTs from high voltage - Sec 2 row 31 col 20 harn 11 chan 45 HV 34 because HV was tripping -- Sec 7 row 1 col 6 harn 7 chan 1 HV 2 because it was noisy -- Sec 7 row 1 col 12 harn 11 chan 0 HV 0 because it was noisy -- Sec 10 row 20 col 3 harn 2 chan 24 HV 25 because it was noisy
    23 Jan 03 BH Both valves leading to the DIRC chiller replaced
    22 Jan 03 GV NTR
    21 Jan 03 GV NTR
    20 Jan 03 GV NTR
    19 Jan 03 GV NTR
    18 Jan 03 GV NTR
    17 Jan 03 GV NTR
    16 Jan 03 PG On Jerry's request, I changed the thresholds on the Sector 3 Scaler: Yellow: 150000 --> 180000, Red: 180000 --> 200000. Due to high background, this scaler gives yellow and Red alarms quite often.
    15 Jan 03 PG NTR
    14 Jan 03 PG NTR
    13 Jan 03 PG NTR
    12 Jan 03 PG NTR
    11 Jan 03 PG NTR
    10 Jan 03 PG NTR
    09 Jan 03 PG NTR
    08 Jan 03 PG NTR
    07 Jan 03 TH NTR
    06 Jan 03 TH I masked the channel in sector 3 DFB 9 channel 53 which is disconnected from the PMT and still noisy.
    05 Jan 03 TH Today the chiller SIAM went into alarm state twice causing the FEEs to be switched off. The problem cured itself within seconds, so switching off the FEE was not necessary (the alarm was cleared even before the FEEs were off). I went to IR-2 to check and found the chiller to be close to (and sometimes in) the level warn state. This does not affect the chiller performance and is a warning only. However, it still causes an DIRC chiller alarm. I refilled the reservoir. There are two possibilities:
    a) we have a leak but I cann't see were
    b) I filled the chiller on Thursday only barely above the level warn limit. I noticed the lower cooling capacity in sector 2 to 4, so maybe there was some air in the system. If this air has found its way out of the chiller, it might have lowered the water level getting the chiller just barely in the level warn limit.
    We will have to check the water level more carefully over the next few weeks.
    04 Jan 03 TH NTR
    03 Jan 03 TH Found HV group #34 in sector 2 at 10V. Trips as soon as it is above 800V. Put this group to 750V.
    02 Jan 03 TH SEM filled up chiller with 4 gallons. Could switch DIRC on without problems.
    DCH wanted to do some noise measurements, so I switch FEE and chiller back off. Noisy DCH channels did not change, i.e. DIRC is not causing the noise. When switching back on, the chiller was again in level warn and water was running out of the back of the chiller. It looks as if the spare water container is not big enough to hold the water that flows out of the piping into the chiller when it is switched off. So, please remember that you have to top-off the chiller (when it is running) after switching the chiller back on !
    We have three "warm" FEEs, inlet temperatures for sectors 2, 3, and 4 went into the yellow range. I (temporarily) raised the limit from 33 to 34 deg. I assume this is due to the chiller still trying to cool down all heat exchangers. I will monitor the temperatures.
    After three hours of running, the temperature of the three sectors has decreased to by 2 degrees. I put the old limits into place.
    01 Jan 03 TH Tried to switch on DIRC, HV went fine, chiller has level warn light on, causes inhibit of FEEs. Informed SEM.
    12/18/02 MK Glink problem in sec 4, replaced the DCC, checked fiber light attenutaion, fiber is fine, did a stand alone calibration and random trigger run.
    12/15-12/17/02 MK DRC is fine
    12/14/02 MK Due to the power outage, the relay on top of the detector which talks to the front end stopped working, had  to call an
    electritian who has put in a new rely.  The power suppy for this rely is in the DCH rack in the electronics hut.
    10 Dec 02 QZ we addressed the problem of loss photons in sector 5 in the operation meeting
    09 Dec 02 QZ NTR
    08 Dec 02 QZ NTR
    07 Dec 02 QZ NTR
    06 Dec 02 QZ out of sync error. ODF people will investigate this problem.
    05 Dec 02 QZ NTR
    04 Dec 02 QZ there was power outage in IR2, the circulating pump did not restart by itself(manual), this caused a SOB dump alarm. the degasing pump died, it was replaced by the spare one and Bob Reif odered new one and installed. The backward door was open today, Philippe replaced one DFB Board and unplugged HV cable for one PMT in the sector 2(Xmas tree) and one in sector 3(noise channel). but this noisy channel does't disappear. 
    03 Dec 02 QZ NTR
    02 Dec 02 QZ Tow problems, 1.dataflow problems in dirc. 2. one noisy channel in sector 3
    01 Dec 02 QZ NTR 
    30 Nov 02 QZ new spurious noise channel in section 3 and in run 32765 FX status word has some value at bin 30.
    29 Nov 02 QZ Decrease the inlet pressure of dirc gas system this evening. 
    28 Nov 02 QZ NTR
    27 Nov 02 QZ NTR
    26 Nov 02 QZ IOC was rebooted and Dirc passed the new release calibration. 
    25 Nov 02 QZ NTR
    24 Nov 02 QZ NTR
    23 Nov 02 QZ Some dead PMTs appeared in section 3 in low row range. 
    22 Nov 02 QZ Filters for dirc barbox gas system were replaced this morning(see babar log book), and dirc IOC was rebooted this noon(see babar log book).QZ failed to response to page,Hot channel appeared in DIRC plots "Occupancy per Harness" in sector 7( due to noise channel). 
    21 Nov 02 QZ NTR.
    20 Nov 02 QZ NTR.
    19 Nov 02  PG Last night, the IOC of the gas system (drc-gas) was not responding (see babar log book). This morning the IOC was replaced by a new one. It had to re-loaded with correct boot parameters: it took some time as some of the parameters were out-of date. We find the correct setting with the help from Dayle.
    I took a random trigger run: no problems. 
    18 Nov 02  PG  NTR. 
    17 Nov 02  PG  NTR. 
    16 Nov 02  PG  Few problems today. First we got an "GLSTAT"error on sectors 8 and 9. This was coming from the ROM. It has been (the ROM) changed by Chris. 
    Then, we got no data in sectors 2 and 3. Called Chris again. After swapping and re-swapping the fibers, the problems dissapeared. Chris swapped the ROMs (sector 0-1, sector 2-3) to see if the problem occurs again and if it follows the ROM. Right now everything looks fine. 
    While debugging the first problem, Chris has seen many "damages"in the data. He sent a message to Vasia. Those "damages" were not seen again afterwards, while debugging the second problem. 
    15 Nov 02  PG  Vasia took a bunch of high statistics LED runs for ADC calibrations, with and without LED pulsing. 
    14 Nov 02 PG  This morning, all crates in electronics house were switched OFF for purge/cleaning of rack cooling water. After re-starting HV crates, 2 boards in crate 1 (Bd 7 and 8) were not working: the HV values displayed did not make any sense. We formatted the eeprom of the 2 boards and it fixed the problem. 
    13 Nov 02 PG  NTR. 
    11 Nov 02  PG  We actually had 2 Xmass trees: the one in sector 11 and another one in sector 0: HV35, Row=27, Col=14. These 2 PMTs have been disconnected from the HV. The Xmass trees have disappeared. 
    10 Nov 02  PG  We have discovered a HV channel in sector 11 (channel 36) that was at 8V. It didn't give an alarm, since it was not at 0V. After reloading the nominal value, I observed a X-mass tree in this sector 11. We will decide tomorrow if we open the doors to disconnect this PMT (Row 28 and Col 5). 
    09 Nov 02  PG  Two spurious noisy channels. One in sector 6 and one in sector 7. 
    08 Nov 02  PG  Almost all FEE crates tripped last night at 6:00pm. Other crates in the experiment tripped in the same time. It was probalby due to a loss of the main AC power.
    The pump of the SOB water system tripped at midnight. The flow was going down pretty quickly. Had the shitfter to call Bob Reif. Bob understood very quickly what was the problem.
    Filters in gas system will be changed on Monday. 
    07 Nov 02  PG  Flow controller installed this morning on the chiller.
    After restarting FEE, one DCC (sector 4) did not come up. It's the second time in 3 powercyclings that it happens to this DCC. I changed it and did the exercise of powercycling a few times. I didn't see any problems. Vasia took a LED calibrations:OK. I took a random trigger run: OK. We will leave this DCC in the detector. 
    Summary: DCC 25 changed by DCC 31. 
    06 Nov 02  PG  All FEE crates went down today at 11:00 am. Mystery. 
    05 Nov 02  TH  Steve was out sick yesterday and today, will change gas filter one of the next days.
    Bob Reif changed outgoing water resistivity sensor yesterday
    Fire alarm in EH, had to reboot drc-mon, CAEN HV crates and drc-hv (twice). Everything back online. 
    04 Nov 02  TH  Flow meter in DIRC water chiller has been replaced by HVAC. No flow control valve is available, has to be ordered.
    We are going to test the spare DFBs in the system today.
    Steve Dardin will replace the filter in the nitrogen gas system today. 
    01 Nov 02  TH  Steve Dardin added restraints to gas flow for humidity sensors in order to equalize flow in the 12 sensors. 
    31 Oct 02  TH  Scheduled water chiller maintainance for Monday Nov. 4th, 7:30am. Flow meter will be exchanged 
    30 Oct 02  TH  Adjusted gas flow
    Discussion with Chris O'Grady and Vassia led to increasing the time out for the minor transition by 5 seconds. However, we should try to find a better solution for this. 
    26 Sep 02  TH  Fans of all Wiener crates at the detector have been cleaned. 
    The chiller is in principle back in operation but they might have to drain it again in order to install the reinstalled flow panel. 
    17 Sep 02  TH  Chiller and super heat exchanger work done. They would like to make a pressure test with gas. Does anyone know the maximal pressure of our system ? 
    11 Sep 02 TH  Philippe and Georges and I removed the DFBs to be ready for shiping them to Naprotek. 
    10 Sep 02  TH  Philippe and Georges and I switched the FEEs off and unpluged all the harnesses from the DFBs. We are waiting for the foam so that the DFBs can be taken out and stored for shipping to Naprotek. 
    05 Sep 02  TH Philippe unplugged the PMT that my analysis suggested to be the source of the X-mas tree: sector 9, row 6, column 9 (numbering from 0), HV group 7, harness 8. After that, HV could be raised to full 1099V for this channel without triping. 
    04 Sep 02  TH  Philippe and Georges inhibited the SIAM so that FEE can be switched on with chiller being off. Switched FEE back on. 
    03 Sep 02  TH  DIRC HV MON crate will be moved to a lower position in the same rack. 
    02 Sep 02  TH  Switched off the FEE and the chiller so that the pipes for the super heat exchanger can be attached to the DIRC circuit. 
    28 Aug 02  TH  We switched on the DIRC HV, chiller and FEE. While switching on, one of the HV channels triped (crate 5, board 5, channel #7) and switching it on causes also three other channels to trip. We lowered the voltage of this channel to 950V (instead of 1099V). 
    We tried a global calibration of the DIRC but it failed. Three problems seem to be around. At first, our calibration rejects a large amount of fits and, therefore, prints a lot of debug output. This causes the next transition to fail (dropped contribution). Vasia is checking this. The next problem causes dropped contribution on the final transition causing the calibration to fail. Chris O'Grady is working on this. It might be faulty hardware in the newly split crates. The last problem prevents to restart the calibration. This is also a ODF problem and Chris is looking into this. 
    22 Aug 02  TH  Still shutdown, last week we had a power outage in IR-2. The water pump does not automatically go on after a power outage. This was noticed only three days later by Jerry. The water in the Stand-Off Box dropped to a level of about 4m due to the small resistivity and pH measurement pipe which is not closed. The corresponding EPICS alarm is not enabled due to previous spurious alarm caused by false readings. 
    17 Jul 02  TH  Philippe and I did an inspection of the east half of the SOB to see which of the dead or inefficient PMTs are connected and which are disconnected.
    Jerry had a look into the SOB and visually checked the PMTs. No obvious change in water/light catcher/PMT quality.
    We removed the PMT base of the PMT in sector 8 row 40 column 29 (on the edge of the display). This PMT was very inefficient before but was not disconnected, i.e. did not die as X-mas tree. We would like to see if there is any problem in the base. Jerry will analyse the base. We put a new base on as a replacement.
    10 Jul 02  TH  The chiller has been serviced by the HVAC group and the head pressure controller was replaced. We leave the chiller up and running to see if the repair was successfull.
    In addition, Joe and I did an inspection of the west half of the SOB to see which of the dead or inefficient PMTs are connected and which are disconnected.
    01 Jul 02  TH  PEP has switched off for the summer. This morning I took three standalone runs. The old recipe was no longer working, I updated the documentation to the new procedure. One thing that is obvious is that we have to update our babardrc account to SunOS58. I will do this during the summer.
    After the standalone runs, Philippe and I switched off the HV and Wiener crates.
    30 Jun 02  TH  Last day of run 2, NTR.
    29 Jun 02  TH  Today was a trickle test with the full detector on. PEP had problems and we only had 3 minutes trickle within 7 hours until the test was canceled. DIRC was fine.
    28 Jun 02  TH  During the evening shift, the shift leader reported that scaler #3 went up to 240 kHz and scaler #4 to 200kHz during normal, high luminosity running. I raised the threshold of the yellow alarm of scaler #3 from 220kHz to 240kHz and the one for scaler #4 from 200kHz to 220kHz.
    Results from Tuesdays trickle injection: No problem for the DIRC. For details see Hypernews posting
    Jerry discussed with MCC people and they found a collimator that was open a bit too much. After closing it, the situation improved, I still left the limits at the higher thresholds.
    27 Jun 02  TH  I found the DIRC scaler alarms to be disabled in the shift leader alarm handler. This was done by the shift leader on Monday evening as he was obviously annoyed by the alarms. The reason for the alarms is the strongly increased background due to a new PEP tune. Going back to the old tune, however, decreased the luminosity by 20%.
    I raised the threshold of the yellow alarm of scaler 3 from 200kHz to 220kHz.
    26 Jun 02  TH  DIRC sees some high backgrounds, otherwise fine.
    24 Jun 02  BTM  Gas pressure was high - meter read 7.24. Lowered and tweaked gas flows.
    23 Jun 02  BTM  I turned off the e mail message "HV Trip" since it seemed to be spurious. Had a call about 10:30 pm that there was a yellow alarm on sector 5 FEE which had been there "about three hours". On checking, there were no obvious problems, and my alarm handler was clear. I suggested the shifter restart the alarm handler at IR2 which still had the yellow alarm. This cleared it. I conclude that my change to the script that sent e mail had triggered the yellow alarm.
    11-22 Jun 02  BTM  NTR 
    10 Jun 02  MP  NTR 
    07 Jun 02  MP  There was a red alarm on the DIRC scalers during the DAY shift. It was during injection, but DAQ was running without any beams for some tests. So nothing to worry about. 
    06 Jun 02  MP  NTR 
    05 Jun 02  MP  NTR 
    04 Jun 02  TH  NTR 
    03 Jun 02  TH  NTR 
    02 Jun 02  TH  NTR 
    01 Jun 02  TH  NTR 
    31 May 02  TH  NTR
    30 May 02  TH  DIRC calibration failed as the calibration node was not cleaned up after SunOS 58 tests. 
    02/05/28 TCP
    02/05/27 TCP At 11:51 DQM noticed a major drop (to ~1-2% of normal) in the sector 11 occupancy. Thomas (who was also paged, as I would out of my office) suggested ending the run, unconfiguring-configuring, and that cured the problem.
    From the fast monitoring I can see, that this problem was there from the beginning of the run, and so run 28605 should be marked flawed and 28604 should be checked in OPR.
    Seemingly, this was a configuration problem of the evil kind, as some but not very many hits actually were recorded. CMlog was down, so we don't know, what the error message there was, if any.
    Regarding the detector itself, nothing to report.
    02/05/26 TCP Nothing to report.
    02/05/25 TCP Nothing to report (apart from an ill commissioner!).
    02/05/24 TCP Nothing to report.
    02/05/23 TCP Nothing to report.
    02/05/22 MP Xmas-tree in HV channel 52 sector 2, George saw it monday morning but he did not mentionned it and put 10 V, so it appeared as dead on spew hist files. Now we put 860 V.
    02/05/21 MP NTR
    02/05/20 MP As the spare crate (from SVT) did not solve the problems, we investigate in the different boards. Two of them were changed (the first one was broken during the night). But the scalers values for channels 6 and 7 were still at zero. We then thought it was wise to return to the DIRC crate which was removed and supposed to be repaired (the 5 Volts line of this crate was not working) but it did not help. At last, we decided to put the DIRC spare crate (third try !) ==> run 28422 started at 13:48 ===> DIRC is back to normal : good data ! 
    20 May 02  TH  Tonight we had some action. At about 1am the scalers 6 and 7 were reading 0. I checked EPICS and found the crate 4 in NC. When I came down to IR2 it was already clear that the HV PS was off, i.e. the key was still in on position, the Main and 310V light were on but the the rest (LCD display controll light, HV on light) were off. On the back the main red lamp on the PS was on but all red lights on the HV board were off.
    Power cycling (with the CAEN connection off) did not change the status, the fuses were ok. After calling quite a few people and we finally came to the conclusion that it was the main board, i.e. we had to change the whole HV crate. It took pretty long as the boards are not easy to handle, get in and get out and Giovanni got his finger badly hurt.
    The scalers went 0 for another time when we just thought it was fixed. We had to reboot the Power Supply and it worked ... for 10 to 15 min ... so obviously no fixed solution.
    The crate is a new SVT crate with updated communications chips. Giovanni and Ray just changed the chips so that we downgraded the PS.
    This did not solve the problem and we tried to run the crate with only four of the HV boards in. Goerges arrived and we did some test, with all cards in but board 8 it was running fine. With all cards in but board 5 (which as 16 channels used and thus draws more current than board 8 which has 8 channels used only), it switched off nearly immediately. We therefore replaced HV board 8.

    I would like to thank all the people that helped:
    The shift crew : Sandrine Laplace, Oliver Buchmueller
    RC: Usha
    Ray Rodrigues, Maha, Stefan
    HV crate expert: Giovanni Calderini
    and those that would have helped in case we could have reached them.
    In addition, two DIRC proxies are dead and/or not runnable but DIRC is perfectly green and there are no DIRC alarm.

    05/16 MP NTR
    05/15 MP NTR
    05/13-05/14 MK DIRC is FINE!!
    05/12 MK & SMS Stefan is trying to improve the fast mon so we did some test to incorporate beam conditions and scalar rates per sector, got stuck on shell scripts but are working on it. There is already a water monitor that commissioners should look at the entry zero means no data base entry was found at that time.
    05/10-05/11 MK NTR
    05/09 MK Took a stand alone run, seen that reducing the volatge in sector 2 does not really have a big impact on efficiency.
    05/06-05/08 MK NTR
    05 May 02 TH 12:30: I was paged as the drc-mon IOC died, I rebooted it and everything looked fine for me. The shift leader still had a white mark on the alarm handler. I asked him to restart the alarm handler, after doing that, the alarm was also gone on this console.
    05-02-02 MK Thomas and I did a study of Hv Vs Current for channel 52 and compared it with channel 48, the chiller maintainance work was done and we need to replace the head pressure controller, which should be done during the shutdown.
    30 Apr 02 TH Troels noticed a missing HV channel in sector 2. I checked and found it at 10V setting, which seems to have been the automatic trip recovery procedure (By the way: We should implement an automatic email-notification to the commissioner, I will talk to Georges). I checked the SPEWhist files and it seems to have happened during run 24913 around 20:36 yesterday. Ingrid (pilot) said that she saw a few alarms from DIRC (scalers, etc) but all went away automatically; so she might have missed a HV alarm. During a top-off I switched the HV channel 52 in sector 2 back to first 500V, then to 900V where I already saw a current value of 2180 (close to the 2200 limit). Putting the HV higher or to the default of 1040V produces an OVC warning, so I switched back to 900V and it is stable for quite some time. I will monitor it throughout the evening from home.
    Addition: the current rose by 8uA between 5pm and 10pm, so I reduced the voltage by another 10V to 890V. The current current is 2164uA which will hopefully work till tomorrow. Otherwise, it looks very promising: Looking at the FastMonitor, the 16 tubes are back and look normal ! However, there is no obviously noisy/hot tube and thus we do not know what causes the higher currents. Some studies needed during the next ROD/MD.
    30-04-02 TCP
    29-04-02 TCP Got paged again in the morning because scaler 3 was again high. Perhaps a small entry in the short term instructions would be good. NtR.
    Comment by Thomas: I asked for such an entry in the short term instructions on Thrusday and it was added on Friday (26 Apr 02)... but obviously it did not help.
    28-04-02 TCP Got paged twice - first time because of downtime and secondly because of yellow alarm scalar rates, which were not persistant. As usual channel 3 was the highest.
    27-04-02 TCP Stefan made slight but to the point corrections to the JAS DIRC instructions. Nothing to Report on the DIRC itself.
    26-04-02 TCP Got paged late night because the FEX bin 10 was ~300. There were large backgrounds, so nothing to worry about. However, I will contact Stefan regarding changing/adding the JAS instructions. DIRC itself: NTR.
    Comment by Thomas: I asked Stefan to change this about a month ago, but did not get any reaction till I called this morning. Stefan will change the plot such that it will show the relative number of error instead of the absolute number. After a few minutes this will allow the DQM person to make a trival decision about whether to call or not.
    25-04-02 TCP Gave monthly report. Trickle window seemed to work. Nothing to report.
    24-04-02 TCP PEP-II having MD. Took two stand-alone-runs with the usual time window, and after some troubles solved by Vasia, another one with double the time window intended for use during tricle injection.
    23-04-02 TCP Nothing to report.
    22-04-02 TCP A hot PMT (sector 9, harness 10, HV group 1) was reported by DQM. Thomas looked back, and this channel has been noisy in ~15 runs this month, but far from always. Also, a FEE alarm persisted to return for 10 minuttes Sunday night. The pilot said, that it was mainly in sector 3, but not always.
    Comment by TH: As I read it, it was a scalar alarm during high background, so perfectly normal !
    22-04-02 TCP Got paged 2am because of FEE alarm, which went away, and which the shifter had no details about. Most likely due to high backgrounds at the same time. Nothing to report.
    21-04-02 TCP The bar box manifold pressure increased further to 7.3 after which I was paged. Turned down the flow to 6.7, where it seemed stable.
    20-04-02 TCP Bar box manifold pressure increased from 6.30 Thursday to 7.03 Saturday. Epics seem to have a lower and not very stable value (6.3-6.8). Nothing serious to report.
    19-04-02 TH+TCP Nothing to Report.
    18-04-02 TCP Nothing to Report.
    17-04-02 TCP I tried to do a stand along run, but did not succed, which turned out to be because I should have been logged onto bbr-farm03 (pointed out by Chris O'Grady), after which it worked. Also I reported on our needs in terms of ambient database and DAQ during the shutdown (two weeks after ~9th of August). DIRC itself was fine.
    16-04-02 TCP Nothing to report. Bar box manifold gas pressure was 6.7.
    15-04-02 TCP Bar box manifold gas pressure was low, so Thomas showed Muriel and I how to adjust it, which we did. Apart from this, nothing to report.
    14-04-02 TCP Nothing to report.
    13-04-02 TCP The vacuum of the water degassing system reads 31''(should be 20-30''). Could be connected with the change of water filters.
    12-04-02 TCP Epics died for no reason (Netscape not open), but otherwise NTR.
    11-04-02 TCP Nothing to Report.
    10-04-02 TCP Change (and cleaning) of water filters - eventless.
    07-10/04/02 MK NTR
    04/07/02 MK High bck caused 100% dead time in the drc, did the same procedure to get back up and running
    06 Apr 02 TH since several weeks, we sometimes observe that the validation of the DIRC in the global calibration fails.
    I asked Vasia several times to check it out and the answer always was that the calibration was fine, but the validation timed out in a database access. This problem is not yet solved and does not happen in standalone calibrations so it seems to be in the common code for the global calibration. (The DIRC validation does lots of fits and database accesses so it is somewhat slower than the validation of most other systems.)
    So far, the commissioner could not give any statement at the 3:30 meeting about the reason and severity of the "calibration failure". This also leads to the fact that no one really works on this problem.
    I mailed with Vasia and he send me more information that allows the DIRC commissioner to check the problem and the reason for the failure. Here is his receipe:
    --- begin quote
    What you need to look into central
    logfile repository:
    
     /static/logfiles/Orc/2002/DRC
    
    
     on ir-2 subnet machines(eg bbr-farm02, bbr-dev20).
    
    It's plenty of information, especially in
    
     OepSimpleDaemon.log
    
    files. You will see for example if one module is
    not validated, ther will be printout ssying which
    one. If DB times out, it will say so too. So I think
    it's plenty for 3:30 meeting.
    --- end quote
    4/05/02 MK 100% dead time due to the DRC doing a complete shut down and restarting a new run solves the problem. Call Chris if problem re-occurs.
    01 Apr 02 MK NTR
    03/31/02 MK tweaked the N2 flow in sec 2 and 7 to nominal values, increased the barbox manifold pressure to 6.1,
    there was a transient red alarm in sector 6.
    03/27-03/30 MK NTR expept bad bck as PEP is having a lot of trouble with the beam

    28-03-02

    TH

    News about the chiller experts:
    Contact persons in case of further problems are:
    • Rene Candelario, he will take care if he is around.
    • Marvin Jones, head of HVAC, will be able to send someone in case we can not reach HVAC.
    • Kingston Chan: He was on our expert list but he is no longer working at HVAC. However, he is will to help in case we can not get any other expert.
    26-03-02 MP NTR
    JS Copied new DrcOED binaries to $drc/bin
     Recipe for DrcOED binaries:
     log in to tersk-26,
     create test release for 10.4.4-online-1, go to that new directory
     srtpath 10.4.4-online-1 SunOS5
     addpkg workdir
     addpkg DrcOED V00-03-01
     addpkg L1FctOep
     cp ~jochen/Hbook/Vnew/L1FctOep/L1FctDigiFromTransition.cc L1FctOep
     gmake workdir.setup
     gmake lib
     gmake bin
    25-03-02 MP NTR
    23-03-02 MP I got paged at 13:47. The fee supplies tripped because of the power glitch. The problem was solved by reset the fee status with epics.
    22-03-02 MP NTR
    21-03-02 MP The chiller triped today at about 11:20 (seen by DQM shifter + alarm). Ray reseted it and it was working well again at 11:50. It caused 100% dead time so the run was ended but the data quality doesn t seem to be affected. There were white alarms on epics, IOC's were then rebooted.
    20-03-02 MP The magnetic field was really high in the DIRC today. The bucking coil should have been ramped down since the solenoid was down but it was many hours later. We dont know if it caused any damages on the DIRC. The calibration failed today for the same reason as last week : Vasia answered that it failed to report to dataflow within timeout, but the calibrations were successful from DIRC point of view. He is investigating to solve this problem.
    19-03-02 MP NTR
    18-03-02 MP NTR
    15-03-02 MP NTR
    JS Chris O'Grady replaced FCDM in crate 17 (counting from zero) because we've seen a stuck bit in the fullmask register. That FCDM has been responsible for recent dead times as well as the timing shifts seen last year. Good riddance.
    14-03-02 MP NTR
    13-03-02 MP NTR
    12-03-02 TCP NTR
    11-03-02 TCP NTR
    10-03-02 TCP Got paged at 15:42 - same HV problems as yesterday. Once again high backgrounds associated and again no problems afterwards. Probably the channel should be masked in the alarm handler, but this has not been done yet. Got paged again at 20:53 - time plots showed large backgrounds on the negative side of the peak, and bin 10 of the feature extraction was again at 500. However, the backgrounds were very high, not the least from the HER, and so this is what was also seen Friday. Half an hour later everything had returned to normal. I will inspect this and the Friday run, when it comes out from OPR.
    9-03-02 TCP At 19:04 I got paged due to 100% deadtime caused by DIRC. The run was ended, and after Joe's suggestion (configure/unconfigure) everything worked again, and the deadtimes were back at 0.5%. Matt Weaver was paged, but we never heard from him! I checked the DIRC again, and everything seemed fine. However, at the time of the incident, the scaler rates had a large bump: all scalers ranging from 200-400 MHz. At 22:10 I was paged again, this time due to HV problems in Sector 10, board 2, channel 0. However, the problem disappeared, and the incident was most likely caused by high backgrounds. The DIRC HV showed no sign of problems afterwards.
    8-03-02 TCP Nothing to report upon inspecting the detector. However, the DQM shifter saw background left of the timepeak and bin 10 of the feature extraction was very high (400-600). The backgrounds were very high, and Joy judged that the DIRC was fine, and that this was just a natural effect of high backgrounds.
    7-03-02 TCP Gave monthly DIRC report. Otherwise, nothing to report.
    6-03-02 TCP At 9:00 we started doing standalone runs, first without the EMC source calibration, then with, and then without again. This is to see, if the blob we see in sector 5-6 is from the EMC. Regarding performance of the DIRC: Nothing to report.
    5-03-02 TCP Nothing to report.
    4-03-02 TCP Joe asked me to request for a DIRC standalone run with the EMC calibrating, so that we can look into the blop in sector 5 and 6. As it happens, we are going to have two hours without beam on Wednesday, and so we will do it then. Standalone runs should be done every Wednesday with EMC's source calibration.
    3-03-02 TCP Corruption of a register that can delay (but should normally be 0) the L1accept caused the DIRC information to be flawed in run 26610. Chris O'Grady looked into this, and says that it requires engineering. Bar box pressure normal.
    2-03-02 TCP Nothing to Report. However, the bar box manifold preasure was a bit low (5.60 compared to 6.30+-0.75).
    1-03-02 TCP Nothing to Report.
    TH Masked PMT in sector 7 Harness 11 Channel 61 in addition as this is noisy.
    28-02-02 TH Changed configuration to mask Sector 7 Harness 11 Channel 60 = Row 39 Column 22 instead of the wrongly masked Sector 9 Harness 13 Channel 60. Valid since run 26562.
    27-02-02 TCP Nothing to Report.
    25-02-02 TH Changed configuration to mask Sector 7 Harness 11 Channel 60 = Row 39 Column 22 ... however turned out to be Sector 9 Harness 13 Channel 60. Valid since run 26500.
    22-02-02 MP Every thing is OK. Thomas gave a short report on the studies made yesterday.
    20-02-02 MP DIRC is OK. The request to do some tests during the trickle injection tomorrow is not understood by the run coordinator (Dave) but Rainer seems to find it interesting.
    19-02-02 MP Nothing To Report.
    15-02-02 MP Nothing To Report.
    14-02-02 MP Nothing To Report.
    13-02-02 MP After standalone runs, it appears that the noisy channel in sector 7 is not due to the PMT but to the TDC. The PMT stays unplugged anyway.
    12-02-02 MP Nothing To Report.
    11-02-02 TCP+MP Nothing To Report.
    08-02-02 TCP+MP Nothing To Report.
    07-02-02 TCP+MP Nothing to report on Detector. Yannis asked for 'typical' background when beams were lost, and Muriel showed these (~hadig/public_html/dirc/weekend_feb/run25417_zoom.ps) at the meeting.
    06-02-02 TCP+MP Nothing To Report (ROD all day). Did a stand-alone run guided by Thomas, and the noisy channel in sector 7 was gone. Presented the weekends end-of-run backgrounds at 3:30pm.
    05-02-02 TCP+MP Nothing To Report. The backgrounds during the weekend were low!
    04-02-02 TCP+MP Nothing To Report - The Epics problem was solved (simply due to Netscape using too many colors). However, during the weekend, SVT and IFR had problems, and so all subsystems were asked to report backgrounds during the weekend.
    01-02-02 TCP+MP Regarding the detector itself, there was nothing to report.
    However, when we tried to look at the HV, Epics crashed! It seems to be a problem only on bbr-dev00, and Dominique and Dayle are looking into it. In order to check the HV, one should for the time being go over to the shift leader!
    31-01-02 TCP+MP Nothing To Report
    30-01-02 TCP+MP Nothing To Report
    1/16/02->1/29/02 MK Nothing To Report
    1/12/02 BM /nfs/bbr-srv01 crash recovered. Was able to restart bbr-dev00 successfully.
    1/11/02 BM Tried to restart bbr-dev00 with Muriel and Troels. Unfortunately, /nfs/bbr-srv01 crash prevented this.
    1/10/02 BM NTR. Muriel and Troels are here.
    1/9/02 BM Attempted to clear all alarm masks Central calibration now validates properly. Thanks Vasia and Chris.
    1/8/02 BM Drc successfully turned back ON. Shift leader asks if disabled alarms are correct. I note that the SIAM alarms are disabled as in the JS entry of 03/16/01 (view of epics panel on 03/16/01). Question: Do we still need this? I will ask Richard Kadel.
    1/8/02 BM Fire in PEP2 p/s's causes Drc to be turned off for safety.
    1/7/02 BM NTR
    1/6/02 BM NTR
    1/5/02 BM Processed random trigger run of two days ago. Ran HV ON/OFF/ON/OFF/ON test of noisy channel (HV#49) in sector 7. Difficult to conclude with certainty, but tube probably OK and FEE threshold probably low.
    1/4/02 BM Vasia has run a standalone T0 calibration. Vasia has also tried to re build DrcOedPlayback and needs help. I passed this problem on to Chris O'Grady who has offered to help too.
    1/3/02 BM Turned FEE on. Sector 4 DCC needed cleaning and re seating. HV turned on. Sector 7(#25) tripped. This was the Christmas tree. Sector 2(#19) also tripped. This was a new problem. Tubes were unplugged in each case and no trips now. Random trigger and calibration runs attempted. Several Solaris58 problems encountered. Some fixed. More tomorrow.
    1/2/02 BM Work begun. Chiller back on but FEE not. ROM's are off and DAQ is N/A. We hope to start in earnest tomorrow. Adjusted gas flows.
    12/17 BM NTR!
    12/16 BM NTR!
    12/15 BM NTR!
    12/14 BM NTR!
    12/13 BM NTR!
    12/12 BM Online problem not so bad - we do get central calibrations. Last one at 8:10 today is OK.
    12/11 BM Online problem means we may not get central calibrations until after the shutdown (next Thursday).
    12/11 BM Slightly reduced gas flow (7.1->6.4)
    12/10 BM NTR!
    12/09 BM NTR!
    12/08 BM NTR!
    12/07 BM NTR!
    12/06 BM NTR!
    12/05 BM NTR!
    12/04 BM NTR!
    12/03 BM NTR!
    12/02 BM Pep2 has now set up for lower luminosity (and lower background) so scalar ch.3 is further from the yellow limit now. I guess this high background was causing SVT and even DCH trips too often.
    12/01 BM Still getting yellow alarms from scalar ch.3. Solenoid went off and (of course) this caused a red alarm in the Dirc magnetic sensors.
    11/30 BM We get quite a few yellow alarms now from scalar ch.3. This is since the MD when PEP2 changed the vertical focussing. They anticipate higher backgrounds, and want any help in finding where these come from.
    11/29 BM bbr-dev00 upgraded to Solaris 8. A few teething problems: Did not source drcSetup.csh, Fastmon needs to be fixed to use "convert" rather than "ps2gif" for plots; Netscape applications were clobbered (somehow!).
    11/28 BM NTR!
    11/13 BM NTR!
    11/12 BM No calls, but sector 6 PMT still causing log entries. If we get some substantial beam down time I will try to catch it in action (it is intermittent - sometimes running at 1-2 MHz and sometimes quiescent). If I can, will see if HV makes any difference. If it is electronics, I guess we can do something at next door open access?
    11/11 BM NTR
    11/10 BM NTR
    11/09 BM NTR
    11/08 BM NTR
    11/07 BM NTR
    11/06 BM MD today. Made random trigger run to find noisy channel(s) - particularly the one in sector 6 generating e mail and a page. Only at 20 kHz (sec. 6#0). Also saw sec 7#49 (800 kHz) and sec 9#0 (30 kHz). Noted them in DQM instructions. One other thing - found in spewhists, starting at 0023812 at 8 pm last night - that we are getting significant number of chip errors now. I called George.
    11/05 BM DQM complains about sector 6 harness 0. There is a tube there that is sometimes hot. Better just watch it. I will do a random trigger run tomorrow during MD.
    11/04 BM NTR
    11/03 BM NTR
    11/02 BM NTR
    11/01 BM Most of day spent studying HV settings for sector7 Christmas tree.
    10/31 BM Slight adjustment to gas inlet pressure.
    10/29 TH NTR
    10/29 TH This morning we had a power glitch and several Wiener crates did not get back on automatically but needed to be switched by hand on through EPICS.
    10/28 TH This morning one of the HV groups in sector 7 appeared noisy, I looks like a PMT in row 40 produced a high rate. Quiet again in the next run.
    10/27 TH NTR
    10/26 TH NTR
    10/25 TH Ajusted gas inlet pressure. Attention: cap was/is missing, so you immediately change the gas pressure when you might think that you are only unscrewing the cap...
    10/24 TH NTR
    10/23 TH NTR, see hypernews for new trickle injection studies.
    10/22 TH NTR
    10/21 TH NTR
    10/20 TH Noisy PMT back to normal after a few hours.
    10/19 TH Joe helped to make some plots to check the influence of the trickle injection run yesterday on the DIRC. If confirms that we see a 20% increase in all scaler rates. See the plots here and here. One new noisy HV group in sector 7. One tube (sector 7, row 39, col 23, HV group 49) shows a high rate. Other noisy channel new in this run was not seen by DQM in sector 4.
    10/18 TH During trickle in injection. We do not see a significantly higher rate in our scalers, maybe 10-20% higher. See the plot here.
    10/17 TH Used MD to make a stand-alone run, results can be seen here. Obviously, a new Xmas tree is developing in Sector 7, Row 24, Col 3. Switched off that HV group (sec 7, HV #25) and will disconnect the PMT at the next ROD/access.
    10/13-10/16 MK NTR
    10/12 MK Fan in sec 7 died and it was replaced with a spare, the Glink in sector4 came back and DCC in sector 4 was replaced
    10/11 MK NTR
    10/10 MK Glink problem in sect4 this morning, rebooting the roms did NOT solve the problem, on swapping the fiber it was found that the problem followed the fiber that shows that the ROM is not at fault. Checks were done on the fiber and there was no attenuation in the fiber so could be a front end problem!! In April Carsten had jiggled the fibber in Sect 4, a loose connection??? DRC succeed Global Calibration !!
    10/9 MK Had ROD today, where the fans in the 12 sectors were cleaned. Tried to do a stand alone calibration and it did not produce any HBook file, the problem is referred to the OEP guys by Vasia our Cali expert.
    10/8 MK Glink problem in sector 4, doing a shutdown and starting a new run solved the problem, this essentially reboots the ROMS which the data flow guys donot like, the next time we have a glink problem and we know its not our front-end electronics that causes the problem then we should rool the ball to the Data flow and let them handle it !
    10/3-10/7 MK DRC FINE
    9/25 GV NOT YET
    9/24 GV Power glitches all evening. Shifter took care by phone all the times.
    9/23 GV DRC Fine
    9/22 GV DRC Fine
    9/21 GV Fan-tray on Sec#7 taken out clean and put back. Chassis was down
    9/20 GV DRC Fine
    9/19 GV DRC Fine
    9/18 GV DRC Fine
    9/17 GV DRC Fine
    9/16 GV DRC Fine
    9/15 GV DRC Fine
    09/14 GV DRC Fine
    9/13 GV DRC Fine
    09/12 GV DRC Fine
    08/28 - 9/12 MK DRC Fine
    08/27-28 MK NTR
    08/26 MK DRC proxies die, Boda heps to get it back , i donot know what to do if the proxy dies !
    08/25 MK DRC fine
    08/24 MK a site wide power glitch, DRC recovers well, there is an alarm on the DRC status and scalars as the bucking coil is not ramped down, the alarm goes off after the coil is ramped down.
    08/22-08/23 MK NTR
    08/18-20/01 CH Notor
    08/17/01 CH G-Link status error in sector 4. Power cycled the front end electronics but only a reboot of the ROM helped. Chris o'Grady suggested next time to swapp fibres first to figure out wheather it is a problem of the ROMs or our front end.
    08/16/01 CH Georges repaired the intermittend HV power supply not connected Epics alarm
    08/10-15/01 CH Notor
    08/08-09/01 CH Stefan and Andreas masked the electronicaly noisy channel sector 9, row 32, column 27, HV group 44. The change was done by editing the apropriate .xtc file. This hase to be changed to editing the configuration database. The masking is in effect starting Aug 9th 15:30.
    08/7/01 CH 1/2 HV crate (crate 3) sector 5 turned itself spontaneously off. We have no understanding of how that can happen. 1 crate, ok, but 1/2 crate. Maybe someone pushed a button....
    08/02-06/01 CH Notor
    08/01/01 IO, BM, CH, TH PMT in sector 0, row __, sector __, was disconnected because it spewed light onto other tubes in sector 0 and 11. "Warm" PMTs in sector 6, 7 and 9 was found to be completely spurious: fireing even with HV group switched off! => faulty electronics? 6, 7 and 9 were put back in
    07/31/01 IO,BM ROD: LBNL has replaced two moisture sensors, Jerry has installed more shielding, and the bad PMT in HV group 43, sector 2 has been disconnected (it was right behind the thick HV cable & FEE board, almost unreachable). Standalone random trigger run done. Since backward doors will be closed tomorrow evening at the earliest (water leak inside BaBar), the decision is to wait with doing a calibration untill tomorrow morning.
    07/28-30/01 IO,BM Different attempts to calm the ALH have not succeeded: we still see short alarms from DIRC HV CRATES ON/OFF.
    07/27/01 IO,BM The ALH fix works - too well(?). We had alarms from HV crate 1 twice - Sect 1 had status NC (not connected) - but only for a few seconds, then the alarm disappeared. HV for PMTs was on at all times. One limo-cable at the crate was tight and in not too good condition, and touching this caused another short alarm.
    07/26/01 IO,BM Did a DRC stand-alone random trigger run while EMC did their source calibration. Georges have improved ALH to cover situations like the one yesterday.
    07/25/01 IO,BM HV crate#1 had some problems (16:00pm, run 21588) causing all HV goups in sector 0 and 1 to switch off. The crate was on (lights on) but in an undefined state (no text in the display) when we got there. The shifter only noticed something was strange because scalers read 0 and caused red alarm. No HV alarm.
    07/20-24/01 IO Nothing to report
    07/12/01 GV (from DetOper-hn) Fuse for one HV crate blew last night so this was replaced today. This was not noticed until 6am. Apparently this condition did alarm, but the alarms were ignored for some time.
    07/04/01 IO Reports from Owl-shift: hot channel in sect. 7; FX status fatal 31 caused shifters to end run and mark the data as flawed. "Data Flawed"-tag has later been removed by run-coor.
    07/03/01 IO Nothing to report.
    07/02/01 IO Brownout. The procedures were tested for the first time. Ok. Will add some comments to the page about: IOC alarm + turning on Crate-4-Board-3 causes the board to trip -> need to set IORD to 2200 by hand.
    07/01/01 IO Nothing to report.
    06/30/01 IO Paged because of new noicy PMT in sector 7.
    06/29/01 IO Nothing to report.
    06/28/01 IO HV group 43 of sector 2 tripped right before midnight last night (seen in run 21071). Auto-recovery worked fine (-> 10 V). Tried to put it back to 1292 V, but it tripped again. Short test today: 900 V at beginning of run The occupancy still showed no entries, so even at 900 V the whole HV group seems dead. Tripped again at 1000 V. I leave it at 10 V.
    06/27/01 IO Text for FX plot in DIRC JAS-page have been modified.
    06/26/01 IO Nothing to report.
    06/25/01 IO Had another incident of entries in channel 30 of the Jas-plot Status Word of Feature Extraction today which caused Navigator to page me at 15:05. Message about "FX status fatal HOT occured in channel 31" was sent out at the same time. Both at beginning of run. Vasia and Chris O'G. concluded that bit 30 is Dataflow's problem and DRC commissioner should NOT be paged when this happens. JAS-page instructions must be changed. Water level gauge is back in EPICS :-) Alarm has correspondingly been enabled.
    06/22-24/01 IO Nothing to report.
    06/22/01 IO The red alarm regarding SOB status and water level has been disabled by run-coordinator, who has has asked us to check that it has been done correctly. I have propagated the task to Georges.
    06/21/01 IO DIRC monthly report was given after the operations meeting today. The two new degassing pumps have arrived, and one is installed permanently into the water system. This part of the water plant is now back to normal :) Doing ADC-calibrations during weekly EMC source calibration was discussed. Vasia will consider how this can be implementeted.
    06/20/01 IO The reason why the main water circulation pump lost power yesterday turned out to be that the wires leading to the power outlet got damaged during some work in the ir-2 hall involving brazing a copper pipe next to them. The heat from the torch melted the wire insulation. The wires supply power to four 208 Volts power outlets next to the water system. Now two of the power outlets are not working, the other two seem to be ok. The pump was brought back into operation by plugging the cable into one of the outlets that are still working. We are currently running almost as normal. The only thing that has no power is the water level gauge. So the water level is currently not possible to read from EPICS, and there is a corresponding red alarm. The plan now is to replace all wires to all 4 outlets, in case others are damaged as well. Waiting for some temporary power source / transformer. Time estimate : July...?
    06/19 MK The de-ionising pump died as it had no power, this manifest itself in the water tank being almost full ! Jerry came in and fixed the power problem which got the pump back up and this inturn took care of the almost full tank.. We have a blinking SOB unit and this will remain as i guess the values have to be reset ?? Jerry knows about this. Should remind him to fix it f not done.
    06/18 MK The degassing pump died, this was observed by the vacumm gauge reading 5" , Jerry has put in a temperory pump and a new one is ordered.
    06/06-06/17 MK NtoR
    05/30-06/06 CH NoToR
    05/25-29/01 CH NoToR except that there is a new intermittend channel in sector 7. This might cause some hundred fatal errors in readout module 3 (connected to channel 7)
    05/20-24/01 CH NoToR
    05/19/01 MK Fine
    05/18/01 MK Fine. There was a transient red alarm in the DRC and the occupancy in sector 1 was very low, could not find any problem with the DRC so reconfigured the DAQ which seemed to solve the problem. Looked into the database and there too could not find anything wrong with the DRC
    05/17/01 MK Fine. Drc failed callibaration waiting for the database to respond. Boda is looking into it
    05/16/01 MK Fine
    05/15/01 MK In run 20193 there were 13 events with FX fatal status >10 and in fast mon it was see that this run had a lager than average size but these events will not be processed and DRC is fine
    05/14/01 MK Decreased the bar box manifold pressure from 7.2 to 6.6, The DRC is fine
    05/11-13/01 MK DRC Fine
    05/7-8/01 IO It's really hot outside.... this is causing lots of problems for PEP and EMC chillers. But the DIRC is fine.
    05/04/01 IO Procedures for Blackout/brownout for all subsystems, and the DIRC in particular, are now available.
    05/02/01 IO Machine Development (day shift) to explore RF problems. Successful. No particular DIRC activities.
    04/27-05/01 IO Stable beams, but low currents. Double set of DRC alarms have been taken out of alarm handler, together with a few that were never used. Another EPICS panel for on/off is added with a button from the DIRC slow control panel.
    04/26/01 IO Skipped monthly reports from DRC and IFR in today's operations meeting because of the ongoing collaboration meeting. Next time will be May 24.
    04/25/01 IO Collaboration week continues, nothing to report...
    04/18-20/01 CH NToR
    04/17/01 CH Calibration failed: waiting for database, but it is a general dataflow problem
    04/15-16/01 CH NToR
    04/14/01 IO Seems to be a Christmas tree evolving in sector 9 (according to DQM histograms). HV group 44, row 32, column 27 (possibly the other way around). XmasSector9.gif
    04/11-13/01 CH NToR
    04/10/01 MK NtoR
    04/9/01 MK See HV spikes in sector 3 also see warinings in fast mon on hot channel 31. Probably a new Christmas tree comming up.
    04/8/01 MK NtoR
    04/07/01 MK There is onces again the broken secondary sensor in bar box 6 but some how I did not get the page. The shifters were able t ocontact Shaha who came in and disabbled the alarm
    04/05/01-04/07/01 MK NTOR
    04/04/01 MK We had a ROD. Diconnected the 2 PMT's and exchanged a HV crate in sector 8 and a DCC in sector 11. Had calibaration problems, which got itself fixed after carsten jiggled the fiber in sector 4 and 5
    03/31/01-04/03/01 MK NtoR
    03/30/01 IO There was a spurious red alarm at 02:11 last night from Crate 3 Section 5 - board 7 - channel 6. The alarm went away by itself.
    03/29/01 IO DIRC monthly report by Georges at the 3:30 Operations meeting in IR2.
    03/28/01 IO Information on the new fast (bucket-by-bucket) background monitoring system given by Jerry Va'vra on the 3:30 Operations meeting in IR2. The output from this is broadcasted on channel 62 and can be seen on the TV screen in the BaBar control room
    03/27/01 IO MD-day from noon to ca. 22, no special DIRC activities. Gas SIAM_1, Channel 0 no longer has red diodes.
    03/26/01 IO Nothing to report.
    03/25/01 IO Nothing to report.
    03/24/01 IO Nothing to report from DIRC commissioning. Background studies: Jerry and Ingrid went in during controlled access to take out a quartz rod and PMT from next to CsI #16. This background detector will be improved and moved to another location.
    03/23/01 IO Nothing to report.
    03/22/01 IO 10 a.m. Gas SIAM_1, Channel 0, signals "Broken water sensor cable". Joe is able to reset the SIAM trip in epics. 12:30 p.m. The same thing happens again, now it's not possible to reset in epics. Either there's a broken cable or an electronics fault. Either way it has to wait for a ROD. Alarm was masked out for now.
    03/21/01 IO Nothing to report.
    03/20/01 IO Nothing to report.
    03/19/01 IO Nothing to report.
    03/18/01 JS Nothing to report.
    03/17/01 JS Nothing to report.
    03/16/01 JS Moisture alarm from slot 6. Same problem as seen last Oct-Dec, suspect faulty electronics. Sasha has to go in during controlled access and check out electronics. He is on the call list. Masked DRC-GAS SIAM alarms in Babar Alh. See view of epics panel for this type of error.
    03/14/01 MK the HV on crate 1 had NC so GV rebooted the DRC HV IOC and all was fine
    03/12/01 GV Nothing special.
    03/11/01 GV Nothing special.
    03/10/01 GV Xtree Sec#9, HVchannel#9.Autodetected-corrected. [io] Scaler sec. 9 seems to be dead, although this happened at the same time as the HV for the PMT group with the X'mas tree was set to 10V and could be just an effect of that.
    03/09/01 GV Nothing special.
    03/08/01 GV Nothing special.
    03/07/01 GV Nothing special.
    03/06/01 GV Nothing special.
    03/05/01 GV Nothing special.
    03/04/01 GV Nothing special.
    03/03/01 GV Nothing special.
    03/02/01 GV Nothing special.
    03/01/01 GV Updates Alh with NC for the HV crates. Restart the Chiller [press and hold the reset button on the back until the pump starts]. This way you get rid of the SYS FAIL warning.
    02/28/01 GV Nothing special.
    02/27/01 JS Calibration failed, Vasia is working on code. Yet again a new trigger config, starting with run 18556. Best TDC time resolution ever with a small time shift of +25nsec.
    02/26/01 JS Problem with CAEN crate 1 (sec 0&2). Just like yesterday, turned itself off, no alarms. Needed two powercycles to come back up. Run 18552 affected. Vacuum pump of degasser died. Bob replaced it.
    02/25/01 JS Problem with CAEN crate 3 (sec 4&5). Turned itself off, no alarms. GV changed fuse, problems getting sec 0,1 crate up and running. Runs 18542, 18543 affected. Swapped scaler calbels sec 0 with sec 4.
    02/24/01 JS 500nsec time shift in JAS gone with new build. New -100nsec time shift due to EMT problem (see HN). Runs 18504-18516 affected - data flawed. See HN 1081 for details.
    02/23/01 JS DRC-HV IOC problem fixed. DIRC alh and expert modes work again.
    02/22/01 JS New epics version installed. Problem with DRC-HV IOC bogus readings. New online build is in place and should take care of 500nsec shift. No data yet. DIRC feature extraction code was modified to be capable of event batching by Vasia. Comparison of OdfNtupleMaker and DrcOedPlayback from xtc shows that scaler readings in sec 0,5 are indeed off by factor 2/0.25.
    02/21/01 JS The 500nsec timing shift has been traced to an online release bug. GDF will patch things up. We now have very ugly looking time peaks in the playback fastmon for some runs (18436 et al). Not clear why they look bad.
    02/20/01 JS OdcNtupleMaker proves that sec 5 scaler problem was present since first readout in 2001. Sec 0 scaler, on the other hand, "broke" when the pulser tests for sec 5 were performed.
    02/19/01 JS Power outage after fire alarm. Crate 4, board 3 comes up with wrong trip currents (100muA). Reset manually to 2200muA. Also reset alarm levels for sec 0 scaler back to 3* normal.
    02/18/01 JS 500 nsec timimg shift persists in spite of ROM replacement and master crate reboot. GV installs new alh panel path for glink errors and fixes library path problem.
    02/17/01 JS More glink errors - cpo replaced sec 2&3 ROM.
    02/16/01 JS DIRC timing problem during owl shift, 500 nsec jump of Cherenkov peak in JAS, see HN 1081. Vasia&cpo working on glink and timing problem. Added instructions for DQM pages to ignore unphysical timing shifts in JAS until further notice. ODF decides glink problem is ROM problem and cpo replaced sec 8&9 ROM.
    02/15/01 JS Tested scalers in sec 0 and 5. Both read same old strange rates when connnected to other channels. Plan to look at signal with scope. Raised alarm threshold for sec 0 scaler temporarily. Updated contact info in shifter web pages.
    02/14/01 JS Reduced N2 flow (bar box manifold pressure, gauge in back of rack 38) from 7.30 to 6.49. The scaler in sector 0 reads values that are too high compared to occupancy and rates in neighbouring sectors. Have to investigate.
    02/08/01 CH reinserted fiber in sector 11, now abundance of light
    02/01-02/07 CH NtoR
    01/26-31/01 CH NtoR
    01/27/01 CH Power Supply on Sector #8 replaced, Sec#8 back in order. Dead PMT channel : Sec#1, Row 36, Col 25.
    01/26/01 CH NtoR
    01/25/01 CH Recovered from power outage ok. Calibration looks strange: Vasia says to wait for a complet DIRC calibration on Saturday.
    01/22-24/01 CH NtoR
    01/20-01/21 MK NtoR
    01/19/01 MK The HV was off last night, got a page informing me that there is a red alarm in the DRC-Scalar even so all the scalar reads zero. The FEE in sec 1 and sec 3 turned off by them selves so turned off all the FEE and then turned them back on everything came back up fine execpt .sec 8 which is still off Did a callibaration on half the DIRC ie from 0-5 all looks good.
    01/16/01 MK Carsten and me tried to do a callibaration but thins looked screwed up so called Vasia who is fixing the callibaration up.
    01/15/01 MK Sec#8 Canbus was lost, so we lost sec #8 and sec #9 have to wait till 25th when the doors open again. Also in sec 1 HV section 48 the voltage is set to 9V as we had a trip.
    01/12/01 MK DIRC backward doors closed
    12/20/00 JS Nothing to report.
    12/19/00 JS Sasha found problem: faulty electronics. Fixed now.
    12/18/00 JS Another moisture alarm in slot 6. Sasha is working on tests. Created new checklist for winter shutdown for run coordinators. We will have one cryo shift and one run-coor who will regularly check the detector.
    12/16/00 JS Moisture alarm in slot 6
    12/15/00 JS Nothing to report.
    12/14/00 JS Nothing to report.
    12/13/00 JS Nothing to report.
    10/31/00 JS Nothing to report.
    10/30/00 JS Nothing to report.
    10/29/00 JS Nothing to report.
    10/28/00 JS Nothing to report.
    10/27/00 JS Nothing to report.
    10/26/00 JS Two runs were marked as flawed by DQM shifters (16569, 15575) without explanation. Try to figure out why with RQM.
    10/25/00 JS Nothing to report.
    10/23/00 JS Low depoint alarm from BB3&6. Reset lower alarm threshold to -80deg for all BBs. Secondary water alarm in BB6 had latched, brief flicker of primary BB6 humi alarm observed, too brief to dump SOB but very worrisome. Maxed out N2 flow to all BBs to dry out any slow seeping leak or condensation.
    10/22/00 JS Nothing to report.
    10/21/00 JS Nothing to report.
    10/20/00 JS Nothing to report.
    10/19/00 JS Nothing to report.
    10/17/00 JS Nothing to report.
    10/18/00 JS Nothing to report.
    10/16/00 JS Nothing to report.
    10/15/00 JS Nothing to report.
    10/14/00 JS Nothing to report.
    10/13/00 JS Lowered PMT HVs back to nominal values. Last run with +40V setting was 16659.
    10/12/00 JS Nothing to report.
    10/11/00 JS Nothing to report.
    10/10/00 JS Calibrations failed 10/9 and 10/10 first time, succeeded second time. After Mon Calib, -0.5ns shift in sectors 10,11. On Thue, -100ps shift in all sectors. Vasia is looking into it. ADC calib always strange, ignore for now until Vasia says otherwise.
    10/09/00 JS GV raised PMT HV to +40V. Run 16561 is first run with +40V.
    10/08/00 JS Nothing to report.
    10/07/00 JS Nothing to report.
    10/06/00 JS Nothing to report.
    10/05/00 JS GV raised PMT HV to +10V. Run 16470 is first run with +10V. Plan to keep that level for the week to test influence on photon yield. During downtime took random trigger run. No hot spots visible. With GV took "HV calibration", ran standalone DIRC calibration for PMT HV settings between -50V and +50V.
    10/04/00 JS Nothing to report.
    10/03/00 JS Nothing to report.
    10/02/00 JS Nothing to report.
    09/26/00 GV. Problems with DCC sector 2 since last night. Access today: exchanged sec2 and all remaining 5 old DCCs with "good" ones. All DCC are now with the good polarity. runs in range 16234-16271 marked "data flawed".
    09/26/00 GV. NoToR
    09/25/00 GV. NoToR
    09/24/00 GV. NoToR
    09/23/00 GV. NoToR
    09/22/00 GV. NoToR
    09/21/00 GV. NoToR
    09/20/00 GV. NoToR
    09/19/00 CH. NoToR
    09/18/00 CH. NoToR
    09/17/00 CH. NoToR
    09/16/00 CH. NoToR
    09/15/00 CH. NoToR
    09/14/00 CH. DCC in Sector 10 broke, replaced sectors 9. 10, 11. 9 was bad and had to be replaced again. Calibration didn't produce plots.
    09/13/00 CH. NoToR
    09/12/00 CH. NoToR
    09/11/00 CH. NoToR
    09/10/00 CH. NoToR
    09/09/00 CH. NoToR
    09/08/00 CH. NoToR
    09/08/00 CH. NoToR
    09/07/00 CH. NoToR
    09/06/00 CH. NoToR
    09/05/00 SmS. standalone calib makes no plots; ask Vasia ?
    09/04/00 SmS. Nothing
    09/03/00 SmS. Nothing
    09/02/00 SmS. DCC replace; christmas off; see HN 966 ff.
    09/01/00 SmS. DCC sector 1 broken; take out ROM
    08/31/00 SmS Nothing
    08/30/00 SmS. noisy tube in sector 4 probab readout (chip error)
    08/29/00 SmS. power outage; recovery; see HN 968
    08/28/00 SmS. Nothing
    08/27/00 SmS. Nothing
    08/26/00 SmS. Nothing
    08/25/00 SmS. Nothing
    08/24/00 SmS. christmas trees see HN 958
    08/23/00 SmS. Nothing
    08/22/00 SmS. Nothing
    08/21/00 SmS. Nothing
    08/20/00 SmS. Nothing
    08/19/00 SmS. Nothing
    08/18/00 SmS. Nothing
    08/17/00 SmS. Nothing
    08/16/00 SmS. Nothing
    08/15/00 SmS. Nothing
    08/14/00 SmS. Nothing
    08/13/00 SmS. Nothing
    08/12/00 SmS. Nothing
    08/11/00 SmS. Nothing
    08/10/00 SmS. Nothing
    08/09/00 SmS. Nothing
    08/08/00 SmS. Nothing
    08/07/00 SmS. Nothing
    08/06/00 SmS. Nothing
    08/05/00 SmS. Nothing
    08/04/00 SmS. switch on; DFB in sector 10 broken; replace, see HN
    08/03/00 SmS. Nothing
    08/02/00 GV. Nothing
    08/01/00 GV. Nothing
    7/31/00 GV. Nothing
    7/30/00 GV. Nothing
    7/29/00 GV. Nothing
    7/28/00 GV. DCC sector#5 died. Doors opened and replaced.
    7/27/00 GV. Nothing
    7/26/00 GV. Nothing
    7/20/00 CH. FEE 5V in crate 5 was 3.3V. Switched powersupply off/on and was ok. Don't forget to reboot the ROMs when you switch off the electronics after a begin run transition.
    7/08/00 CH. notor
    7/07/00 CH. notor
    7/06/00 CH. pager works: we had high background.....
    7/05/00 CH. notor
    7/04/00 CH. notor
    7/03/00 CH. two consecutive playback monitors crashed into each other (Stefan will put a hold onto the new one when the old one is still running)
    7/02/00 CH. notor
    7/01/00 CH. notor
    6/30/00 CH. notor
    6/29/00 CH. notor
    6/28/00 CH. notor
    6/27/00 S.
    6/26/00 S. two consecutive playback monitors crashed into each other
    6/25/00 S. notr
    6/24/00 S. integrated lumi record
    6/23/00 S. exclude shared memory application from fast monitor
    6/22/00 S. sector 4, row 0 possible hot candidate: watch it
    6/21/00 S. notr
    6/20/00 S. notr
    6/19/00 S. first output from automatic monitor 2000.06.19-14:27:21 DRC FX status fatal expect OK, is HOT occurred in channel 26 ... after high background and immediate HER beam dump and run stop. Run 14013 o.k.
    6/18/00 S. notr
    6/17/00 S. notr
    6/16/00 S. DQM shifter page because of 1inch higher water level in recycle tank
    6/15/00 S. notr
    6/14/00 S. Unplug PMT sector 8 row 2 column 3
    6/07/00 GV. Rien special.
    6/06/00 GV. Remedy for FastMon. Type the line : fs flushvol on the machine FastMon runs ( after S.Spanier).
    6/13/00 GV. Nothing special.
    6/12/00 GV. Nothing special.
    6/11/00 GV. Nothing special.
    6/10/00 GV. Nothing special.
    6/09/00 GV. Nothing special.
    6/08/00 GV. Nothing special.
    6/07/00 GV. Nothing special.
    6/06/00 GV. FastMon updates died.
    6/05/00 GV. Nothing special.
    6/04/00 GV. Spike on Sector#4. Disapear after. Keep an eye.
    6/03/00 GV. Small Calibration problems related to RC. Stefania takes care.
    6/02/00 GV. Nada
    6/01/00 GV. Nada
    5/31/00 GV. Nada
    5/30/00 JS. DIRC pager broken - replaced with new model, same number. Gregory reminded all subsystems (again) to provide comparison histos for aautomatic monitoring. DIRC is one of the main offenders. E-mailed Stefan.
    5/29/00 JS. Nothing to report.
    5/28/00 JS. Nothing to report.
    5/27/00 JS. Calibs OK again - pervious problem probably due to quirk in online software updates during MD&ROD.
    5/26/00 JS. Updated fastmon web page and reference plots to show that only FX status > 15 is fatal error. DIRC calibs did not validate today.
    5/25/00 JS. Nothing to report.
    5/24/00 JS. ROD. UPS changes to dump system. Disconnected HV from two PMTs (x-mas trees) in sector 2 and 3, confirmed success with random trigger run. Tested new power-up epics panel.
    5/23/00 JS. Nothing to report.
    5/22/00 JS. Nothing to report.
    5/21/00 JS. Took random trigger run (setup works again) - no new hot or warm areas.
    5/20/00 JS. Nothing to report.
    5/19/00 JS. Nothing to report.
    5/18/00 JS. Tried to take random trigger run during weekly calibrations - failed with OepLog error.
    5/17/00 JS. Announced new condidtion on DIRC scaler rates - alarm only when DAW running.
    5/11/00 GV. Nothing to report.
    5/10/00 GV. MD day. Runnable Flags installed Everything OK.
    5/9/00 GV. Nothing to report. Calibration works now.
    5/8/00 GV. UPS from Gas rack burned out. The cause of Fire alarms.
    5/8/00 GV. Kadel informed about Crate. We keep on eye.
    5/6/00 GV. Power cut at 3:00 in the morning. VESDA alarm. Might come from Kadel's crate. Investigating. Re-power all DIRC.
    5/5/00 GV. My birthday. Calibration failed again. Vasia's took care.
    5/4/00 GV. Nothing to report.
    5/3/00 GV. (18:15) Calibration failed twice due to disk space problems.
    5/2/00 JS. Nothing to report.
    5/1/00 JS. Nothing to report.
    4/30/00 JS. Nothing to report.
    4/28/00 JS. Calibration validated again.
    4/28/00 JS. Nothing to report.
    4/27/00 JS. Took a random trigger run and saw no new hot tubes. DIRC has been failing daily calibration. Vasia informed, error appears to be partly our fault and partly ORC mistake.
    4/26/00 JS. ROD day. Disconnected 4 hot tubes: sec 0, hv 43, har 3, chan 46, col 8, row 34; 2/5/1/4/0/4; 4/18/13/14/21/23; 5/11/13/2/18/15. Jerry et al added lead shielding, modified CsI setup. Accidental SOB dump caused by work on power-supply of drc-gas crate. Many problems with settings of HV crate (trip currents crate 4, board 2) after turning HV back on. After door closing, one new x-mas tree in sector 3: 3/53/2/61/6/39, turned HV down to 10V. One hot channel in sec 8: 8/4/3/2/3/2 at ~100kHz, no light in neighbors. Took several stand-alone runs with fcgui: pulser and random trigger.
    4/24/00 CH. Nothing to report.
    Sun 4/23/00 CH. Nothing to report.
    Sat 4/22/00 CH. Nothing to report.
    Fri 4/21/00 CH. Nothing to report.
    Thu 4/20/00 CH. Nothing to report.
    Wed 4/19/00 CH. Nothing to report. JS. installed swapped PM map into ir2boot
    Tue 4/18/00 CH. Nothing to report.
    Mon 4/17/00 CH. Nothing to report.
    Sun 4/16/00 CH. Nothing to report.
    Sat 4/15/00 CH. Nothing to report.
    Fri 4/14/00 CH. Run Controll took reboots for ROMs out after doing calibration ==> Data are screwed up. It's a Central Calibration problem, will be corrected. For time being: reboots are back
    Thu 4/13/00 CH. New x-mass tree: I switched off crate 2 card 0 channel 5 first run effected is 12748.
    Wed 4/12/00 CH. Nothing to report.
    4/11/00 JS Found some 40 swapped signal cables that will have to be corrected (using StoreDrcCableErr) in the fdb (ir2boot etc). Andreas will prepare a data file and the executable. Announced the study in the meeting. On the ROD we have to check if the cables are in fact swapped, in particular for one group with a complicated swap pattern. No MD this week.
    4/10/00 JS Nothing to report.
    4/7/00 JS Jerry completed installation of CsI monitors (5 on LER, 4 on HER, see HN). To do for ROD: add more lead shielding (need crane) and remove HV from the two light emitting PMTs. The inlet N2 pressure had dropped to 5.81 so I adjusted the regulator in the back of rack #38 together with Rich. Set the pressure to 6.54 and re-adjusted the individual flowmeters to their nominal values. Pressure after adjustment 6.81. Rich is not worried, thinks the regulator is simply drifting. MD next week may get cancelled.
    4/6/00 JS vacuum leak of HER gave Jerry et al a chance to add new CsI radmons today. Will try to add more tomorrow during brief controlled access but there's not enough time for more lead shielding. Richard worked on GMB card, turned off gas/water safety crate. When turned back on the crate produced SIAM alarm in one channel, Richard rebooted it. The past couple of days the flow in the gas flowmeters has been slowly decreasing. I adjusted the flow several times, need to keep an eye on that.
    4/05/00 JS e-mails to babardrc@slac.stanford.edu will now be forwarded to all four RCs. Stefan has added a section on the stand-alone fastmon to the DIRC operations manual. ROD day now moved to April-27.
    Mo 4/03/00 CH. CH. Raised the minor/major alarms for the number of allowed clients on drc-mon from 60/70 to 90/100. The change will be permanent after the next ioc reboot.
    Su 4/02/00 CH. New gigantic x-mass tree. Reduced HV to 10V for crate 3 board 1 channel 18. Raised current limits to 2500mA for one other supply.
    Sa 4/01/00 CH. Nothing to report.
    Fr 3/31/00 JS Updated QA flag info from old gif to new html file. Updated fastmon info regarding FX errors.
    Mo 3/27/00 CH. Nothing to report. JS Updated DQM instructions wrt time register overwrite. The watchdog process is still running on our crates and will stop the run, causing a "DIRC full" error that is fixed by rebooting (not power-cycling our ROMS. Error has not happened in 3 weeks, ODF believes that new backplane build includes fix. SVT trigger time jump ~+20nsec again.
    Su 3/26/00 CH. Calibration failed (not our fault --> operators should have rebooted the ROMS)
    Sa 3/25/00 CH. Nothing to report.
    Fr 3/24/00 CH. Nothing to report. JS Updated DQM instructions wrt. FX errors.
    3/23/00 JS SVT observes time shift (trigger) back to old value, i.e. -~20 nsec. Confirmed by our fastmon. ROD probably Apr. 12
    Mar-14-00 JS Nothing to report.
    Mar-13-00 JS Fastmon fit of TDC time spectra fails frequently. Informed Stefan.
    Mar-12-00 JS Started this logbook web page.

    JavaScript Quarterly Calendar

    Free JavaScripts provided
    by The JavaScript Source

    DIRC commissioner links


    Old schedules

    2006 Commissioner Shift Schedule:

    Date: Commissioner: Backup:
    December 29 - January 11 Nicolas Rolf
    January 11 - February 8 Rolf Emmanuel and Nicolas
    February 8 - March 8 Emmanuel Rolf and Nicolas
    March 8 - March 22 Jerome Emmanuele, Rolf, Emmanuel and Nicolas
    March 22 - April 5 Emmanuele Rolf, Emmanuel, Jerome and Nicolas
    April 5 - April 19 Emmanuel Rolf, Emmanuele, Jerome and Nicolas
    April 19 - May 3 Rolf Emmanuel, Emmanuele and Nicolas
    May 3 - May 24 Jerome Emmanuel, Emmanuele, Rolf and Nicolas
    May 24 - June 14 Emmanuel Emmanuele, Jerome and Rolf
    June 14 - June 28 Rolf Emmanuel, Emmanuele, Jerome and Nicolas
    June 28 - July 26 Jerome Rolf, Nicolas and Bradley
    July 26 - August 08 Rolf Emmanuele, Jerome and Bradley
    August 08 - End of Run 5b Bradley Emmanuele, Rolf and Nicolas
    August 22 - September 5 Rolf Emmanuele, Bradley and Nicolas
    September 6 - September 27 Bradley Emmanuele, Rolf and Nicolas
    September 27 - October 11 Rolf Emmanuele, Bradley and Nicolas
    October 11 - October 25 Bradley Emmanuele, Rolf and Nicolas
    October 25 - November 1 Rolf Emmanuele, Bradley and Nicolas
    November 1 - November 15 Bradley Emmanuele, Rolf and Nicolas
    November 15 - November 29 Rolf Emmanuele, Bradley and Nicolas
    November 29 - December 11 Bradley Emmanuele, Rolf and Nicolas
    December 11 - January 1 Rolf Emmanuele, Bradley and Nicolas
    January 1 - startup Bradley Emmanuele, Rolf and Nicolas

    2006 Oncall balance:

    2005 Commissioner Shift Schedule:
    Date: Commissioner: Backup:
    Jan 03 - Jan 19 Kalanand Nicolas
    Jan 19 - Feb 16 Nicolas Kalanand
    Feb 16 - Mar 09 Kalanand Nicolas
    Mar 09 - Mar 23 Nicolas Kalanand
    Mar 23 - Apr 13 Kalanand Nicolas
    Apr 13 - May 20 Nicolas Kalanand
    May 20 - Jun 01 Kalanand Nicolas
    Jun 01 - Jun 22 Nicolas Kalanand
    Jun 22 - Jul 19 Kalanand Nicolas
    Jul 19 - Aug 08 Nicolas Kalanand and Emmanuele
    Aug 08 - Aug 22 Emmanuele Nicolas and Kalanand
    Aug 22 - Sep 12 Nicolas Kalanand and Emmanuele
    Sep 12 - Shutdown Nicolas Kalanand
    On call during some weekends
    Shutdown Nicolas Kalanand
    On call during some weekends
    November 14 - November 28 Kalanand Rolf and Nicolas
    November 28 - December 12 Rolf Kalanand and Nicolas
    December 12 - December 29 Kalanand Rolf and Nicolas
    (in France between 12/18 and 12/29)
    December 29 - January 9 Nicolas Rolf and Kalanand

    2005 Oncall balance:

    Summer 2004 Activites:

    Summer 2004 Shutdown Activities
    calibrate helium detector Jerry
    install shielding (?) Jerry
    finish air flow installation Jerry- Bob
    check disconnected PMTs José - Philippe
    recheck and finish power cable installation Jerry, Bob, Walt, José, Philippe, Sam
    clean fan trays José - Philippe
    test more spare HV boards José - Philippe
    improve DIRC calibration Jim
    improve HV ramping speed Thomas
    Test Gate Generator Thomas - Philippe - José
    Test calibration LED module José - Philippe
    Finish to modify DCCs Philippe - Lupe
    Test all DFBs and DCCs on test-stand Philippe - José
    Improve HV ramping-up time Thomas

    Commissioner Shift Schedule April 2004 - July 2004:

    Date: Commissioner: Backup:
    Mar 31 - April 22 José Kevin
    April 22 - May 12 José Qinglin
    May 12 - June 2 Qinglin Kevin
    June 2 - June 23 Qinglin José
    June 23 - July 14 Kevin Qinglin
    July 14 - July 31 Kevin José

    Shift balance:

    Commissioner Shift Schedule Jan 2004 - Mar 2004:

    Date: Commissioner: Backup:
    Jan 02 - Jan 07 Philippe x
    Jan 07 - Jan 14 Kevin Philippe
    Jan 14 - Feb 04 Kevin Qinglin
    Feb 04 - Mar 03 Qinglin Kevin
    Mar 03 - Mar 17 Kevin Qinglin
    Mar 17 - Mar 31 Qinglin José

    Shift balance:

    Commissioner Shift Schedule Nov 2003 - Dec 2003:

    Date: Commissioner: Backup:
    Nov 26 - Dec 03 Guillaume Qinglin
    Dec 03 - Dec 10 Guillaume Kevin
    Dec 10 - Dec 19 (Friday) Guillaume Brandon
    Dec 19 - Dec 23 Qinglin Thomas

    Shift balance:

    Commissioner Shift Schedule Sep 2003 - Nov 2003:

    Date: Commissioner: Backup:
    Sep 03 - Sep 17 Kevin Guillaume
    Sep 17 - Oct 01 Brandon Guillaume
    Oct 01 - Oct 08 Brandon Qinglin
    Oct 08 - Oct 15 Qinglin Brandon
    Oct 15 - Oct 29 Qinglin Kevin
    Oct 29 - Nov 12 Kevin Guillaume
    Nov 12 - Nov 26 Guillaume Brandon

    Shift balance:

    Commissioner Shift Schedule Jul 2003 - beginning Sep 2003:


    (during shutdown, no backup assigned, Philippe or I will carry the backup pager during daytime just in case)

    Date: Commissioner:
    Jun 27 - Jul 16 Qinglin
    Jul 16 - Jul 30 Guillaume
    Jul 30 - Aug 20 Brandon
    Aug 20 - Sep 03 Kevin

    Shift balance:

    Commissioner Shift Schedule Feb 2003 - beginning Jul 2003:

    Date: Commissioner: Backup:
    Jan 29 - Feb 12 Brandon Georges
    Feb 12 - Feb 19 Brandon Qinglin
    Feb 19 - Feb 26 Qinglin Philippe
    Feb 26 - Mar 05 Qinglin Georges
    Mar 05 - Mar 19 Georges Brandon
    Mar 19 - Mar 26 Brandon Philippe
    Mar 26 - Apr 02 Qinglin Kevin
    Apr 02 - Apr 09 Georges Thomas
    Apr 09 - Apr 16 Georges Kevin
    Apr 16 - Apr 23 Qinglin Kevin
    Apr 23 - Apr 30 Brandon Qinglin
    Apr 30 - May 07 Georges Qinglin
    May 07 - May 21 Georges Kevin
    May 21 - May 28 Kevin Brandon
    May 28 - Jun 04 Brandon Thomas
    Jun 04 - Jun 11 Brandon Qinglin
    Jun 11 - Jun 18 Kevin Brandon
    Jun 18 - Jun 27 Kevin Qinglin

    Shift balance:

    Commissioner Shift Schedule Nov 2002 - end Jan 2003:

    Date: Commissioner: Backup:
    Oct 30 - Nov 06 Thomas Brandon
    Nov 06 - Nov 13 Philippe Brandon
    Nov 13 - Nov 20 Philippe Qinglin
    Nov 20 - Dec 04 Qinglin Philippe
    Dec 04 - Dec 11 Qinglin Maha
    Dec 11 - Dec 23 Maha Philippe
    X-mas shutdown emergency contacts:
    Jerry Va'vra Robert Reif
    Jan 02 - Jan 08 Thomas Qinglin
    Jan 08 - Jan 15 Philippe Thomas
    Jan 15 - Jan 22 Georges Thomas
    Jan 22 - Jan 29 Brandon Georges

    Shift balance:

    Commissioner Shift Schedule Jul - end Oct 2002:

    Date: Commissioner: Backup:
    Jul 02 - Jul 05 Thomas Phillipe
    Jul 05 - Jul 12 Troels Thomas
    Jul 12 - Jul 31 Eugeni
    Jul 31 - Aug 07 Brian
    Aug 07 - Aug 13 Georges Vasileiadis
    Aug 13 - Sep 30 Thomas
    Sep 30 - Oct 17 Phillipe
    Oct 17 - Oct 23 Georges Vasseur
    Oct 23 - Oct 30 Phillipe

    Shift balance:

    Commissioner Shift Schedule Jan - Jul 2002:

    Date: Person:
    Jan 02 - Jan 16 Brian
    Jan 16 - Jan 30 Maha
    Jan 30 - Feb 13 Muriel and Troels
    Feb 13 - Feb 27 Muriel
    Feb 27 - Mar 13 Troels
    Mar 13 - Mar 27 Muriel
    Mar 27 - Apr 10 Maha
    Apr 10 - May 1 Troels
    May 1 - May 15 Maha
    May 15 - May 22 Muriel
    May 22 - May 29 Troels
    May 29 - Jun 05 Thomas
    Jun 05 - Jun 12 Muriel
    June 12 - June 26 Brian
    Jun 26 - Jul 02 Thomas

    Blackout periods:

    Shift balance:

    Commissioner Shift Schedule 2001:

    Blackout periods:
    Date: Person: (per month) [total]
    Jan 01 - Jan 10 Georges (2) [2]
    Jan 10 - Jan 24 Maha (2) [2]
    Jan 24 - Feb 07 Carsten (2) [2]
    Feb 07 - 14 Georges (1) [3]
    return to monthly schedule with two persons as on-call & backup
    Feb 14 - Mar 14
    (2/14-2/28, 2/28-3/14)
    Joe and George (2/2) [2/5]
    Mar 14 - Apr 11
    (3/14-3/28, 3/28-4/11)
    Ingrid (Joe) and Maha (2(2)/2) [2(4)/4]
    Apr 11 - 25 Carsten (2) [2]
    Apr 25 - May 23
    (4/25-5/9, 5/9-5/23)
    Ingrid and Maha (2/2) [4]
    May 23 - Jun 20
    (5/23-6/6, 6/6-6/20)
    Carsten and Maha (2/2) [4]
    Jun 20 - Aug 01
    (6/20-7/4, 7/4-7/18, 7/18-8/1)
    Ingrid and Georges and Ingrid (2/2/2) [6]
    Aug 01 - Aug 29
    (8/1-8/29)
    Carsten (4) [4]
    Aug 29 - Sep 26
    (8/29-9/12, 9/26)
    Maha and Georges (2/2) [4]
    Sep 26 - Oct 31
    (9/26-10/03, 10/03-10/17, 10/17-10/31)
    Carsten, Maha and Thomas (1/2/2) [5]
    Oct 31 - Nov 28
    (10/31-11/14, 11/14-11/28)
    Brian and Georges (2/2) [4]
    Nov 28 - Jan 02
    (11/28-12/19, 12/19-01/02)
    Brian and Georges (3/2) [5]

    Shift balance based on shifts January 01 through January 02:

  • Thomas: 2 weeks
  • Brian: 5 weeks
  • Commissioner Shift Schedule 2000:

    Blackout periods:
    Date: Person: (per month) [total]
    Jan 01 - Mar 21 Joe and Stefan
    Mar 22- Apr 4 Carsten and Joe
    Apr 5 - May 3 Carsten and Joe (2/2) [2/2]
    May Joe and George (2/2) [4/2]
    June George and Stefan (2/2) [4/2]
    July Carsten and George (3/2) [5/6]
    Aug Stefan (5) [7]
    Sep Carsten and George (2/2) [7/8]
    Oct Joe (4) [8]
    Nov/Dec shutdown time ~7 weeks.
    Oct 30 - Nov 08 George [9]
    Nov 08 - Nov 22 Carsten [9]
    Nov 22 - Dec 06 Stefan [9]
    Dec 06 - Dec 13 Joe [9]
    Dec 13 - shutdown Carsten or George

    Shift balance based on shifts April 5 through October 31:


    Created by Joe. Last major update on 03 Apr 03 by Thomas