SLAC PEP-II
BABAR
SLAC<->RAL
Babar logo
HEPIC E,S & H Databases PDG HEP preprints
Organization Detector Computing Physics Documentation
Personnel Glossary Sitemap Search Hypernews
Unwrap page!
Det. Search
Who's who?
Meetings
FAQ
Images
Archive
Systems
Performance
Intern. region
Vertex Tracker
Drift chamber
DIRC
Calorimeter
IFR
LST
Magnet
Electronics
Trigger
Operations
Run Coordination
Contact Experts
Shift Takers Info
Operations Manual
Electronic Logbook
Ops Hypernews
Shift Signup
Check this page for HTML 4.01 Transitional compliance with the
W3C Validator
(More checks...)
EMC Troubleshooting
Problems coveredAdditional documentation
Power supplies General EMC Monitoring Information
Monitoring (EPICS) Current EMC Hardware Status
Noisy or dead channels (OEP monitoring histograms) List of EMC Subsystem Experts
DAQ: ROMs or whole crates going full or out of synch The EMC Troubleshooting Guide (BaBar-Note 553)
Light pulser problems
EMC chiller alarms
 
Symptom Therapy
LV Power Supply (PS) shows negative temperature readings If data from the corresponding IOB looks OK, and -5V voltage and currents look OK as well, try the following sequence until problem fixed:
  1. powercycle the PS
  2. take PS out, plug it back in
  3. replace PS
If replacing the PS does not fix the temparature reading, the problem is most likely on the backplane and cannot be fixed right away. Arrange for alarm limits to be adjusted (ask the monitoring expert), or the alarm masked in the alarm handler.
Warning: doing any of the 3 steps described above will certainly require run control to be reconfigured. Pay attention to the data taken right after the power supply was touched. Look especially for groups of 72 channels missing.
Yellow or Red EPICS alarm
  1. Make sure the alarm is real
  2. Here are some known symptoms causing alarms to help you identify the problem
  3. If related to a LV power supply, powercycle it. If problem persists, replace it
  4. .
White EPICS alarm Communication has been lost.
  1. Wait. Often the channels come back after a few minutes.
  2. If problem persists: Follow instructions on the Monitoring troubleshooting page
EPICS display does not work (crash or white boxes on the screen)
  1. Are you running Epics from one of the machines on the IR2 subnet (bbr-devXX) ? If no, then your Epics client (display) does not have access to the channel database and you display will be white like this.
  2. log on to bbr-dev03 and restart your EPICS display.
  3. Is Netscape running on the same terminal? If yes, kill Netscape and restart it with netscape -install &. Then restart EPICS with
    source ~babaremc/monitoring/monitorSetup .
Dead channels in OEP histogram Look for a pattern within the dead channels:
  • Group of 720 channels (430 endcap): a whole crate is missing.
    1. Check the corresponding TRB power supply (Click for Floor plan of E-Hut). Replace if broken. If that is not the problem, check points 2-4 with the DAQ shifter:
    2. Was the crate in the partition at all? Check crate mask for that run.
    3. If yes, did it fail to reboot?
    4. If not, did it run full?
    5. If a DAQ failure can be excluded, call the electronics expert.
  • Group of 72 adjacent channels in the barrel (theta runs from 9-32 or 33-56, in three phi counts) or 43 channels in the endcap (appears as three squares in the theta-phi plane): one complete ROM is missing. This could have the following reasons:
    1. The ROM did not reboot. Check if the green i960 LED is on. If not, push the reboot button.
    2. The power supply failed. Check the corresponding EPICS channels, or the PS itself. If the red LED is on, try powercycling it first. If problem persists, replace the power supply.
    3. If none of the above is true, the problem is either the IOB or the UPC(ROM).
  • Group of 24 adjacent channels: one fiber is missing.
    1. Check the corresponding LED on the ROM. If it is stable or flashing red, swap the fiber with the next one on the same ROM.
      1. If the LED moves with the fiber, the problem is either the fiber itself or the frontend (IOB). Nothing you can do about it, fill out an electronics fault report form.
      2. If the same LED remains red no matter where you move the fiber to, the problem is with the UPC. Notify the Odf expert.
    2. Contact Electronics expert.
  • Group of 12 adjacent channels: one ADB is missing. This is a frontend problem. Report it.
  • Group of 4 adjacent channels: Probably a frontend problem. Report it.
  • Single channels missing: frontend problem. Report it.
Noisy channels in OEP histogram Look at the Hit Maps (page 7).
EMT reports hot towers or Level 3 reports hot channels Look at the OEP histograms (page 7, hitmaps for 0.01 GeV and higher thresholds) and try to correlate noisy channels with alleged hot towers. If the problem can in fact be traced back to the EMC, and it causes a serious problem for taking data (i.e. high deadtime), contact the electronics expert.
OutOfSynch errors
  1. In all slots: If you find those red bars all across page 11 ('Emc TC Damage'), it is most probably due to the DAQ having been rebooted just before that particular run was taken. This is normal since at that point everything in fact is out of synch. You can double-check by looking at the next and previous runs.
  2. In one slot: Some UPC modules are known to create "out of Synch" error messages at low rate. The cause for this is unknown and will certainly remin unknown until the end of time. As long as the errors occurs in less than 10% of the events, the error can be ignored. You can always check with the DAQ on the "cmlog" screen if the error is still there and how frequent it is.
Crates go full Check if the problem starts in a particular ROM. The DAQ shifter should be able to tell from the Epics display. If not, watch the yellow 'full' LEDs come alive while the shift starts a run. If there is one particular ROM, see below.
ROMS go full or produce frequent errors Xyplex into the UPC and look for error messages. Record typical output by pasting it into a text file. Notify the Odf expert.
Chiller alarm The BaBar shift normally has to respond to EMC Chiller alarms, the instructions to put the EMC in a safe state are given in the Care and Feeding Manual. The manual also tells the shifter to either call the EMC shifter and/or the chiller expert. Switching ON a chiller MUST be done by a trained electrician using the correct PPE. You should also look at the description of the interlocks used by the chillers to shut down the EMC electronics in case of emergency.
If a chiller has been turned off, make sure that the corresponding power supplies for the frontend have been turned off as well.This should happen automatically via the interlock system, but it is always better to check.
Light pulser Refer to the lightpulser troubleshooting page

Top of page
This page is: /BFROOT/www/Detector/Calorimeter/Operations/EMC_Troubleshooting.html
Ivo Eschrich
Last modified: Mon May 2 17:20:29 PDT 2005 by Wolfgang Gradl