SLAC PEP-II
BABAR
SLAC<->RAL
Babar logo
HEPIC E,S & H Databases PDG HEP preprints
Organization Detector Computing Physics Documentation
Personnel Glossary Sitemap Search Hypernews
Unwrap page!
Det. Search
Who's who?
Meetings
FAQ
Images
Archive
Systems
Performance
Intern. region
Vertex Tracker
Drift chamber
DIRC
Calorimeter
IFR
LST
Magnet
Electronics
Trigger
Operations
Run Coordination
Contact Experts
Shift Takers Info
Operations Manual
Electronic Logbook
Ops Hypernews
Shift Signup
Check this page for HTML 4.01 Transitional compliance with the
W3C Validator
(More checks...)

Survival Guide for the DCH Shifters

[Shift Operations] [Shift Duties] [Contact List] [Current Problems]

Last updated: 19-Nov-2004, M. Kelsey
Note: Images in this documentation are currently somewhat out of date, but will be updated soon.

This is the ``Survival Guide'' for the DCH shifters. The intent is to summarize, in a way that can be looked up in a hurry while you are in panic (as you are probably doing now), the minimum essential knowledge related to the operation of the DCH that will help you pretend as if you know what to do. Please note:

  • You need to have at least passing familiarity to the DCH system, as it is impossible to describe all the basics without bloating this note to the point of unreadability.
  • There will be a larger and more complete body of documentation you should have read before the problem hit.
There is a Page of Current Problems which describes the (hopefully) temporary problems you should be aware of. If the page seems out of date, or if you discover a new and significant problem, contact the DCH Operations Manager.

Good luck.


Useful Links


Index

EPICS Ambient Database StripTool Gas Shack Water Chiller Electronics House

EPICS

EPICS stands for Enigmatic Panels Intended for Confusing Shifters. No, really. It's the best tool you have to check the status of the detector without the health benefit of walking up and down the staircases. You must learn how to play with it, and get familiar with all the panels related to the DCH.

See also: Vincent's Environmental Sensors page.

How to run EPICS

The EPICS display can be run on any bbr-con or bbr-dev machines in the Control Room. See Using EPICS Panels in the BaBar shifter's information for more information.

To start the EPICS display, assuming you are logged-in as babardch, type

    startEpics
This alias runs a script that sets up the environment, moves you to the root EPICS directory, and opens the BaBar main EPICS panel. It may fail if
  • you are not running on bbr-con or bbr-dev,
  • you haven't set up the DISPLAY environment varialbe right,
  • your display has run out of colors (Netscape causes this).
Once you get the main panel, click "DCH" button to get the DCH main panel.

Basic Anatomy of EPICS panels

Most BaBar EPICS panels share common design features to make your life easier (or slightly less painful):

  • Most panels have a ``Close'' or ``Quit'' button to close it. It's usually placed near the top-right corner or, less frequently, near the bottom-right corner.
  • Buttons with a right-pointing triangle are linked to sub-panels.
    • Left mouse button will bring up a default sub-panel for the button.
    • Right mouse button may bring up a list of sub-panels (if there are more than one) from which you can select the one of your interest.
  • Indicators are shown in one of the four colors depending on the status of the channel (e.g. the temperature of the electronics):
    Green = OK
    All channels should be in this state during normal operation.
    Yellow = Warning
    Channels in the warning state must be investigated and fixed.
    Red = Alarm
    Channels in the alarm state generally cause interlocks to operation, (e.g. you may not be able to ramp up the HV).
    White = Unknown
    The channel may be disconnected; there may be a network problem; the monitoring CPU may be hung.

DCH main panel

You can open the DCH main panel by clicking the "DCH" button on the BaBar main panel. The panel consists of:

  • Five buttons (left column) connected to subpanels: The colored box on the left of each button indicates the status of the particular part of the DCH system. The on/off status of the HV and LV are indicated on the right.
  • Runnable/Injectable flag display. "Runnable" means that the DCH system is ready for physics data taking. "Injectable" means that it is ready for PEP-II beam injection. The button labelled "more" is linked to the Status panel which shows the detailed status of these two flags.
  • Access control switches:
    • Shifter permisssion switch is used by the BaBar shifter to allow a HV expert (e.g. Fulvio) to perform dangerous operations to the HV.
    • Superuser HV at work switch is used by the HV expert to exercise his superpower.
    Don't mess with these switches.
  • Alarm and interlock button is linked to the subpanel that displays the status of the SIAMs in the DCH slow control crate. (See below.) You can also reset the alarms (provided that the alarm condition has been cleared) through the sub-subpanels. The "Guidance" buttons give you good explanation of the origin of the alarms.
  • Technical button is connected to an expert-only subpanel you don't want to mess around with.
  • Help button brings up a help window (thanks to Vincent).

High voltage panel

The "compact" HV panel presents the status of the HV system, and allows you to switch it on and off. It can be reached from the DCH main panel by clicking the "High Voltage" button.

  • Three light-blue buttons set the HV to one of the three possible states:
    • OFF = Turned off.
    • Injectable = Safe voltage for PEP-II injection. 1200V nominal.
    • Runnable = Full voltage for data taking. 1960V nominal.
    These buttons can be pressed any time, including when the HV has tripped, and the system will (at least try to) do your bidding.
  • Two thermometer-style gauges indicate the voltage and the current. The full scales (1960V and 1760uA) are indicated at the top.
  • The blue "Global Control?" button, when pressed in, transfers the control to the BaBar shift leader so that he/she can set the DCH HV in either Injectable or Runnable state using the BaBar status panel. You cannot control the HV while global control is in effect.
  • The status indicators show the current state of the HV system.

Clicking the "MORE" button on the compact HV panel brings up the "detailed" panel.

  • The cross-shaped part displays the voltage or the current of either the sense wires (nominal 1960V) or the guard wires (nominal 340V). You can switch among the four possibilities by clicking any of the "Sense: V", "Sense: I", "Guard: V", "Guard: I" buttons in the upper-right corner. The arrangement represents the geometrical segmentation of the HV channels on the chamber, seen from the rear end. The color (green, yellow or red) of the numbers displayed represents the status of the channel.
  • A smaller version of the cross in the bottom-left corner displays the voltage or the current of the zipper wires, depending on the status of the "Zipper: V" and "Zipper: I" buttons. The zipper wires are located at the innermost radius of layer 1 and the outer most radius of layer 40, in order to shield the cells from the electrostatic effect of the inner and outer cylinders.
  • The small triangles around the "cross" displays are linked to panels that displays detailed status of the HV channels in each superlayer.
  • The orange box in the upper-right corner contains three indicators for the overall status of the HV system and the "Runnable" and "Injectable" flags.
  • The orange box in the bottom-right corner contains the same set of control buttons as the "compact" HV panel. There are indicators for the current V0/V1/I0/I1 settings below the switches. These values are read back from the first channel of the HV power supply.
  • There are a bunch of detailed status indicators in the top-left corner.
    • The top-left part has 8 colored circles indicating the hardware status. They must be all green, or you cannot operate the HV.
    • The top-right part has 9 boxes indicating if any of the HV channels are in a particular state. This is summarized into the global state you see near the HV on/off buttons.
    • The bottom-left part shows the status of the V0/V1 and I0/I1 selection lines. They have quite a few seconds of delay.
    • The bottom-right part shows if all the channels are on or off.

How to Control the HV

WARNING: This is by far the most delicate operation you are likely to do on the BaBar Drift Chamber. Read the instructions. Use your common sense (assuming you have one) and minimize the risk of destroying the chamber we have spent so much time and energy for. Now to the business...

  • First, make sure there is nobody working either on the chamber or on the HV power supply.
  • Confirm with the BaBar shift leader that he/she wants you to control the HV, and the background is supposed to be acceptable.
  • Open the DCH main panel. Make sure that:
    • The HV system is in a good state (green).
    • The "Other sens. & Micr." indicator is green. Specifically, the humidity in the front and rear bulkhead must be below 10%.
    • The gas system is also green. The isobutane concentration must be around 20%.
  • Check the "Global Control?" button. If it says "YES", click it so that it says "NO".
  • Now operate the switch:
    • To turn off: Click "OFF".
    • To go to Injectable: Click "Injectable".
    • To go to Runnable: Click "Runnable".
    Huh, wasn't it easy? The status indicator should now show "Ramping Up" or "Ramping Down" depending on which way you are going.
  • Watch the voltage and current (especially the current) and see if it ramps up safely.

How NOT to Control the HV

During normal operation, the DCH HV should be kept under the global control by default. This will allow the BaBar shift leader to control our HV together with the SVT bias voltage and the IFR HV. It will also allow you to leave the control room. To enable global control:
  • Ask the shift leader which state (Injectable or Runnable) the BaBar detector is in.
  • If the DCH HV is not in the same state, set it to the same state.
  • Click the "Global Control?" switch so that it says "YES".
  • Tell the shift leader that he/she is in charge.
Now you are done. Don't forget to that cell phone!

How to Recover from a HV Trip

When a HV trips occurs, the HV system goes into an error state and this causes the BaBar DAQ to pause. It also inhibit PEP-II from injecting beams. To recover from this condition:
  • Find out which channel has tripped. Open the "detailed" HV panel. The ``cross'' display on the HV EPICS panel should have turned red for the tripped channel. Click on the triangle next to the superlayer number to bring up the detailed status panel.
  • Find out the status of PEP-II. It might have lost the beams. It might have gone back to the beam tuning mode. Ask the shift leader if you should attempt to reset the trip.
  • If the "Global Control?" button says "YES", click it so that it says "NO".
  • Click "Injectable" if the beams have been lost and PEP-II needs injection. Click "Runnable" if the beams are still there and BaBar wants to continue taking data. If you are not sure, try "Injectable".
  • Wait up to 30 seconds. See if the channels recover.
  • If it didn't work, Click "OFF". Note: Going to "OFF" does not clear the "tripped" status! (I know this is weird.) Once everything is off, click either "Injectable" or "Runnable" to bring the HV back up. This will clear the "tripped" status.
  • Record on the e-Logbook which channels tripped, and the circumstances such as:
    • Was the HV at V0 or V1?
    • Were the beams stable?
    • Were the beams lost after the trip?
    • Did any other subsystems (e.g. SVT) trip?

Low voltage panel

The low voltage panel has three sections:

  • FEA LV indicator looks like a bulls-eye. The circles represents inner, middle and outer FEA boxes in 16 wedges, viewed from the rear end. Actual voltage values (nominal 5V) can be checked by clicking the triangle next to "ALL QUADS."
  • Power Supplies indicator is a simple red-or-green thing. Clicking the triangle gives you actual readings of the voltage (nominal 8V) and the current (around 200A) of the LV power supply.
  • GLINK indicators tell you the status of the fiber-optic transceivers in the Trigger I/O modules and the Data I/O modules. The Trigger I/O modules report only the power supply voltage (nominal 5V). The Data I/O modules report also the temperature and the lock status (locked or unlocked). The optical links must be "locked" in order to be able to transfer the data.

Temperature panel

The temperature panel has five sections:

  • FEA temperature indicator looks like a bulls-eye. The circles represents inner, middle and outer FEA boxes in 16 wedges, viewed from the rear end. Actual temperature values (between 30 and 50 degrees C) can be checked by clicking the triangle next to "ALL QUADS."
  • End plate temperature indicators show the temperature of the front/rear end plates. Actual temperatures (20 to 25 C) can be seen by clicking on the triangle.
  • "Cylinders at End Plates" means the temperature of the inner and outer cylinders close to the end plates. Actual temperatures (20 to 25 C) can be seen by clicking on the triangle.
  • GLINK indicators tell you the status of the fiber-optic transceivers in the Trigger I/O modules and the Data I/O modules. The Trigger I/O modules report the temperature of the transmitters (around 25 C). The Data I/O modules side is linked to the identical panel accessible from the low voltage panel.
  • "Chiller System" indicator shows the status of the water chiller. Clicking on the triangle shows a panel displaying the supply/return water temperature of the three water cooling circuits, as well as the status of the flow switch.

Other sensors and microcontrollers panel

The "Other sens. blah blah" panel has 5 sections, of which probably only one is interesting you.

  • End plate status panel shows the humidity and the accumulated radiation at the front and rear end plates. Clicking the triangles brings you either to:
    • the humidity panel showing the relative humidity inside the bulkhead containment volumes. Note that the bulkhead areas are both flushed with dry nitrogen, and the humidity should normally be around 5% or lower. Higher humidity indicates either
      • The bulkhead flush gas is not flowing (check it in the gas shack), or
      • The bulkhead cover is open.
      High voltage cannot be ramped up if the humidity is higher than 20%.
    • The radiation level panel showing the accumulated dose measured by the RadFETs. Note that the RadFETs have two different sensitivity ranges (rad and krad).

Gas system panel

See Vincent's Gas System page for a more authoritative guide.

The gas system panel has a big indicator showing the current status (or "mode") of the gas system. It should say "Running mode" with a green box on the right.

Several subpanels are linked to the gas system panel. The most useful one is the shifter's summary panel, which is linked to a large friendly button (labelled succinctly as "Panel") surrounded by a white rectangle. This subpanel shows all the essential parameters you are likely to be interested:

  • The current running mode, which should usually be "Running Mode", and the status of the gas system, which should be green.
  • Gas sampling point, which should usually be "output".
  • Isobutane concentration after the pressure correction (this makes it a little different from what you read in the gas shack).
  • Oxygen concentration. This is usually below 10ppm after the oxygen scrubber was installed.
  • Water concentration. This is usually around 10ppm.
  • Atmospheric pressure.
  • DCH pressure, which is controlled to 4mbar.

The gas system panel also has a help button which is actually linked to some helpful information.

Status Panel (Runnable/Injectable)

The DCH Status Panel shows the current state of the "state machines" that determines if the DCH is:

  • Runnable, i.e. suitable condition for data taking. This means no alarms and the HV at V1.
  • Injectable, i.e. suitable conditin for PEP-II beam injection. This means no alarms and the HV at V0 or lower.
There are two buttons on this panel that allows you to bypass the state machines and force the Injectable/Runnable flags to be true.

How to Bypass Injectable/Runnable

WARNING: You are about to defeat the protection mechanism which are there for good reasons. You must have even better reasons why you should bypass them.

  • Understand what is causing the problem. (Is it a gas alarm? Is the chamber on fire?)
  • Make sure that the cause of the problem is being taken care of. (Call the experts.)
  • Make sure that the chamber is in a safe state. (The HV must be V0 or lower to allow Injectable.)
  • Think - Are you sure that it is a good idea to bypass? This is a good point to page the commissionner if you feel unsure.
  • Ask the shift leader to depress the blue "Shifter Permission" button on the Main DCH EPICS panel.
  • Depress the blue "SU HV at work" button on your side. If this does not work, see Superuser button problem below.
  • Open the Status panel. Click on either "Injectable Bypass" or "Runnable Bypass".
  • The flag is now bypassed. Write it down on the e-Logbook. Tell the shift leader you are now bypassed.
  • Remember to unbypass!
You must stay in IR-2 while either Injectable or Runnable is bypassed. I.e. you cannot go on-call while the protection mechanism is defeated. If this causes excessive physical discomfort for you, that means we should not be bypassing Injectable/Runnable so long. Demand the expert to solve the problem.

Superuser Button Problem

If you press the "SU HV at work" button and it pops right up,

  • Make sure that the username is babardch.
  • Make sure that the shift leader pused the "Shifter Permission" button.
  • OK, you've got a problem. DCH-MON has forgotten who is allowed to press which button.
  • Open a terminal window. Type:
       telnet bbr-tty1 3600
    
    You are now logged-on to DCH-MON IOC.
  • Type:
       asInit
    
  • Close by typing control-], then "quit".
You should be able to press the "SU HV at work" button now.


Ambient Database

What is it?

The "Ambient Database" is (you guessed it) another one of those Objectivity databases. Pretty much everything you see in the EPICS are recorded continuously (that is, when it's not broken for one reason or another) into it. You can retrieve it either using the Browser, which can plot the values of each EPICS variables against time, or the Ntuple Maker.

NB: Owing to the limited stability of the current software, the data is available for retrieval only after certain delay (about 30 minutes). You must use the StropTool to monitor the up-to-date history of, e.g. the HV current.

How to run the Browser

To start the Ambient Database Browser, assuming that you are logged-in as babardch, type:

    startBrowser
This alias moves you to the database directory and opens the Browser. More information will appear here when I know more about it.

How to run the Ntuple Maker

See the HyperNews by George Zioulas.

StripTool

The StripTool plots several (up to 10) EPICS variables in the way an old-fashioned chart recorder would, except that:

  • the plot moves right to left rather than top to bottom;
  • you lose the data as the plot flows outside the screen;
  • getting hardcopy is trickier than it was with a chart recorder.
On the plus side:
  • you can choose the channel (assuming you remember the name), adjust the range, change the color, etc., all on the fly.
  • you don't run out of paper.

How to run the StripTool

To start the StropTool, assuming you are logged-in as babardch, type:

    cd
    NewStripTool
A big grey window titled "StripDialog" appears. Click on "File" menu and select "Load Configuration...". Select "dchmon.stl" in /u/ec/babardch. Click "OK".

Once you get the chart window, you can probably click around to find out what to do. Tip: right-click in the chart window will get you a command menu. Another tip: only 10 variables can be monitored simultaneously. If you want to add another variable, you must remove one.

How to get a Hardcopy

I think the following instruction is obsolete. Right clicking on the chart window should give you an option of printing directly. But I have to check it myself first...

To get a hardcopy of the StripTool window:

  1. Run xv. This pops up a small green window.
  2. Right click in the xv window. This pops up a "xv controls" window.
  3. Fiddle with the StripTool until you get the right picture.
  4. Make sure that the StripTool window is not obscured by any other window AND the "Grab" button at the bottom-right corner of the xv control window is visible.
  5. Click the "Grab" button.
  6. Middle-click anywhere inside the StripTool window. This copies the window into the xv window.
  7. Click "Save" button in the xv controls window. This pops up the "xv save" window.
  8. Select PostScript, specify the file name, and click "Ok". This pops up the "xv postscript" window.
  9. Click "Maxpect", and click "Ok". DON'T try "Landscape".
  10. Click "Quit" in the xv controls window.
  11. You've got the postscript file! Print it and stick it on the logbook.


Gas Shack

As you enter the Gas Shack

First thing: Check the indicater on the right of the door. Does it say ``OK to enter'' in green? If not, Don't Enter. You have better ways to risk your life than to sniff hazardous gases.

Where are the racks?

The first rack (B636-03) facing the entrance has

  • The HAD sensor displays. Numbers displayed are % of the lower explosion limit (LEL). Any number greater than 10 means a trouble.
  • A VME rack full of SIAMs that interlocks our gas system. Watch out for red indicators on the SIAMs. If there are any, look it up in the channels list (posted on your left) and see if that's related to your problem.

Turn to your right now, and you find a row of racks used by the DCH gas system.

The rightmost rack (B636-02) has

  • The DCH gas pressure/mixture controllers. There are a bunch of pressure gauges, most of which labelled rather well. It is a delicate feedback system that maintains the chamber pressure slightly (4 mbar) above the atmospheric pressure while partially recirculating the gas. The differential pressure is displayed on a ``DCH Pressure'' gauge. Anything that goes wrong in this rack requires a gas-system expert to fix.

The second-from-right rack (B636-04) has

  • The System Status panel at the top. All the LEDs must be green except for the one labelled ``C4H10 not present'', which may be red while the chamber has isobutane in it (i.e. while we are running).
  • The flowmeters (good old rotometers are used there) for the inert gas (i.e. nitrogen or CO2). The gas flushes
    • Front endplate region
    • Rear endplate region
    • Outer cylinder
    • Exhaust gas from the system (to make it unflammable)
    The flush gas for the endplate areas can be switched between inert gas (while we are running) and air (during access to the electronics). The valve for this switching has a pedlock on it to make sure it won't be accidentally switched to nitrogen during an access. Warning: Attempting to access the electronics without switching the flush gas to air is harmful to your health!

The third-from-right rack (B636-05) has the gas analyzers.

  • The isobutane analyzer
  • The oxygen analyzer
  • The water analyzer
The analyzers sample the gas from one of the few possible sample points, selected by solenoid valves. One of the transparent buttons on the panel below the analyzers turns green, indicating which sample point is being used. You must select either ``Input'' (to the chamber) or ``Output'' in order to analyze the gas flowing through the chamber.


Water Chiller

Where is it?

The DCH Water Chiller is located on the Electronics House side of the shield wall. It sits among all the other subsystem's chillers, on the far side (seen from the control room) of the Electronics House. The DCH chiller can be identified by the label as well as by the down-to-earth bare aluminium enclosure (others are all painted).

Control Panel

The DCH chiller has a very simple control panel.

  • There are three sets of temperature controllers and flow indicators. Each set corresponds to one of the three cooling water circuits:
    • Front endplate cooling
    • Rear endplate cooling
    • Electronics cooling
    The temperature controllers are all set to 18 degree C. The indicators show either
    • Flow OK (green), or
    • Low flow (red).
  • There is an extra temperature display for the return water from the electronics. The normal temperature is between 19 and 20 degree C.
  • There is an array of LED indicators labelled ``Main'':
    • Level warning (yellow) indicates slightly low water level in the reservoir.
    • Low level (red) indicates too low water level.
    • Flow OK (green) indicates good total flow.
    • Low flow (red) indicates low total flow.
  • There are two push buttons:
    • Black button to start the chiller.
    • Red button to stop the chiller.
  • There are three sets of heater indicators. Each set has two LEDs:
    • Heat on (yellow) indicates the heater being on.
    • Overtemp (red) indicates the water being too hot.
  • There are two round pressure gauges:
    • Supply pressure (left) is normally around -1 psi.
    • Return pressure (right) is normally around -18 psi.
    The system runs under ``negative pressure'', i.e. the pump ``sucks'' the water instead of ``pushes'' it.

How to start

When the DCH chiller trips, you should first try to identify what's gone wrong. Once you cleared the problem,

  1. Push the big black button and hold it.
  2. Wait for 30 seconds, or until all flow indicators turn green.
  3. If it fails after 30 seconds, try it again.
  4. If it fails in the second trial, call an expert.
This simple procedure may fail, for example
  • If the chilled water is cut off. In this case, all other system's chillers will fail, as well as the cooling inside the Electronics House.
  • If the water temperature is very low or very high. This must be cleared by an expert by temporarily adjusting the temperature controller setting.


Electronics House

Most of the DCH electronics is in the first floor of the Electronics House. You should know where our stuff are just in case they catch fire. (By the way, there is an automatic fire suppression system in the Electronics House. So don't try to get in there if there is a fire alarm.)

Power cut! What to do?

There really isn't too much you can do while the power is out in the Electronics House. Restoring the power is the BaBar shifter's responsibility and you should just wait and see if things come back. Once the power is back, check things in the following order:

  1. High voltage supply. Check if it is powered, and it has the green "check passed" LED on.
  2. Low voltage supply. Check if it is powered.
  3. Slow control crate. Check if it is powered. Reboot the IOC once again so that it will start communicating with the HV supply.
  4. DAQ crate. Check if it is powered. Check if all the ROMs have the green "i960" LEDs on. You may have to wait until the network switch and the file server comes back on.
  5. EPICS panels. Check if it is back to life. If the DCH panels are all white, the IOC did not boot; see What to do when IOC doesn't boot. Go through the panels to find anything bad. You may have to wait for the other part of the system, e.g. the Fast Control Master crate, to recover before you get everything working.
    If the EPICS panels show white (disconnected) for the forward endplate monitors (temperature, humidity, etc.), you should switch off, then on, the power button on the module located in the DCH rack (electronic house). This module is located on top of the VME crate, and is labelled 'Dch Fwd End Environmental Monitor'.
Keep your cool. Call the experts whenever you are in doubt, or as soon as you start to panic.

DAQ crate

The DCH DAQ crate is located at the top of rack B620B-27. It is a standard BaBar DAQ crate, i.e. a 9-U VME crate with a split backplane. It has 5 ROMs (Read Out Modules), a digital delay generator, and a Fast Control Distribution Module.

The crate itself has a few switches and a display panel which can show the temperature and the power supply voltage, current of the crate. It's usually showing the air temperature, which should be around 23 degree C. The "mode" switch, immediately on the left of the display, toggles the display through the possible items.

Read Out Modules

The DCH DAQ system uses 5 ROMs. The left-most ROM (aka "slot-1 ROM") collects data from the other 4 and sends it to the event builder. Each of the four "garden-variety" ROMs reads data from one quadrant of the DCH. The data is transported from the front-end electronics through the optical fibers connected to the upper one of the two front-panel connectors of the ROMs. (There fibers in the lower connectors are dummy.)

All ROMs have a rather dazzling array of LED indicators. Knowing a few of them may be somewhat useful:

  • A red LED above the fiber connectors indicate that the communication to/from the front-end electronics (Data I/O Modules) is lost. This may mean that the low voltage is off, that the Data I/O Module got confused, or that the fiber has gone bad.
  • A green LED labelled "i960" should be on. This means that the ROM is ready to receive data. If this LED is off, chances are that the ROM failed to boot and you need to reboot it.
  • A yellow LED labelled "full" may come on during data taking when the ROM is overwhelmed by the data size. More likely causes are:
    • The data from the front-end is corrupt and the feature extraction is busy in error handling.
    • The network is slow and the data doesn't go out from the ROM.
    • The DAQ software is hung or crashed.

Digital Delay Generator

A delay generator is used to generate trigger signals for the online calibration. The module is a normal 6-U VME item which looks awfully out of place in the 9-U crate. A red "TRG'D" LED should be brightly on as long as the Fast Control Master Crate is alive. There is also a less bright "OVEN" LED which tells you that the time-base oscillator in this module is being temperature controlled.

Fast Control Distribution Module

Don't worry about it.

Slow Control crate (DCHMON)

The Slow Control crate for the DCH is called DCH-MON, and is located in rack B620B-10. This is a 6-U VME crate, with a very similar control for the switches and the display as the DAQ crate.

The crate contains quite a few modules, most of which are a little too obscure for you to know except for:

  • The CPU (also called the "IOC" for I/O Controller) is on the left-most slot. It's a Motorolla MVME 177 (a good ol' 68k series single board CPU). It's job is to scan the I/O modules such as the SIAMs and the VSAM and communicate with the EPICS.

    Sometimes, for example after power cycling the HV supply, you have to reboot this CPU. Do it by pushing the red recessed button on the front panel. Remember that the EPICS panels go blank and, more importantly, the injectable signal to PEP-II is lost while the CPU boots, which takes 1 to 2 minutes. So make sure to talk to the shift leader before pushing the button.

  • SIAMs. They are safety interlock devices, except for SIAM1 (the one with a silver front panel) which is setting the injectable flag. The meaning of the SIAM inputs and outputs are shown on a sheet of paper posted above the DCH-MON crate.

What to do when IOC doesn't boot

So you've got all the EPICS panels blank. You've pushed the red button on the IOC a couple of times and it doesn't help. This probably means that the IOC in the DCH-MON crate is not booting for some reason. You should also notice that:

  • There is a red LED labelled "sysfail" on near the power switch of the VME crate. A few of the VME modules are also reporting "sysfail" on their front panels.
  • When you push the IOC's "reset" button, the red "FAIL" LED comes on for a few seconds and goes away. Nothing else happens. If, on the other hand, you see the green "RUN" and "LAN" LEDs flickering, the IOC is booting OK and the problem is in the EPICS software.
Now you know that the IOC is not booting. Do the following:
  1. Open the back of the rack. Look at the back of the crate. Near the top-right edge, find a beige plastic thing that looks as if it would accept two phone cable connections. This is the serial terminal connection for the IOC.
  2. If a cable is connected to the serial terminal, unplug it and try to reboot without this cable.
  3. Didn't work? Too bad. You need to call either the detector-control or HV expert.

High voltage supply (CAEN)

The high voltage power supply is in rack B620B-11. The system is called SY527, made by CAEN. The front panel has a barely readable LCD screen, flat-panel style control buttons, bunch of LEMO connectors into which a bunch of LEMO cables are connected, and a few items of your interest:

  • A power switch with a key. It should be kept on the "local" side. An expert may pull off the key to lock it out during a hardware intervention, which you probably don't want to get involved.
  • A "MAIN" indicator lights up when the power is on.
  • A "310V" indicator lights up when the internal circuit has the power. This stays on for a while when the power is turned off, until the capacitor is emptied.
  • A "CH ON" indicator lights up when at least one high voltage channel is on.
  • An "interlock" toggle switch. This should be at "open" position during normal operation, otherwise the system is interlocked.
  • An "enable" toggle switch. This should be at "remote" position (down), otherwise the system cannot be operated.
The last two toggle switches are sometimes used by experts to keep the system off together with the power key. If you find these switches in an unusual position, find a HV expert and ask what's going on. Don't flip the switches by yourself! Always find a scapegoat first.

Low voltage supply

The low voltage power supply is also in rack B620B-11, below the high voltage supply. It's labelled "20T250" in big letters, which means "a big power supply." It has a manual switch, two knobs to set the voltage and the current limits, and LCD indicators for the voltage and the current. The voltage is set to 8V, and the current is normally around 215A. You shouldn't have to touch any of these.

The power is sent to the front-end electronics through a pair of Really Thick cables. By the time it arrives at the rear bulkhead, the cable loss makes the voltage to about 6V (we should have measured this already, but haven't). The electronics in the rear bulkhead has voltage regulators in each single unit (actually more than one in each box) that drops the voltage to 5V. The voltage you see in the EPICS panel is the output of these regulators, which explains why they are so boringly uniform.


Comments to Michael Kelsey <kelsey@slac.stanford.edu>

Originally written by Masahiro Morii, 1998