Instructions to recover a failed DIRC HV supply

If a power supply fails, the HV panel in EPICS will either show a red OFF for one crate (two sectors) or a white NC.

If all crates are white at the same time, it is most probably a problem with the DRC-HV IOC. Please check the channel CEN -> IOC-STATS -> ZONE-1 -> DRC-HV first.

In case of OFF, press the blue ON button for both sectors and wait a few seconds to see if the crate reacts (it tends to feel pretty long). In case the HV crate come back up without a problem, start taking data again.

In case it does react or you have a NC, go to the electronics hut and restart the CAEN power supply.

Restarting a CAEN power supply

Check the supplies in racks 5 and 6 in the electronics hut.

Find the corresponding crate and check the LCD display. If there is nothing on the display, it has died.

If you have not yet informed the DIRC commissioner, page him now and explain your observations.

Before trying to repair the crate, try to restart the crate.

In the lower right of the crate, two cables with Lemo connectors are plugged in (crate 6 has only one cable and a terminator). Unplug both cables. Note: There is no need to remember which was in the left plug and which in the right as they can be interchanged.

One of the crates has a key inserted in the on/off switch at the lower left of the crate. Pull that key out (do not turn the key) and insert it into the dead crate. Turn the key to the middle (Off) position. Wait 30 seconds and turn the key to the Local On position.

If you find the key in the Off position, do NOT switch on. Someone might be working at the detector and might have switched it off on purpose. Either wait for the commissioner or check that there is no access or both backward doors are closed before turing the HV crate on.

You should see the crate initialize. Wait till the light between the connecters you removed has blinked and you hear a beep. Then plug the two cables back in (it does not matter which cable goes where).

Check that the LCD display shows the welcome screen.

After some time, in EPICS, the HV crate should be back in either OFF or ON mode. If the crate is still NC, you will have to reboot the IOC.

If the crate does not come back on or fails after a short time again, go to the troubleshooting section below.

Rebooting the DIRC HV IOC

IOCs should not be rebooted by shifters: it is better to page ODC first and to let the expert investigate the origin of the problem. When he/she is done, he/she will reboot the IOC properly.
If you want to reboot the IOC despite the above advice, the corresponding instructions can be found here.

Troubleshooting

If restarting the crate does not help, check for the following two failure modes we have seen. Check whether the LCD display is showning some information and check the two lights on the left side of the crate: the MAIN and 310V.

• LCD dead, but (one or two) lights are one : The internal 5V power supply might have died, the whole main frame of the power supply has to be changed.
• LCD and both lights are off: The fuse in the back is blown.

In case everything is off, replace the fuse. Otherwise, replace the main frame.

Replacing the fuse of a DIRC HV crate

Note to shifters: Do not try any repairs that you do not feel comfortable doing. In case you are not an expert on electronics repair, wait for the DIRC expert.

Do the following steps:

1. Switch off the crate by turning the key to the off position.
2. Check the rack number and height of the crate.
3. Go to the back of the rack and open the door.
4. Unplug the power line. This is done by iteratively turning the metal ring for a few turns and then pulling at the plug. After a few iterations, the plug come off.
   
5. Wait till the red light is off. Be careful. As long as the light is still on, charge remains in the capacitors and thus some parts inside the crate have high voltage applied.
6. Take a screw driver or a coin and unscrew the fuse holder.
7. Visually check that the wire in the glass tube is broken. If not, check the other fuse.
8. Get a new fuse from the upstairs lab area (the room next to the coffee room) close to Ray Rodriguez office. Find a matching fuse (20A, 220V).
   Troubleshooting: if you don't find new fuses quickly enough, you can always 'borrow' those of the spare HV crate! In that case, do not forget to put new ones there later and make sure they are working.
   
9. Put the fuse into the holder and screw the holder into the crate.
10. Reconnect the power plug and fasten the metal ring.
11. Try to switch the crate on using the method described above in Restarting a CAEN power supply.
12. Make a logbook entry.

Replacing the main frame of a DIRC HV crate

Note to shifters: Do not try any repairs that you do not feel comfortable doing. In case you are not an expert on electronics repair, wait for the DIRC expert.

In order to do this, you need at least two people. Three to four is best.

Do the following steps:

1. Switch off the crate by turning the key to the off position. Unplug the serial RS232 connector and the other two cables.
2. If you have more than two people, let two do this step, while the others continue with step 3.
   Go upstairs to the lab area (the room next to the coffee room). In the back of the room at the DAQ test stand is a spare crate mounted. Unplug the power line (see also point 5), unscrew the crate (see also point 9) and bring the crate to the electronics hut. If there is no spare, borrow a spare from the SVT test stand. Inform the SVT expert that you took it !
   
   
3. Check the rack number and height of the crate.
4. Go to the back of the rack and open the door.
5. Unplug the power line. This is done by iteratively turning the metal ring for a few turns and then pulling at the plug. After a few iterations, the plug come off.
   
6. Write down the slots and the serial numbers of the HV boards.
7. Search for a box or similar where you can put the HV boards on. Depending on the elevation of the HV crate, the mail box or the foot step is fine.
8. Wait till the red light is off. Be careful. As long as the light is still on, charge remains in the capacitors and thus some parts inside the crate have high voltage applied.
9. Pull all eight HV boards out of the crate. The are not screwed in, but the small handles at the top and bottom ends of the boards might extend into the crate holder. Do not unplug the HV cables from the boards as they are only barely labeled with the channel number. If you have to unplug cables (normally you won't have to), never have cables from more than one board unplugged at the same time. It might not be possible to get the last two boards out, then do the next two steps first.
10. Go to the front of the rack. Unscrew the six screws.
    
11. Slowly pull the crate out from the front of the rack. Please be extremely carefull with your hands. There is absolutely no space for your fingers between the door frame and the crate. If your fingers get screezed between the crate and the door frame it will be very difficult to get them out again ! If you were not able to get all the HV boards out of the crate in step 9, carefully pull of the back the out while the crate is pulled out of the front of the rack.
12. Put the spare care in to the rack.
13. If you had problems to remove all HV boards before taking the crate out, you better put those boards back in before pushing the crate in completely.
14. Fix it with the six screws.
15. Put the (remaining) HV boards in.
16. Make sure the HV boards are pushed in completely.
17. Reconnect the power plug and fasten the metal ring.
18. Try to switch the crate on using the method described above in Restarting a CAEN power supply.
19. If everything works, put a repair tag on the broken supply.
20. Remember to check that the caenet address of the new crate is consistent with the crate number slot (these two numbers, integers between 1 and 6, should match). To modify the caenet address, turn the crate ON, touch one arrow on the front panel and browse the menubar until you find a menu called 'Caenet config.' (or something similar). There, one just needs to enter the crate slot and then to reboot the IOC.
    Troubleshooting An address conflict would have a characteristic signature in epics:
    • A very long (~ 10-15 minutes) IOC reboot time;
    • zillions of errors in the logfile, all pointing to the crate which has the wrong address;
    • after the reboot, the slot number 'shared' by two crates will show a 'NC' (white) status in epics while the boards of the slot which is not allocated will be missing (red instead of green dots at the bottom of the 'DIRC High Voltage Control' panel).
    These features should tell you which crate has a wrong caenet address. You can check your diagnostic by opening a xyplex session on this crate to verify that its address is indeed uncorrect.
21. Bring the crate to Paul Stiles' workshop after having let Ray Rodriguez know about the problem. Paul will take care of getting the crate repaired.
22. Make a logbook entry.
23. Add a trouble report to the Hardware Database.
24. Update the DIRC hardware status monitoring webpages linked to the DIRC Operations webpage.