Exploration: Empty both rings. Fill bucket zero only. Note that it was actually bucket 600 (!) getting the fill. Timing obviously very bad.
In IR6 temperatures are very high (even though crate temp only 38 or so). The Shared memory lights are behaving strangely, i.e. both Local and Remote should blink at about 1 Hz - Local (VXI crate containing Bunch Current Monitor) was at about 1/3 Hz.
Attempts to fix: Rebooted the HER VXI. Tried older versions of the DECIBOTH executable. Nothing helped.
What fixed it: Mike Stanek IPLed PR06. Timing returned to normal.
Probable cause: Timing device unhappy.
How this caused symptoms:
Recovery: Make sure bucket zero is bucket zero. Check phasing. Set Deconvolution values.
Exploration: With no beam in HER, note large bunch current monitor readout fluctuations.
In IR6 the door is still open, hence the temperature is fine. Examined contents of the HER shared memory from the BIC. Even buckets show raw ADC counts of about FFFFE000, odd buckets are about FFF0C00. This corroborates the discrepencies seen in the displays. HER local shared memory light flickering at about 1/3 Hz.
Fix: Recycled power in the HER VXI. The idea was to do a hard reset of the VXI crate. HER VXI NT rebooted upon power up. Things looked fine.
Probable cause: Fallout from previous evening. VXI in a funny state.
Recovery: As before - Bucket 0 - check phasing - check deconvolution - go!
Exploration: Note that the LER background levels on the expanded display also indicate large negative numbers. The Deconvolution display shows unprocessed data and the data are an even bucket/odd bucket sawtooth.
Examined the LER 60Hz data memory: Odd buckets all FFFF8000 or close. ADC probably setting only one bit occasionally.
Tried reseating the cables running from the Decimator to the ADCs. Tried switching the cables. Problem did not move with the cables.
Fix: As yet unknown. Hardware fault to be diagnosed on 10/5. M. Chin came and looked with us. We learned how to start the LabView diagnostics and ran into some VXI mode problems. The channels seem fine now. More diagnostics added to the VXI/VME interface for directly viewing the Decimator data. More diagnostics added to watch VXI interrupt rates, etc. as well.
Probable cause: One Decimator seems not to be delivering data. The interrupt seems to be generated at the wrong time, or the two Decimators get out of synch. Starting the LabView, then resetting the VXI for the normal running cured the problem.
Recovery: Terminate the normal data acquisition. Start up LabView to verify correct data input. Reboot the VXI NT to get the normal program restarted.
Exploration: Dump memory. Obvious pattern. Filling modulo(bucket,12) leads to no change there, but small increment at +2 buckets.
Fix: Stopped the DECIBOTH program. Reran VXI-INIT. Restarted DECIBOTH data acquisition.
Probable Cause: Messed up VXI while switching cables to test odd bucket problem.
Fix: Did the LER startup process (VXINIT, RESMAN, DECIBOTH)
Probable Cause: LER needs manual restart after power outage.
Next Symptom: Very bad fills in LER. HER ok.
Exploration: Look at expanded display. Odd buckets all very largely negative. LER Odd bucket problem has returned! Goal Ratio display displayed an interesting "fuzz" - no hint of regularity.
Fix: Reset the deconvolution function to a delta function; i.e. don't try to use the spill over into the odd buckets to clean up the buckets with current.
Probable Cause: Deconvolution function picked up real garbage. "Fixed" data wasn't very good.
Next Symptom: Both DCCT stop working.
Exploration: DVM Hardware had been reset after a BIC reboot. DVM reads correctly locally, but no data being read.
Fix: Hit the RESET button on the DCCT panel. NB: The GPIB functions invoked by these buttons take about 5 seconds to complete. Patience is required. Often two pushes are necessary due to GPIB magic. Pushing the button often and quickly will almost always fail.
As of November 9 the odd buckets in LER are still bad. A. Fischer has investigated and further work is planned when there is a ROD.
Exploration: Goal Ratio display shows this clearly. Also, individual buckets and some contiguous chunks of filled buckets are sometimes about .9, sometimes about .7 on that display. Scope display shows undershoot following filled buckets, especially for low bucket numbers.
Fix: Shut off Deconvolution.
Probable Cause: Deconvolution improperly processing non-goal buckets, due to their being over threshold. This really distorts the real data. Phasing probably needs to be set in another of the three possible "good" phases to avoid the undershoot and "ringing".
On the BIC, the LER data can be seen with the command: d 0xf0020000,20,4
For the HER, use d 0xf0010000,20,4.
The first four words are header, then come bucket 0,1,... These values are the sum of 60 readings each of 256 readings of the 8 bit ADC in the Bunch Current Monitor.
The HER VXI crate is on the right, the LER on the left. The monitor to do work with the VXI is under the LER crate. A switch on top of the VME crate switches the keyboard and monitor between HER and LER. The login prompt has either LER or HER in the name. The password for both is "bxbcm". The DECIBOTH window is green on LER and red on HER.