Maintaining Dataflow Hardware

Last update by cpo on 11/24/04


  • Introduction
  • About the IR2 Teststand
  • Teststand Setup (FCDM Dip Switch Settings and ACLs)
  • Short Instructions for Switching between IFR/EMC Setups
  • Recommended Tests
  • How to Connect to IR2 Teststand ROMs
  • How to Compile Standalone TPC/UPC/FC Software
  • Finisar Light Levels and Waveforms
  • Running TPC Standalone Test Software
  • Running Fast Control Standalone Test Software
  • Running UPC Standalone Test Software
  • Sample Output from UPC Standalone Tests
  • Running the VME DMA Tester
  • Running the I960 DMA Tester
  • Running Miscellaneous Utilities
  • Running TPC Core Dataflow Software
  • Running UPC Core Dataflow Software
  • Damage Bit Definitions
  • Running PPCBug Diagnostics
  • Setting ROM Boot Parameters (and What To Do if the ROM nfsMount Hangs)
  • Installing Kernel into MVME2306 Flash Memory using PPCBug
  • Reprogramming the Flashed Kernel from the VxWorks Prompt
  • Re-installing PPCBug and Other Flash Commands
  • SBC Memory Addresses
  • Some Schematics
  • FCTS Firmware Programming
  • I960 Firmware Programming
  • TPC Firmware Programming
  • CC Firmware Programming
  • UPC Firmware Programming
  • Xyplex Setup
  • System Ugliness That Should Not Be Forgotten
  • Why Repairs Are Frustrating

    Introduction

    This is an attempt by cpo to tidy up some of the testing utilities that were used for validating boards as they were produced. This also contains instructions for running dataflow test software, firmware programming, and schematics locations.

    Many of the instructions below only apply to the special isolated dataflow setup on bbr-dev15.

    The standalone software ("romtest","fctest","upctest") was originally written by a number of people:

  • leonid sapoznikov (fctest, romtest)
  • jeff olsen (upctest/upctest.c, multiple command files)
  • a uc Irvine grad student (upctest/barreltest.c and upctest/endcaptest.c)

    Other software (sbcTest, Odf) was written by core dataflow.

    Tests that are either missing entirely, or even more woefully inadequate than usual:

  • tests for the PMC board (leonid says that historically the PMC boards were tested indirectly using the CC/TPC tests). We have had one case where the PMC board failed to execute DMAs correctly, which is untested by the other standalone software.
  • more complete tests for the FCTM, FCGM
  • tests for the dataflow gigabit card

    About the IR2 Teststand

    Over the years, Ray has expressed some concern over the fact that the software he has been running has been changing frequently (on the timescale of board-repair marathons). An example of some of the changes: the EMC changing the format of constants, dataflow changing the directory structure of /dataflow to support LINUX, dataflow changing the VxWorks kernel. Most of the changes we make to core Odf should not impact the tests he runs.

    To try to create a more stable environment for Ray, bbr-dev15 has it's own private /dataflow area, and a second ethernet interface plugged into a small 100Mbit switch (along with the ROMs in his test crate). The second ethernet interface has the IP address and ethernet address of odf-srv02, so that ROMs taken from the electronics hut will boot dataflow correctly without changes to their NVRAM settings.

    Steffen has told me that this special /dataflow area on bbr-dev15 is being backed up daily.

    Two of the resources that are needed to maintain ROMs are tftp and rsh (for loading in the VxWorks kernel in various ways described below). Steffen has promised that those will remain enabled on bbr-dev15.

    There are two ways I can think of that the software can change in this environment without ray's approval:

  • upgrades to the solaris operating system in a non-backwardly compatible way
  • using a ROM with a different version of the VxWorks kernel

    At the time of this writing, the operating system on bbr-dev15 is SunOS58. The dataflow software and /dataflow/teststand/setup will have to be modified if that is changed. The VxWorks kernel version can be kept constant by Ray (if desired) by using the /dataflow/tgt/startup.save-kernel-to-flash script described below.


    Teststand Setup (FCDM Dip Switch Settings and ACLs)

    On a UNIX window on bbr-dev15,

    
    source /dataflow/teststand/setup
    
    This currently executes
    
    setenv ODF_PLATFORM 19
    setenv LD_LIBRARY_PATH /dataflow/package/shlib/SunOS58:/usr/local/lib
    setenv PATH /bin:/usr/local/bin:/dataflow/teststand:/dataflow/13/14/release/bin/SunOS58
    

    That setup may break other non-dataflow software, but is necessary to keep bbr-dev15 isolated, so that it's dataflow software is not broken by external changes.

    Two scripts are here: setodf and unsetodf. For different tests described below, booting of dataflow has to be either enabled or disabled. These two scripts do that. They must be run on bbr-dev15.

    FCDM DIP switches:

    Local ID 
    0-3    Detector Id
    4-7    Lowest bits of platform
    
    Secondary ID
    0-4    Crate number
    5-7    Upper bits of platform
    

    The crate number should be set to ZERO for the teststand.

    Make sure to read the SILK SCREEN to see which direction is 0 and 1, and also for the bit numbering.

    When dataflow boots, it executes the startup script /dataflow/XX/YZ/dfs/startup where

  • XX is the platform number in hex
  • Y is 1 for a slot0 ROM, and 0 for a slotN ROM (unfortunately, the intuitive ordering of those two is flipped for historical reasons)
  • Z is the "Detector Id"

    For repairing UPCs, set the detector-id dipswitch to 4 (EMC) in the FCDM, and put a TPC in slot0 and the UPC in slotN. Hook up the TPC fibers to the TRB, and UPC fibers to the IOB. The ROMs will boot dataflow from the following directories:

    
    /dataflow/13/14/dfs/startup  (slot0)
    /dataflow/13/04/dfs/startup  (slotN)
    

    For repairing TPCs, set the detector-number dipswitch to 5 (IFR) in the FCDM, and put the TPC in slot0 (or to test VME behaviour, put a second TPC in slotN). The ROMs will boot from the following directories:

    
    /dataflow/13/15/dfs/startup  (slot0)
    /dataflow/13/05/dfs/startup  (slotN)
    

    The /dataflow area on bbr-dev15 is protected by ACLs (like the official /nfs/bbr-srv02/dataflow area). At the time of this writing, the ir2odf unix group and user rayr are permitted to modify the area. Additional users can be added using the script OdfApps/acl/aclusr, in the CVS OdfApps repository. This has to be done as the superuser (or with sudo privilege) as follows:

    
    aclusr 7 rayr /dataflow
    

    Short Instructions for Switching between IFR/EMC Setups

    For the front-end electronics crate:

  • plug in the EMC front end electronics into the 208V plug in the back
  • turn on the TRB power (black switch on front of leftmost power supply)
  • then (order is important) turn on the IOB power (black switch on front of rightmost power supply)

    For the ROM crate:

  • put a TPC in slot0
  • put a UPC in slot1
  • plug the TRB fiber (rightmost in front-end crate) into the TPC fiber A
  • plug 3 single fibers into the IOB and UPC
  • eject FCDM. set "logical id" bit 0 to "0-on". read the SILKSCREEN to get the right bit and polarity.

    Recommended Tests

    For a PMC card:

  • make sure the board boots dataflow correctly (watch for odfBootTask error messages)
  • make sure version 8 socketed firmware chip is installed
  • make sure the PMC light of the ROM comes on (this is the i960 code polling the Controller Card for transitions/events)
  • run dataflow in two ways: once with a high rate of l1accepts and once using the calibration sequencer with 5 minimally spaced L1Accepts (see "Running Dataflow" section below).

    For all ROMs run

  • sbcTest (with board in both slot0 and slotN position)
  • The i960 DMA test
  • CCScreen (Note: currently CCScreen doesn't seem to work on UPCs. Somehow the "byte enables" tests always fail. Might be this feature (buscontrol) is controlled by the UPC, and it handles the byte-enables differently?).
  • linkinfo
  • run dataflow in two ways: once with a high rate of l1accepts and once using the calibration sequencer with 5 minimally spaced L1Accepts (see "Running Dataflow" section below)

    In addition for UPCs:

  • barreltest (for barrel UPCs) or endcaptest (for endcap UPCs)

    In addition for TPCs:

  • TPCScreen (the TPC needs a loopback fiber for this test)

    For repaired TPCs, I recommend that they be inserted into SVT crates in ir2 immediately after repair (for a test run, turn the SVT bias off to increase the event size). The SVT feature extraction code does some significant data integrity checks. Watch for "svt damage" messages in the ROM xyplex. Note that this is different from dataflow damage. Also ask the DQM to be vigilant.

    There is a piece of ROM software in OdfApps/mtest/svtRawCheck.cc which is able to recognize and dump out a subset of the conditions which give SVT damage. The bad regions are marked with "*" and "^". If that isn't enough, The SVT has a feature where their ROM software saves the last 4 events with SVT damage (and perhaps dataflow damage, I don't know). If you see damaged events, you can dump them by typing svtTestStand(1) at the vxWorks prompt, and then use the "d" command to dump out a saved event.

    For Fast Control Boards:

  • the appropriate fast-control standalone test documented below
  • run dataflow (as described in the section below) with either a TPC or a UPC setup

    How to Connect to IR2 Teststand ROMs

    To connect ("xyplex") to the ROMs in the IR2 teststand do the following on bbr-reflin ONLY (this was restricted to this machine by steffen for security reasons):

    
    xyplex -f tt16-01 (slot0 ROM)
    xyplex -f tt16-02 (slotN ROM)
    

    How to Compile Standalone TPC/UPC/FC Software

    To compile, first you have to source the dataflow windriver setup, which is at the time of this writing:

    
    source /afs/slac/g/babar/package/WindRiver/windSolaris-1.0.1.d
    

    Next, check out the packages (CVSROOT environment variable must be set at the time of this writing to /afs/slac/g/babar/repo):

    
    cvs co OdfHWTest
    

    Change directories to any of the three test packages (romtest, fctest or upctest) and type make. That will compile .o files that can be used. There are also vxworks scripts present (labelled with the .cmd suffix) that are useful.


    Finisar Light Levels and Waveforms

    This small software program is in CVS in OdfApps/mtest.

    For this test, core dataflow must be loaded into the ROM and running, and loopback, IFR, or EMC front-end fibers should be plugged into a TPC ROM, and the front-end IOB fibers should be plugged into a UPC ROM.

    At the ROM prompt, type

    
    ld < /dataflow/tgt/linkinfo
    linkinfo
    

    TPC light levels should be >-10dBm, although I don't feel like I have alot of experience on the TPC (finisar docs say that it should work if >-15dBm).

    UPC light levels should be >1000mV (which is -7dBm). Empirically, if UPC light levels fall below this value, there are intermittent damage problems, which is one of the fundamental unfixed problems in the EMC electronics.

    Some Finisar Waveforms:

    Dave Hamilton and I (cpo) used a fancy 1GHz TD5104B scope that with a differential probe we borrowed from Dave Nelson and Gunther to look at waveforms. These are when we are receiving "data frames" where the data is zero. You can look at the glink document to understand the structure of these data frames, although there is a feature where we see "one high" or "one low" where we would expect to see "two highs" (from the 4 control bits in the data frame). Not understood.

    Board 2300 Receiver A (board showing link errors on both links)
    Board 2300 Receiver B (board showing link errors on both links)
    Board 2300 Receiver A, triggering on ERROR (board showing link errors on both links)
    Board 1933 Receiver A (a good board)
    Board 1933 Receiver B (a good board)
    Board 0852 Receiver A (board showing link errors on both links)
    A UPC that has link errors

    Mike Przybylski also used the same scope/probe to look at tpc and upc waveforms:

    SLAC ID 20001933 TPC Waveform
    SLAC ID 20001338 fiber C (finisar receiver) UPC Waveform

    And here are some attenuated and unattenuated waveforms to TPC/UPC:
    Unattenuated (UPC 20001338)
    Attenuated (UPC 20001338)
    Unattenuated (TPC 20001933)
    Attenuated (TPC 20001933)

    No obvious TPC/UPC differences with the attenuation.


    Running TPC Standalone Test Software

    For this test the dataflow startup must be moved aside script so dataflow software is not loaded or run. Reboot the ROM if necessary with ^X or "odfVmeReset".

    NOTE: For these tests, you need a loopback fiber plugged into both A and B finisars!

    ALSO NOTE:Currently the buffer control register test fails routinely, as follows:

    ***************************************
    * Testing the Buffer Control Register *
    ***************************************
    
    *ERROR* Read 0x00000000 expected 0x00000001
    *ERROR* Read 0x00000000 expected 0x00000002
    *ERROR* Read 0x00000000 expected 0x00000004
    *ERROR* Read 0x00000000 expected 0x00000008
    *ERROR* Read 0x00000000 expected 0x00000010
    *ERROR* Read 0x00000000 expected 0x00000020
    

    On the ROM type:

    cd "/dataflow/OdfHWTest/romtest"
    <romtest.cmd (to load files)
    TPCScreen
    CCScreen
    ROMScreen (which runs TPCScreen and CCScreen)
    
    TPCScreen program runs ALL tests on the TPC. If it passes TPC is fine, if not, you have to fix something. Runs:
  • TestTPC (tests all registers and IS memory for consistency)
  • TestTROM (uses playback memory through the loopback fiber in order to test fixed/variable length transfers, overflow condition, link enables/disables). In general, simulates real operation of the board.
  • pcr reset PMC card which resets the TPC using the i960 bus reset
  • pcl clears intermediate store memory (puts zero).
  • pc_mask_test (tests integrity of setting link masks)
  • TestISMem1 (using playback memory, tries to write to many IS locations and reads it from PPC side, to ensure that writing over fiber to IS works).

    The full tests do not have to be run. For example, after loading the software, one can do the following:

    pcr
    pcl
    TestISMem1
    

    Note that the DCH ROMs (4 in ir2 and 1 in their teststand) have special firmware that gives them 4 large istore buffers instead of 8 small ones. I suspect leonid's TPCScreen software will not work on those ROMs, unless the TPC JP1 jumper is switched to give it the standard 8 buffers (see firmware section below for a few more details).


    Running Fast Control Standalone Test Software

    For this test the dataflow startup must be moved aside script so dataflow software is not loaded or run, and no ROM other than that in slot0 should be in the crate. Reboot the ROM if necessary. On the ROM type:

    cd "/dataflow/OdfHWTest/fctest"
    <fctest.cmd (to load files)
    
    Then execute the following commands for the appropriate boards:
    fcdmTest
    fcpmTest             (sysclk/n plugs into trig31)
    fcprTest
    fcgm_regT
    fctm_regT
    

    Running UPC Standalone Test Software

    For this test the dataflow startup must be moved aside script so dataflow software is not loaded or run. Reboot the ROM if necessary. For jjotest.c see the "diagnostic peek/poke code description" (as well as schematics, firmware and the like) look here.

    Load the code as follows:

    
    cd "/dataflow/OdfHWTest/upctest"
    ld<jjotest.o
    

    The following ROM scripts are useful:

    <tstest.cmd (test the trigger summer)
    <link0.cmd (test the first fiber)
    <link1.cmd (test the second fiber)
    <link2.cmd (test the third fiber)
    <istoretest.cmd (test the intermediate store)
    <luttest.cmd (test the lookup tables)
    

    Sample output from these tests can be found here.

    tstest.cmd is useful for checking basic trigger functionality. (If one wants the most serious test we have, Matt Weaver has software that runs in IR2 that compares every bit for every UPC (1) the EMC waveform (2) the trigger sums on the UPC (3) the trigger sums on the EMT). For each of the 72 crystals handled by a UPC, tstest.cmd fills in a "constant" lookup table value (meaning that the output of the LUT is independent of the data coming in the fiber). It then computes the expected trigger sum for the tower, and compares that with the observed trigger sum.

    For tstest.cmd, the important lines of output to look at are:

    Expected Trigger Sum = 0XCEDF
    Sample   WallClk   OpCode  TAG    TrigSum  Trig    TPhase    Cal  CalPhase
       1      0x028D    0x04    0x0E    0xCEDF    0    0x0F        0    0x00    
    
    (and similarly for the other fibers). If the observed trigger sum does not match the expected trigger sum, then the TrigSum that was found is preceded by a "<" character.

    I have noticed that the first time after a poweron of the UPC, tstest.cmd will often produce bad results. But subsequent attempts will succeed. I tried for a couple of hours, but I still don't understand the reason for this. Sometimes things look like junk, other times the trigger sums appear shifted by 4 bits. Perhaps it is because we need a "synch" to line things up?

    Running barreltest and endcaptest

    I believe these two pieces of code are pretty much identical, except one just loops over two fibers and the other loops over three. Ideally, they should be only one program, from a code-maintenance point of view. Like all standalone tests, one has to "unsetodf" and reboot before running (if dataflow was previously running in the ROM).
    cd "/dataflow/OdfHWTest/upctest"
    ld<barreltest.o
    barreltest(NITERATIONS)
    

    endcaptest works similarly.


    Sample Output from UPC Standalone Tests

    
    -> cd "/dataflow/OdfHWTest/upctest"
    value = 0 = 0x0
    -> ld < jjotest.o
    value = 33417024 = 0x1fde740
    -> <luttest.cmd
    upcmemi 98304,0xc5200000
    value = 98304 = 0x18000
    upcmemt 98304,0xc5200000
    value = 98304 = 0x18000
    upcmemi 98304,0xc5280000
    value = 98304 = 0x18000
    upcmemt 98304,0xc5280000
    value = 98304 = 0x18000
    upcmemi 98304,0xc5300000
    value = 98304 = 0x18000
    upcmemt 98304,0xc5300000
    value = 98304 = 0x18000
    -> <istoretest.cmd
    upcmemi 8192,0xc7000000
    value = 8192 = 0x2000
    upcmemt 8192,0xc7000000
    value = 8192 = 0x2000
    upcmemi 8192,0xc7020000
    value = 8192 = 0x2000
    upcmemt 8192,0xc7020000
    value = 8192 = 0x2000
    upcmemi 8192,0xc7040000
    value = 8192 = 0x2000
    upcmemt 8192,0xc7040000
    value = 8192 = 0x2000
    -> 
    -> <link2.cmd
    upcluten
    value = 0 = 0x0
    upcrst
    value = 0 = 0x0
    upctrigen
    value = 0 = 0x0
    upcflinken 2
    value = 2 = 0x2
    upccrystala 0
    Crystal =  0 Address =  0xC5200000 Data = 0x00000100 
    Crystal =  1 Address =  0xC5204000 Data = 0x00000101 
    Crystal =  2 Address =  0xC5208000 Data = 0x00000102 
    Crystal =  3 Address =  0xC520C000 Data = 0x00000103 
    Crystal =  4 Address =  0xC5210000 Data = 0x00000104 
    Crystal =  5 Address =  0xC5214000 Data = 0x00000105 
    Crystal =  6 Address =  0xC5218000 Data = 0x00000106 
    Crystal =  7 Address =  0xC521C000 Data = 0x00000107 
    Crystal =  8 Address =  0xC5220000 Data = 0x00000108 
    Crystal =  9 Address =  0xC5224000 Data = 0x00000109 
    Crystal = 10 Address =  0xC5228000 Data = 0x0000010A 
    Crystal = 11 Address =  0xC522C000 Data = 0x0001010B 
    Crystal = 12 Address =  0xC5230000 Data = 0x0001010C 
    Crystal = 13 Address =  0xC5234000 Data = 0x0001010D 
    Crystal = 14 Address =  0xC5238000 Data = 0x0001010E 
    Crystal = 15 Address =  0xC523C000 Data = 0x0001010F 
    Crystal = 16 Address =  0xC5240000 Data = 0x00010110 
    Crystal = 17 Address =  0xC5244000 Data = 0x00010111 
    Crystal = 18 Address =  0xC5248000 Data = 0x00010112 
    Crystal = 19 Address =  0xC524C000 Data = 0x00010113 
    Crystal = 20 Address =  0xC5250000 Data = 0x00010114 
    Crystal = 21 Address =  0xC5254000 Data = 0x00010115 
    Crystal = 22 Address =  0xC5258000 Data = 0x00010116 
    Crystal = 23 Address =  0xC525C000 Data = 0x00010117 
    Crystal =  0 Address =  0xC5280000 Data = 0x00000200 
    Crystal =  1 Address =  0xC5284000 Data = 0x00000201 
    Crystal =  2 Address =  0xC5288000 Data = 0x00000202 
    Crystal =  3 Address =  0xC528C000 Data = 0x00000203 
    Crystal =  4 Address =  0xC5290000 Data = 0x00000204 
    Crystal =  5 Address =  0xC5294000 Data = 0x00000205 
    Crystal =  6 Address =  0xC5298000 Data = 0x00000206 
    Crystal =  7 Address =  0xC529C000 Data = 0x00000207 
    Crystal =  8 Address =  0xC52A0000 Data = 0x00000208 
    Crystal =  9 Address =  0xC52A4000 Data = 0x00000209 
    Crystal = 10 Address =  0xC52A8000 Data = 0x0000020A 
    Crystal = 11 Address =  0xC52AC000 Data = 0x0001020B 
    Crystal = 12 Address =  0xC52B0000 Data = 0x0001020C 
    Crystal = 13 Address =  0xC52B4000 Data = 0x0001020D 
    Crystal = 14 Address =  0xC52B8000 Data = 0x0001020E 
    Crystal = 15 Address =  0xC52BC000 Data = 0x0001020F 
    Crystal = 16 Address =  0xC52C0000 Data = 0x00010210 
    Crystal = 17 Address =  0xC52C4000 Data = 0x00010211 
    Crystal = 18 Address =  0xC52C8000 Data = 0x00010212 
    Crystal = 19 Address =  0xC52CC000 Data = 0x00010213 
    Crystal = 20 Address =  0xC52D0000 Data = 0x00010214 
    Crystal = 21 Address =  0xC52D4000 Data = 0x00010215 
    Crystal = 22 Address =  0xC52D8000 Data = 0x00010216 
    Crystal = 23 Address =  0xC52DC000 Data = 0x00010217 
    Crystal =  0 Address =  0xC5300000 Data = 0x00000300 
    Crystal =  1 Address =  0xC5304000 Data = 0x00000301 
    Crystal =  2 Address =  0xC5308000 Data = 0x00000302 
    Crystal =  3 Address =  0xC530C000 Data = 0x00000303 
    Crystal =  4 Address =  0xC5310000 Data = 0x00000304 
    Crystal =  5 Address =  0xC5314000 Data = 0x00000305 
    Crystal =  6 Address =  0xC5318000 Data = 0x00000306 
    Crystal =  7 Address =  0xC531C000 Data = 0x00000307 
    Crystal =  8 Address =  0xC5320000 Data = 0x00000308 
    Crystal =  9 Address =  0xC5324000 Data = 0x00000309 
    Crystal = 10 Address =  0xC5328000 Data = 0x0000030A 
    Crystal = 11 Address =  0xC532C000 Data = 0x0001030B 
    Crystal = 12 Address =  0xC5330000 Data = 0x0001030C 
    Crystal = 13 Address =  0xC5334000 Data = 0x0001030D 
    Crystal = 14 Address =  0xC5338000 Data = 0x0001030E 
    Crystal = 15 Address =  0xC533C000 Data = 0x0001030F 
    Crystal = 16 Address =  0xC5340000 Data = 0x00010310 
    Crystal = 17 Address =  0xC5344000 Data = 0x00010311 
    Crystal = 18 Address =  0xC5348000 Data = 0x00010312 
    Crystal = 19 Address =  0xC534C000 Data = 0x00010313 
    Crystal = 20 Address =  0xC5350000 Data = 0x00010314 
    Crystal = 21 Address =  0xC5354000 Data = 0x00010315 
    Crystal = 22 Address =  0xC5358000 Data = 0x00010316 
    Crystal = 23 Address =  0xC535C000 Data = 0x00010317 
    
    Sum Fiber 0 should be   0x00000A9D  Offset = 0x00008040
    
    Sum Fiber 1 should be   0x0000179D  Offset = 0x00008040
    
    Sum Fiber 2 should be   0x0000249D  Offset = 0x00008040
    value = 57 = 0x39 = '9'
    upcrun
    value = 0 = 0x0
    upcwistore 0,0x11111111
    Starting Address =  0xC7000000
    value = 31 = 0x1f
    upcwistore 1,0x22222222
    Starting Address =  0xC7020000
    value = 31 = 0x1f
    upcwistore 2,0x33333333
    Starting Address =  0xC7040000
    value = 31 = 0x1f
    upctrig
    value = 0 = 0x0
    upcristore 2,10
    Starting Address =  0xC7040000
    Data =  0x00030403
    Data =  0x0803002D
    Data =  0x0C031003
    Data =  0x140307B9
    Data =  0x01030503
    Data =  0x09030004
    Data =  0x0D031103
    Data =  0x15030742
    Data =  0x02030603
    Data =  0x0A0300C0
    value = 19 = 0x13
    upcimap  2,4
    
    Package:   1
                   0x2D00   0x0308     0x0304   0x0300
                   0xB907   0x0314     0x0310   0x030C
                   0x0400   0x0309     0x0305   0x0301
                   0x4207   0x0315     0x0311   0x030D
                   0xC000   0x030A     0x0306   0x0302
                   0xC007   0x0316     0x0312   0x030E
                   0xC001   0x030B     0x0307   0x0303
    END 0xC704003C 0x8087   0x0317     0x0313   0x030F
    
    Package:   2
                   0xAD00   0x0308     0x0304   0x0300
                   0xB907   0x0314     0x0310   0x030C
                   0x0400   0x0309     0x0305   0x0301
                   0x4207   0x0315     0x0311   0x030D
                   0xC000   0x030A     0x0306   0x0302
                   0xC007   0x0316     0x0312   0x030E
                   0xC001   0x030B     0x0307   0x0303
    END 0xC704007C 0x8087   0x0317     0x0313   0x030F
    
    Package:   3
                   0x2D00   0x0308     0x0304   0x0300
                   0xF907   0x0314     0x0310   0x030C
                   0x0400   0x0309     0x0305   0x0301
                   0x4207   0x0315     0x0311   0x030D
                   0xC000   0x030A     0x0306   0x0302
                   0xC007   0x0316     0x0312   0x030E
                   0xC001   0x030B     0x0307   0x0303
    END 0xC70400BC 0x8087   0x0317     0x0313   0x030F
    
    Package:   4
                   0xAD00   0x0308     0x0304   0x0300
                   0xF907   0x0314     0x0310   0x030C
                   0x0400   0x0309     0x0305   0x0301
                   0x4207   0x0315     0x0311   0x030D
                   0xC000   0x030A     0x0306   0x0302
                   0xC007   0x0316     0x0312   0x030E
                   0xC001   0x030B     0x0307   0x0303
    END 0xC70400FC 0x8087   0x0317     0x0313   0x030F
    value = 37 = 0x25 = '%'
    upcrwallclk 2,25
    Starting Address =  0xC7040004
    
    
    Expected Trigger Sum = 0X249D
    Sample   WallClk   OpCode  TAG    TrigSum  Trig    TPhase    Cal  CalPhase
       1      0x03A4    0x07    0x00    0x249D    0    0x0F        0    0x00    
       2      0x03A5    0x07    0x00    0x249D    0    0x0F        0    0x00    
       3      0x03A6    0x07    0x00    0x249D    0    0x0F        0    0x00    
       4      0x03A7    0x07    0x00    0x249D    0    0x0F        0    0x00    
       5      0x03A8    0x07    0x00    0x249D    0    0x0F        0    0x00    
       6      0x03A9    0x07    0x00    0x249D    0    0x0F        0    0x00    
       7      0x03AA    0x07    0x00    0x249D    0    0x0F        0    0x00    
       8      0x03AB    0x07    0x00    0x249D    0    0x0F        0    0x00    
       9      0x03AC    0x07    0x00    0x249D    0    0x0F        0    0x00    
      10      0x03AD    0x07    0x00    0x249D    0    0x0F        0    0x00    
      11      0x03AE    0x07    0x00    0x249D    0    0x0F        0    0x00    
      12      0x03AF    0x07    0x00    0x249D    0    0x0F        0    0x00    
      13      0x03B0    0x07    0x00    0x249D    0    0x0F        0    0x00    
      14      0x03B1    0x07    0x00    0x249D    0    0x0F        0    0x00    
      15      0x03B2    0x07    0x00    0x249D    0    0x0F        0    0x00    
      16      0x03B3    0x07    0x00    0x249D    0    0x0F        0    0x00    
      17      0x03B4    0x07    0x00    0x249D    0    0x0F        0    0x00    
      18      0x03B5    0x07    0x00    0x249D    0    0x0F        0    0x00    
      19      0x03B6    0x07    0x00    0x249D    0    0x0F        0    0x00    
      20      0x03B7    0x07    0x00    0x249D    0    0x0F        0    0x00    
      21      0x03B8    0x07    0x00    0x249D    0    0x0F        0    0x00    
      22      0x03B9    0x07    0x00    0x249D    0    0x0F        0    0x00    
      23      0x03BA    0x07    0x00    0x249D    0    0x0F        0    0x00    
      24      0x03BB    0x07    0x00    0x249D    0    0x0F        0    0x00    
      25      0x03BC    0x07    0x00    0x249D    0    0x0F        0    0x00    
    value = 1 = 0x1
    -> 
    -> <tstest.cmd
    #comments by cpo 10/1/03
    
    #empirically, this appears to be necessary, since sometimes the UPC
    #powers up with the "hp reset" line (glink reset, I think) high.
    upchprst
    value = 0 = 0x0
    taskDelay(30)
    value = 0 = 0x0
    upcrst
    value = 0 = 0x0
    taskDelay(2)
    value = 0 = 0x0
    
    #start putting the ctrl register into a known state
    upcrun
    value = 0 = 0x0
    
    #current defaults for ir2 data taking
    #since we only look at 1 sample, probably not necessary
    upcwsamples(48)
    value = 48 = 0x30 = '0'
    upcwdepth(44)
    value = 44 = 0x2c = ','
    
    #enable all the fibers
    upcflinken(0)
    value = 0 = 0x0
    upcflinken(1)
    value = 1 = 0x1
    upcflinken(2)
    value = 2 = 0x2
    
    #disable writing to the trigger pattern memory
    #which also enables writing to the offset register
    #(I believe the latter is needed for the upcwoffset lines below)
    upcwtrigpatdis(0)
    value = 0 = 0x0
    upcwtrigpatdis(1)
    value = 1 = 0x1
    upcwtrigpatdis(2)
    value = 2 = 0x2
    
    #disable sending the trigger pattern out of the trigger summers, which
    #also enables sending the real trigger sum out of the trigger
    #summers (I'm not sure if this is necessary).
    upctrigpatdis(0)
    value = 0 = 0x0
    upctrigpatdis(1)
    value = 1 = 0x1
    upctrigpatdis(2)
    value = 2 = 0x2
    
    #disables the "divide by 2" operation.  tstest gets things wrong
    #by a factor of 2 if this isn't in place.
    upcwoffset 0,0x8040
    value = 0 = 0x0
    upcwoffset 1,0x8040
    value = 1 = 0x1
    upcwoffset 2,0x8040
    value = 2 = 0x2
    
    #program the lut with a random number for each
    #of the 72 crystals, but the number is constant
    #for each crystal (i.e. independent of incoming fiber data).
    upcluten
    value = 0 = 0x0
    upclutrandom 0xfff
    Crystal =  0 Address =  0xC5200000 Data = 0x00000054 
    Crystal =  1 Address =  0xC5204000 Data = 0x00010833 
    Crystal =  2 Address =  0xC5208000 Data = 0x00010CD7 
    Crystal =  3 Address =  0xC520C000 Data = 0x00010507 
    Crystal =  4 Address =  0xC5210000 Data = 0x00010BFA 
    Crystal =  5 Address =  0xC5214000 Data = 0x000100D6 
    Crystal =  6 Address =  0xC5218000 Data = 0x00010513 
    Crystal =  7 Address =  0xC521C000 Data = 0x000108F9 
    Crystal =  8 Address =  0xC5220000 Data = 0x0001078C 
    Crystal =  9 Address =  0xC5224000 Data = 0x00010868 
    Crystal = 10 Address =  0xC5228000 Data = 0x00010EFA 
    Crystal = 11 Address =  0xC522C000 Data = 0x00010244 
    Crystal = 12 Address =  0xC5230000 Data = 0x00010460 
    Crystal = 13 Address =  0xC5234000 Data = 0x000109B9 
    Crystal = 14 Address =  0xC5238000 Data = 0x00010F85 
    Crystal = 15 Address =  0xC523C000 Data = 0x00010A02 
    Crystal = 16 Address =  0xC5240000 Data = 0x00010D14 
    Crystal = 17 Address =  0xC5244000 Data = 0x0001076D 
    Crystal = 18 Address =  0xC5248000 Data = 0x00010960 
    Crystal = 19 Address =  0xC524C000 Data = 0x000101CC 
    Crystal = 20 Address =  0xC5250000 Data = 0x00010BB8 
    Crystal = 21 Address =  0xC5254000 Data = 0x0001037B 
    Crystal = 22 Address =  0xC5258000 Data = 0x0001087B 
    Crystal = 23 Address =  0xC525C000 Data = 0x00010C25 
    Crystal =  0 Address =  0xC5280000 Data = 0x000109D3 
    Crystal =  1 Address =  0xC5284000 Data = 0x00010474 
    Crystal =  2 Address =  0xC5288000 Data = 0x0001064A 
    Crystal =  3 Address =  0xC528C000 Data = 0x00010A73 
    Crystal =  4 Address =  0xC5290000 Data = 0x00010385 
    Crystal =  5 Address =  0xC5294000 Data = 0x00010BC6 
    Crystal =  6 Address =  0xC5298000 Data = 0x00000037 
    Crystal =  7 Address =  0xC529C000 Data = 0x000103D9 
    Crystal =  8 Address =  0xC52A0000 Data = 0x000103FA 
    Crystal =  9 Address =  0xC52A4000 Data = 0x00010D0F 
    Crystal = 10 Address =  0xC52A8000 Data = 0x000108E0 
    Crystal = 11 Address =  0xC52AC000 Data = 0x00010FF4 
    Crystal = 12 Address =  0xC52B0000 Data = 0x00010DE5 
    Crystal = 13 Address =  0xC52B4000 Data = 0x00010DF4 
    Crystal = 14 Address =  0xC52B8000 Data = 0x000108EE 
    Crystal = 15 Address =  0xC52BC000 Data = 0x00010571 
    Crystal = 16 Address =  0xC52C0000 Data = 0x0001065C 
    Crystal = 17 Address =  0xC52C4000 Data = 0x000107E8 
    Crystal = 18 Address =  0xC52C8000 Data = 0x000107B5 
    Crystal = 19 Address =  0xC52CC000 Data = 0x00010ABC 
    Crystal = 20 Address =  0xC52D0000 Data = 0x000101A1 
    Crystal = 21 Address =  0xC52D4000 Data = 0x0001073B 
    Crystal = 22 Address =  0xC52D8000 Data = 0x000104BF 
    Crystal = 23 Address =  0xC52DC000 Data = 0x00010EB6 
    Crystal =  0 Address =  0xC5300000 Data = 0x00010EA8 
    Crystal =  1 Address =  0xC5304000 Data = 0x00010E1F 
    Crystal =  2 Address =  0xC5308000 Data = 0x00000082 
    Crystal =  3 Address =  0xC530C000 Data = 0x00010A60 
    Crystal =  4 Address =  0xC5310000 Data = 0x0001019A 
    Crystal =  5 Address =  0xC5314000 Data = 0x000108FD 
    Crystal =  6 Address =  0xC5318000 Data = 0x00010685 
    Crystal =  7 Address =  0xC531C000 Data = 0x00010B6D 
    Crystal =  8 Address =  0xC5320000 Data = 0x00010D71 
    Crystal =  9 Address =  0xC5324000 Data = 0x00010CCF 
    Crystal = 10 Address =  0xC5328000 Data = 0x000105E0 
    Crystal = 11 Address =  0xC532C000 Data = 0x000100F6 
    Crystal = 12 Address =  0xC5330000 Data = 0x00010896 
    Crystal = 13 Address =  0xC5334000 Data = 0x00010618 
    Crystal = 14 Address =  0xC5338000 Data = 0x000104D0 
    Crystal = 15 Address =  0xC533C000 Data = 0x00010C90 
    Crystal = 16 Address =  0xC5340000 Data = 0x00010327 
    Crystal = 17 Address =  0xC5344000 Data = 0x00010DB0 
    Crystal = 18 Address =  0xC5348000 Data = 0x00010C84 
    Crystal = 19 Address =  0xC534C000 Data = 0x0001010C 
    Crystal = 20 Address =  0xC5350000 Data = 0x00010BA4 
    Crystal = 21 Address =  0xC5354000 Data = 0x00010572 
    Crystal = 22 Address =  0xC5358000 Data = 0x0001067E 
    Crystal = 23 Address =  0xC535C000 Data = 0x00010200 
    
    Sum Fiber 0 should be   0x0000BB7F  Offset = 0x00000040
    
    Sum Fiber 1 should be   0x0000BD83  Offset = 0x00000040
    
    Sum Fiber 2 should be   0x0000B70F  Offset = 0x00000040
    value = 57 = 0x39 = '9'
    
    #put us back in RUN mode (i.e. not LUT mode), reset the UPC istore
    # pointers, and enable software triggers
    upcrun
    value = 0 = 0x0
    upctrigen
    value = 0 = 0x0
    
    #generate a trigger
    upctrig
    value = 0 = 0x0
    
    #look at the results
    upcrwallclk 0,1
    Starting Address =  0xC7000004
    
    
    Expected Trigger Sum = 0XBB7F
    Sample   WallClk   OpCode  TAG    TrigSum  Trig    TPhase    Cal  CalPhase
       1      0x010B    0x07    0x00    0xBB7F    0    0x0F        0    0x00    
    value = 1 = 0x1
    upcimap 0,1
    
    Package:   1
                   0xAF03   0x078C     0x0BFA   0x0054
                   0x7707   0x0BB8     0x0D14   0x0460
                   0x4B07   0x0868     0x00D6   0x0833
                   0x0B07   0x037B     0x076D   0x09B9
                   0x8007   0x0EFA     0x0513   0x0CD7
                   0x8007   0x087B     0x0960   0x0F85
                   0xC007   0x0244     0x08F9   0x0507
    END 0xC700003C 0x8087   0x0C25     0x01CC   0x0A02
    value = 37 = 0x25 = '%'
    upcrwallclk 1,1
    Starting Address =  0xC7020004
    
    
    Expected Trigger Sum = 0XBD83
    Sample   WallClk   OpCode  TAG    TrigSum  Trig    TPhase    Cal  CalPhase
       1      0x010B    0x07    0x00    0xBD83    0    0x0F        0    0x00    
    value = 1 = 0x1
    upcimap 1,1
    
    Package:   1
                   0xA307   0x03FA     0x0385   0x09D3
                   0x7807   0x01A1     0x065C   0x0DE5
                   0x4D07   0x0D0F     0x0BC6   0x0474
                   0x0B07   0x073B     0x07E8   0x0DF4
                   0x8005   0x08E0     0x0037   0x064A
                   0x8007   0x04BF     0x07B5   0x08EE
                   0xC007   0x0FF4     0x03D9   0x0A73
    END 0xC702003C 0x8087   0x0EB6     0x0ABC   0x0571
    value = 37 = 0x25 = '%'
    upcrwallclk 2,1
    Starting Address =  0xC7040004
    
    
    Expected Trigger Sum = 0XB70F
    Sample   WallClk   OpCode  TAG    TrigSum  Trig    TPhase    Cal  CalPhase
       1      0x010B    0x07    0x00    0xB70F    0    0x0F        0    0x00    
    value = 1 = 0x1
    upcimap 2,1
    
    Package:   1
                   0xAF07   0x0D71     0x019A   0x0EA8
                   0x7007   0x0BA4     0x0327   0x0896
                   0x4707   0x0CCF     0x08FD   0x0E1F
                   0x0B07   0x0572     0x0DB0   0x0618
                   0x8003   0x05E0     0x0685   0x0082
                   0x8007   0x067E     0x0C84   0x04D0
                   0xC007   0x00F6     0x0B6D   0x0A60
    END 0xC704003C 0x8087   0x0200     0x010C   0x0C90
    value = 37 = 0x25 = '%'
    

    Putting the UPC in "Trigger Test Pattern" Mode

    This is useful for testing the output trigger-pattern MACH (U33) and the transmitter chip (U40). This pattern alternates between the "serial number" set on header HD1 and a walking bit pattern. The EMT group uses this to do their "serial number calibration". Do this as follows:

  • enable the CC CLINK. this can be done in a kludgy way by typing *0xc6000004=0x22000060 at the ROM prompt (enables TU and CLINK, and sets number of MUQ and TUQ bufs to 2).
  • load Jeff's standalone code as described above
  • type upctrigtest at the ROM prompt
  • send a startplayback opcode (0x6). this can be done in a kludgy way by typing *0xc6000014=0x19000000 at the ROM prompt (see controller card documentation ... this injects opcode 0x6 on the CLINK after the PPC->i960 byte swapping)

    Running the VME DMA Tester

    This lives in CVS repository in package OdfApps/mtest.

    For this test the dataflow startup script must be moved aside so dataflow software is not loaded or run. Reboot the ROM if necessary.

    Put two (and only two) CPUs in the crate, in slots 0 and 1.

    There is a complication if you run a standalone SBC as a slotN (i.e. no TPC/CC, where it can't read the backplane jumpers to get the slot number). First, it is necessary to do the following:

  • type sysProcNumSet(1) in the slotN SBC.
  • reboot the slot0 Then, on both ROMs type:
    
    </dataflow/tgt/sbcTest
    
    Then (the order is important):
    
    sbcTest  (in the slot0 ROM)
    sbcTest  (in the slotN ROM)
    

    The command sbcHelp also prints some useful information. By default this command does about 21 million DMAs, which takes about 20 minutes.

    A second complication: it seems to be unfortunately necessary to run the VxWorks kernel entirely out of flash. sbcTest fails when the kernel is loaded using the rshell mechanism. You can identify this mode if it gives you the chance to interrupt booting with the statement "Press any key to stop auto-boot...". Execute:

    
    </dataflow/tgt/startup.save-kernel-to-flash
    
    to load the VxWorks kernel into of flash, then put SBC jumper J15 in the "1-2" position and power cycle the board (hard reset (reset button) might also work, not sure; soft reset (^X) doesn't).

    The VME test will fail for Universe1 chips. The universe1 part number is C91C042 while the universe2 part number is C91C142.


    Running the I960 DMA Tester

    To run the i960 DMA tester, execute the shell command set960. This sets up a special startup script that loads the dma testing code. Then reboot the ROM and run fcgui/eventtest exactly as in the dataflow ROM tests below. The "events" are generated by the FCPM front-panel triggers, as in the usual TPC/UPC test.

    Some notes: the special code that is loaded has source in CVS in OdfApps/mtest/i960dmatest.cc (SBC code) and OdfI960/i960/dmaTest.c (i960 code for TPCs) and OdfI960/i960/dmaTestUpc.c (i960 code for UPCs). The i960 image has to be relinked with these special files to get the right behaviour.

    During the test the i960 puts an incrementing counter into the intermediate store for 256 longwords (TPC) and "64 samples at 16 longwords per sample" for the UPC version. The SBC code checks for this incrementing pattern. For the TPC it tests all 32 links and all buffers in the istore. For the UPC it tests all 3 fibers.

    I have tested this with the TPC recently, but not with the UPC (recently). Hopefully it still works.


    Running Miscellaneous Utilites

    To dump out the universe chip configuration execute the following in the rom (this code is in CVS under OdfVxWorks/SLAC):

    
    ld </dataflow/tgt/univShow
    univShow
    

    To dump out the CC Queue memory, personality card "transfer result" and intermediate store data, execute the following in the rom (this code is in CVS under OdfApps/mtest):

    
    ld </dataflow/tgt/ccq
    ccq
    ccqdetail(ccq address from ccq output)
    istore(ccq address,fiber,element,nwords)
    

    ccqdetail dumps out the so-called "transfer result" information from the personality card.

    istore dumps out the contents of the istore for a particular event in the ccq. note that since there are typically fewer istore buffers than there are ccq buffers, not all events in the ccq are guaranteed to be in the istore. If "nwords" above is zero, istore will dump out the total number of words in the event. Currently ccqdetail and istore are only implemented for the TPC, but it is straightforward to implement them for the UPC, I believe.

    To reboot all ROMs in a crate from one of the ROM prompts:

    
    odfVmeReset
    

    Running TPC Core Dataflow Software

    This lives in CVS in the package Odf. Commands to execute on bbr-dev15 to get dataflow environment: With that setup the following commands can be used:
    
    fcgui -c <crateMask> [-t <triggerMask>] [-g <environment_filename>] (starts up control level)
    eventtest -c <crateMask> -i <eventBuilderID> [-d,-e] (starts up event level)
    
  • Command line options in "[]" are optional.
  • All numbers can be specified with the usual C style prefixes (e.g. "0x" for hex).
  • crateMask must be set to the number specified in the FCDM dip switches, which is printed out when the ROM starts dataflow software.
  • triggerMask is useful if a front-panel lemo cable is used from SYSCLK/N output into one of the FCPM front-panel trigger inputs. At the time of this writing TRIG28 will generate a ~1Hz trigger rate, while TRIG29 through TRIG31 will generate a ~4kHz trigger rate.
  • eventBuilderId is a number between 0 and 63 (at the time of this writing)
  • "-d" puts eventtest in "display" mode, where it will dump out damage statistics for crate mask X every Y events. X is the "env" parameter specified in the fcgui BeginMinor transition. Y is the "env" parameter specified in the fcgui BeginMajor transition.
  • "-e" puts eventtest in "event-dump" mode, where it will dump out some detail of every transition that it sees.
  • if neither -d nor -e flag is used with eventtest, it defaults to printing one line for every transition and prints out a message every 1000 L1Accepts.

    For TPC link A testing, we run the ifrTest application (in CVS in OdfApps/mtest):

  • set the FCDM detector ID dipswitches to 5, and the crate dipswitches to 0 (i.e. crate mask 0x1).
  • plug the IFR front-end electronics into fiber A of the TPC you want to test (this can be either a single ROM, or the rightmost ROM of a 2 ROM crate). plug a loopback fiber into B.
  • put a lemo loopback cable from SYSCLK/N into TRIG29 of the FCPM
  • on bbr-dev15: fcg (alias for fcgui -c 0x1 -g /dataflow/teststand/teststand.arg -l /dataflow/teststand/teststand.cal  ). "-g" reads in the list of "arguments" that will be sent with the transitions. In particular the beginmajor env "n" tells "eventtest -d" to dump out info every "n" events. And beginminor env "m" tells "eventtest -d" to dump out the info for crate mask "m". "-l" reads in the list of "calibration" settings that can be used to generate 5 2.7us spaced L1Accepts (at the time of this writing).
  • on bbr-dev15: evt (alias for eventtest -c 0x1 -i 0 -d) &
  • Allocate, then go to the enabled state.
  • Disable will cause eventtest and the ROM to dump out damage statistics. Run for 1,000,000 events and check for no damage. All statistics are reset on the BeginMacro transition.
  • One can obtain higher rates by setting the global variable l1sink=1 in the ROM. The ROM will still dump out damage statistics, but the event level will not. Set l1sink=0 to resume sending data to the event level.

    For TPC link B testing, as above, but:

  • plug the electronics into fiber B, and a loopback fiber into A
  • in the rom, type linksa=0 and linksb=0x1f
  • make sure to re-do the configure transition transition

    If the ROM hangs, some information about the state of the slot0 and slotN ROMs can be dumped by executing the following in the ROM:

    
    odfDump 0,-1
    

    Interpreting this information requires fairly detailed knowledge about how the software and hardware works, unfortunately.

    To generate 5 2.7us spaced L1Accepts (a "recommended test"):

  • set the trigger mask in fcgui "Arguments" window to 0 (puts FCPM sequencer in control of trigger generation)
  • redo the configure transition
  • when you enable, 5 L1Accepts will be sent down (this is taken care of by the "-l" flag to fcgui described above.
  • check for no damage for those triggers
  • set the trigger mask in fcgui "Arguments" window to 0xffffffff and redo the configure transition to put the system back in standard trigger mode described above (controlled by lemo cables plugged into the FCPM)

    Running UPC Core Dataflow Software

    Test the UPCs as described above for the TPCs, but:

  • set the FCDM detector ID dipswitches to 4
  • plug the TRB into fiber A of a slot0 TPC ROM, and a loopback into fiber B.
  • plug the 2 (endcap UPC) or 3 (barrel UPC) IOB fibers into the UPC to be tested

    Dataflow source code for "emcUpcTest" can be found in CVS in OdfApps/mtest/.

    The vxworks executable "emcUpcTest" checks whether the UPC receives a particular test pattern from the EMC ADBs, and otherwise it keeps stats on errors per fiber (up to three), per EMC sample (up to 48), per Crystal (24 being read out as one sample), and per Bit of the 16 bit values of which bit 0 ... 9 are ADC values, 10 and 11 are range bits. Bits 12 ... 15 aren't used by the ADBs, but test the UPC Look-up table.

    The expected test pattern alternates Crystal values "0x688" and "0xFAC".

    The fiber configuration can be set as bit pattern in ROM variable "links"; "links = 0x7" (the default) means all three fibers are to be analyzed. Set "links = 0x3" to test an endcap UPC.

    Other ROM variables "samples" and "presamples" with defaults 48 and 44, respectively, determine the number of samples which are to be read for each Level1Accept trigger event. New values for these three ROM Variables take effect only after redoing "Configure".

    To sink L1Accepts in the ROM (allows running at higher rates) set ROM variable "l1sink" to a non-zero value. This can be done at any time, without re-configuring.

    Error statistics are printed upon "Disable" and reset on "BeginMeta". To inspect the ADC values of the crystals in the first sample in which an error occured, i.e. a deviation from the expected test pattern (if any), set "stoponerr = 1". Subsequently type "cont" to continue.

    Note that this software can be run on the entire EMC ir-2 system in order to look for data corruption there.

    Right now there is a problem that I don't understand. When I first run the software after powering up the UPC, I get FLINK TIMEOUT damage on every fiber on every event. Doing a hard reboot of the system makes the problem go away. I need to work on this.


    Damage Bit Definitions

        enum Value {
               PCI_ParityError   = 0,  DroppedContribution    = 1,
               PCI_TargetAbort   = 2,  PCI_MasterAbort        = 3,
               Shutdown          = 4,  I960_BusFault          = 5,
               I960_MemoryFault  = 6,  PayloadTruncated       = 7,
               Timeout           = 8,  MUQ_NAQ                = 9,
               InvalidTransition = 10, TransitionTimeout      = 11,
               OutOfOrder        = 12, OutofSynch             = 13,
               FEX_Error         = 14, IncompleteContribution = 15,
               CLINK_NotReady    = 16, PCFULL                 = 17,
               FEE_FULL          = 18, CLINKB_NotReady        = 19,
               CLINKA_NotReady   = 20, WriteInProgress        = 21,
               CLINK_NotEnabled  = 22, CLINKA_Throttled       = 23,
               DLINKA_NotReady   = 24, DLINKA_TimeOut         = 25,
               DLINKA_Overflow   = 26, DLINKA_NoStartbit      = 27,
               DLINKB_NotReady   = 28, DLINKB_TimeOut         = 29,
               DLINKB_Overflow   = 30, DLINKB_NoStartbit      = 31};
        enum UpcValue {
               FLINKA_TimeOut    = 24, FLINKB_TimeOut         = 25,
               FLINKC_TimeOut    = 26, FLINKA_NotReady        = 27,
               FLINKB_NotReady   = 28, FLINKC_NotReady        = 29};
    

    Running PPCBug Diagnostics

    On some ROMs, the second bank of flash memory contains the Motorola PPCBug operating system. Not all, unfortunately. Dataflow was not consistent in the way we programmed the flash. Easiest way to give a ROM PPCBug would probably be to move the socketed flash over from a board that has it. We have set aside PROMs in the "ROM repair box of parts" that have PPCBug installed that can be used.

    NOTE: PPCBug will not boot if the PMC card is plugged in. Also NOTE that you should run a newer PPCBug version (4.1) which includes additional tests for the Universe chip, I believe.

    There is a useful SBC test that can be run if a problem is suspected with an SBC. At the PPCBug prompt, type:

    
    sd (switches to from debug mode to diagnostic mode)
    st (runs PPCBug self test)
    

    Setting ROM Boot Parameter (and What To Do if the ROM nfsMount Hangs)

    Sometimes when you move ROMs between subnets, the nfsMount of srv02 hangs when the kernel is starting up (because the IP address of the ROM or srv02 is wrong).

    Ideally the nfsMount shouldn't hang. I tried lowering the nfsTimeoutSec value from 37267 (absurdly high) to 60 in the vxworks kernel, but that didn't seem to help. (I'll put it in the "priorities" file so we don't forget about it).

    But until we fix it: a workaround is unplugging the FCDM. This causes the ROM to do not try to do the nfsMount. You get the prompt right away and can do "bootChange" in the usual way.

    Remember that the way to look at the ROM boot parameters is to use the vxworks commands "bootParamsShow sysBootLine" and "bootChange".


    Installing Kernel into MVME2306 Flash Memory using PPCBug

    Each kernel that is programmed into a new board needs to have a runtime VxWorks license. The SLAC VxWorks web page has intructions for how to do this.

    Note some gotchas that can create problems in the following process:

  • the I960 PMC card should be removed.
  • rsh must be enabled for the correct subnet in /etc/hosts.allow.
  • the IP address of the SBC must be in your .rhosts file, and the name must be "fully qualified" when it's not in /etc/hosts, e.g. rom080.slac.stanford.edu, not just rom080. The vxworks .rhosts is a taylored file, so any changes you make to it will get erased when the next taylor is run. (Matt, later) I also found that the name must be unqualified if it IS found in /etc/hosts, i.e. just rom080.
  • tftp and rsh have to be enabled on the machine that is serving boot.bin. Edit /etc/inetd.conf in order to enable these, and kill -HUP {inetd processid} (to restart with tftp enabled). You need sudo privilege to do this. For example, if rsh is not enabled, you will get the following from the ROM:
    Attaching network interface dc0... done.
    Attaching network interface lo0... done.
    Loading... 
    Error loading file: errno = 0x3d.
    
  • The rsh will try to execute the .cshrc for the selected account name. If that .cshrc tries to use an afs file the rsh will hang, since the rsh doesn't have an afs token (I think).
  • The .rhosts file must have UNIX permissions set to -rw------- or it won't work.
  • If the file you are trying to load with rsh has UNIX ACLs, there are some complexities since vxWorks doesn't understand ACLs. These are described at some level in package OdfApps in the file /acl/aclinit.
  • Even after fixing all of the above, the last time I tried this I couldn't get the rsh using user vxworks to work (gave Loading... permission denied Error loading file: errno = 0x250002). However, it did work with user cpo. I don't understand why. Later: I believe it is because the vxworks account no longer exists (cpo apr 4, 2006). From now on, I guess we have to use individual user accounts.
  • Recently, I have had serious problems with TFTP (network snoop on the failure mode shown below). Tried many things (doing nioc from ppcbug, kill -HUP inetd, half/full-duplex connection, killing arp entries with arp -d). It has worked the last few times if I either wait a long time or switch ip addresses regularly and not just use rom080 (for the TFTP file load only). If you do this though, remember to powercycle the SBC before booting vxworks as rom080 so the switch can learn about the new ethernet address.
    from a ROM vxworks prompt:
    -> printErrno(0x3d)
    S_errno_ECONNREFUSED
    value = 21 = 0x15
    -> printErrno(0x250002)
    S_remLib_RSH_ERROR
    value = 19 = 0x13
    -> 
    
    

    The MVE2306 provides two different types of Flash memory. Flash which is permanently installed on the board (soldered Flash) and Flash which is removable (socketed Flash). Motorola refers the soldered Flash as "Bank A" and the socketed Flash as "Bank B". At any single time ONE of these Flash memories is mapped onto a common PPC address to be used to bootstrap the CPU. Which flash memory is mapped to this common address is selected by a jumper on the board (J15). If the soldered flash (Bank A) is to be mapped, the jumper is set to the position 1-2. If the the socketed flash (Bank B) is to be mapped, the jumper is set to the position 2-3.

    Both the socketed and soldered Flash are distributed by Motorola with a resident monitor called "PPCBug". That is, independent of the jumper settings the board will boot into PPCBug. The function of this installation is two-fold. One: is to install the APPROPRIATE version of PPCBug into the socketed flash, and two: is to install the bootstrap code necessary to boot-up VxWorks into the soldered Flash. Once this is accomplished the jumper is set by default to boot out of the soldered flash (1-2) and thus boot VxWorks. If it is necessary to use PPCBug, the jumper is simply set to the 2-3 position and the board reset.

    The Procedure

  • Check that the jumper setting is in the 1-2 position (factory default)
  • Power the module. It should print out some messages about the configuration of the board and boot into PPCBug and leave you at the prompt (PPC1-Bug>).
  • Initialize the Real-Time-Clock (RTC). At the prompt use the "SET" command along with a data-time string of the form:
    
       PPC1-Bug>SET MMDDYYHHmm
      
       Where: MM is the numeric month
              DD is the numeric day
              YY is the numeric year
              HH is the numeric hour
              mm is the numeric minute
       For example, to set the clock to April 8th, 1998 at 1:30 in the afternoon:
    
       PPC1-Bug>SET 0408981330
    
    Note: If you fail to set the clock the NIOP operations described below will FAIL.
  • Setup the default network configuration. This is accomplished through the NIOT command. After entering this command you will be prompted for a series of parameters. Hit carriage return for those parameters which you will not change. The parameters which must be changed are:
    
         Client IP   - This is the IP address which is to be applied to the board.
         Server IP   - This is the IP address to download from. At the time of
                       this writing, it should be set to the IP address of
                       "odf-srv02", which is currently 172.21.60.92. You may
                       retrieve this address by
                       typing "grep odf-srv02 /etc/hosts" at any generic Unix box
         SubNet Mask - Enter the value 255.255.252.0
    
    When all the parameters have been entered the command will ask whether you wish to commit these new parameters to NVRAM. Answer "Yes".
  • One of the printed message will give the VERSION of PPCBug. The version number must be at LEAST 3.4. At the time this guide was written (4/8/98) Motorola was shipping boards with a version of 3.5. If the version of PPCBug is less than 3.4 you must re-install PPCBug from a binary contained on disk (see Re-installing PPCBug) before proceeding further.
  • Power off the board and change the jumper setting to 2-3 and than reboot the board. This will now boot PPCBug from the SOCKETED flash.
  • You are now ready to install the bootstrap for VxWorks. This involves copying the boot image from disk through the network into RAM and than reburning the soldered flash with the copied boot image. Before proceeding Make sure your network configuration and RTC clock have been setup. Note also that this procedures assumes the VxWorks bootstrap has been complied and linked and resides in the /tftpboot/boot.bin
  • Using the NIOP command copy the VxWorks bootstrap into RAM. At the prompt type:
    
                    PPC1-Bug>NIOP
    
    This command will than prompt you for its parameters. Enter carriage return for those parameters for which the default is sufficient. The parameters you must respond to are:
    
          File Name      -> boot.bin
          Memory Address -> 100000
    
    Note: that since you are performing a read a value of zero (0) is used to indicate that ALL the file will be read, and thus the default is sufficient for the "Length" prompt. This command will take a couple of seconds to complete. When the transfer is complete a message will be printed out.
  • A snoop on a successful tftp looks like
    
          rom080 -> odf-srv02    TFTP Read "boot.bin" (octet)
       odf-srv02 -> rom080       TFTP Data block 1 (512 bytes)
          rom080 -> odf-srv02    TFTP Ack  block 1
       odf-srv02 -> rom080       TFTP Data block 2 (512 bytes)
          rom080 -> odf-srv02    TFTP Ack  block 2
       odf-srv02 -> rom080       TFTP Data block 3 (512 bytes)
          rom080 -> odf-srv02    TFTP Ack  block 3
       odf-srv02 -> rom080       TFTP Data block 4 (512 bytes)
          rom080 -> odf-srv02    TFTP Ack  block 4
    
  • An unsuccessful one looks like (don't know why this happens, maybe duplex mismatch since we don't get acks?):
    
          rom080 -> odf-srv02    TFTP Read "boot.bin" (octet)
       odf-srv02 -> rom080       TFTP Data block 1 (512 bytes)
       odf-srv02 -> rom080       TFTP Data block 1 (512 bytes)
          rom080 -> odf-srv02    TFTP Read "boot.bin" (octet)
       odf-srv02 -> rom080       TFTP Data block 1 (512 bytes)
       odf-srv02 -> rom080       TFTP Data block 1 (512 bytes)
       odf-srv02 -> rom080       TFTP Data block 1 (512 bytes)
       odf-srv02 -> rom080       TFTP Data block 1 (512 bytes)
       odf-srv02 -> rom080       TFTP Data block 1 (512 bytes)
    
  • Move the copied image from RAM to the soldered flash. At the prompt type:
    
                    PPC1-Bug>PFLASH 100000:FFF00 FF000100
    
    The command will than reprint the parameters and ask whether you REALLY want to reburn the flash (answer "yes"). Reburning the Flash takes about 20 seconds. Messages should be shown on the screen as to the progress of the re-burn. When the re-burn is complete you should get back the prompt.
  • Reset the board to make sure that the you have not disturbed the PPCBug you've been running out of (I.e. the socketed flash)
  • Power the board off, Reset the jumper to 1-2 to allow rebooting out the soldered flash. When the board is powered back up it should now attempt to boot VxWorks. Typically this will fail, because the VxWorks "boot line" has not been configured correctly. In the teststand the board should be configured as follows:
    
    boot device          : dc 
    processor number     : 0 
    host name            : odf-srv02 
    file name            : /dataflow/tgt/100FD.vxWorks.st.P02-09-04 
    inet on ethernet (e) : 172.21.61.90:fffffc00 
    inet on backplane (b): 
    host inet (h)        : 172.21.60.92 
    gateway inet (g)     : 
    user (u)             : YOUR_USERNAME_HERE
    ftp password (pw) (blank = use rsh): 
    flags (f)            : 0xa 
    target name (tn)     : rom080 
    startup script (s)   : 
    other (o)            : 
    
    NOTE: The IP address must be changed to be the address on the front panel "sticker" (e.g. rom128) when the SBC is placed in a real ROM.

    Reprogramming the Flashed Kernel from the VxWorks Prompt

    On the ROM, type:

    
    </dataflow/tgt/startup.save-kernel-to-flash
    

    The kernel version can be changed by editting that script.


    Re-installing PPCBug and Other Flash Commands

  • Note: that this procedure assumes that a binary version of PPCBug exists on disk as: /tftpboot/bug34R1.out
  • Using the NIOP command copy PPCBug into RAM. At the prompt type:
    
                    PPC1-Bug>NIOP
    
    This command will than prompt you for its parameters. Enter carriage return for those parameters for which the default is sufficient. The parameters you must respond to are:
    
          File Name      -> bug34R1.out
          Memory Address -> 100000
    
    Note: that since you are performing a read a value of zero (0) is used to indicate that ALL the file will be read, and thus the default is sufficient for the "Length" prompt. This command will take a couple of seconds to complete. When the transfer is complete a message will be printed out.
  • Validate that the image is correct by examining the checksums using the following command.
    
                    PPC1-Bug>MD 1FFFE0
    
    You should see five words of FFFFFFFF and one of FFFF0CFD. Note added in writeup: These were Joe's comments. When performing this operation we noted SEVEN longwords of FFFFFFFF followed by FFFF0CFD (go figure).
  • Move the copied image from RAM to the socketed flash. At the prompt type:
    
                    PPC1-Bug>PFLASH 100000:1FFFFF FF800000;r
    
    The command will than reprint the parameters and ask whether you REALLY want to reburn the flash (answer "yes"). Reburning the Flash takes about 20 seconds. Messages should be shown on the screen as to the progress of the re-burn (Note: its possible that the messages are suppressed if the "r" option is used). When the re-burn is complete the command will automatically reboot the processor. You should now see the new version of PPCBug in the startup message.

    Notes

    The following commands can be used for the various options if you get yourself in trouble:
  • Write SOCKETED flash from RAM:
    
                    PPC1-Bug>PFLASH 100000:FFF00 FF800100
    
    Note: The offset of 100 for the destination address is required for VxWorks (boot.bin) only
  • Write SOLDERED flash from RAM:
    
                    PPC1-Bug>PFLASH 100000:FFF00 FF000100
    
    Note: The offset of 100 for the destination address is required for VxWorks (boot.bin) only
  • Write SOLDERED flash from SOCKETED flash:
    
                    PPC1-Bug>PFLASH FF800000:100000 FF000000
    
  • Write SOCKETED flash from SOLDERED flash:
    
                    PPC1-Bug>PFLASH FF000000:100000 FF800000
    

    SBC Memory Addresses

    Memory addresses as viewed from the SBC:
    
    i960 control registers: 0xc4000000
    personality card reg:   0xc5000000
        int. store:         0xc5200000(TPC)
        UPC LUT             0xc5200000(UPC)
        int. store:         0xc7000000(UPC)
    controller card reg:    0xc6000000
    FCDM                    0xe0240000
    FCPM                    0xe0200000 (every subsequent 0x400 in master crate)
    FCPR                    0xe0210000 (every subsequent 0x100 in master crate)
    FCTM                    0xe0230000
    FCGM                    0xe0220000
    

    Note that accesses to everything except fast-control hardware (i.e. VME) appear byte-swapped.


    Some Schematics

    Paper copies of schematics can be found in Ray's filing cabinet at ir2 (near ray's teststand) and also in a filing cabinet in Mark Freytag's office in Central Lab.

    Some source code for the TPC "control" ORCA (U50) can be found in /afs/slac.stanford.edu/g/cad/archive/V2/babar/DAQ/Pc_ctrl. Mark Freytag also has a bunch of schematics and layouts and firmware source in his afs space. Also I think there Controller Card ABL files there in rom/cc_abel/CC1. I believe layouts can be viewed with the powerpcb program from innoveda (possibly included in epd releases?). Here's a mail message from Mark about what's in his area:

    A lot of the Babar stuff in my afs space.
    /u/ey/mlf/archived_projects/
    
    /rom has the source files for the the pld's
    /archive/jobfiles and pcbfiles have the pads layouts.
    /pvarch/babar has schematics
    
    /archived_projects/archive/jobfiles/35074505.c02.job is the cc layout.
    
    UPC schematics:
    
    /afs/slac/g/babar/detector/emc/elex/upc
    (in particular, the vl directory contains firmware in both schematic
    and .abl form).
    
    Electronic copies of fast control schematics and firmware:
    /afs/slac.stanford.edu/g/cad/archive/V1/babar/FCTS
    /afs/slac.stanford.edu/g/cad/archive/babar/FCTS
    TPC schematics:
    /afs/slac.stanford.edu/g/cad/archive/V2/babar/DAQ/rom/pc_rev3
    PMC schematics:
    /afs/slac.stanford.edu/g/cad/archive/V2/babar/DAQ/rom/pmc_rev4
    UART schematics:
    /afs/slac.stanford.edu/g/cad/archive/V2/babar/DAQ/rom/uart_rev1
    CC schematics:
    /afs/slac.stanford.edu/g/cad/archive/V2/babar/DAQ/rom/cc_rev2
    UPC schematics:
    /afs/slac.stanford.edu/g/cad/archive/babar/UPC
    
    schematic viewing software:
    
    older version: pv61 (powerview)
    newer version: epd11
    
    sample setup for pv61 on solaris:
    
    setenv WDIR ~cpo/vl:/afs/slac/package/ecad/custom/viewlogic/standard:/afs/slac/package/powerview/new/pv61/standard
    setenv SYSPLT ~cpo/vl/plots
    
    source /afs/slac/package/ecad/custom/util/cae_aliases
    
    to open a schematic with viewdraw in pv61, select "create project"
    (even if it's not your project) then set the project pathname to be
    the directory containing the "sch" directory (schematic directory, I
    guess).  this is typically one below the directory name given above.
    viewdraw should then open a window prompting you for a file name.
    select the "yourprojectname".1 file for the top-level schematic.
    
    to print with pv61, use "project print" and "print setup" to print to
    file (have to open viewdraw to get these). (later: cpo couldn't get
    my own instructions to work.  don't understand).
    
    to zoom in on a schematic, in viewdraw select "view zoom".  the middle
    mouse button then will select a region to zoom in on.  F10 will unzoom.
    
    to switch between pages, right click on the schematics and select
    "push" then "sheet".
    
    

    FCTS Firmware Programming

    Currently the fcdm firmware is in /afs/slac.stanford.edu/g/cad/archive/V1/babar/FCTS/fcdm/fcdm/cpld/*/*.jed and /afs/slac.stanford.edu/g/cad/archive/V1/babar/FCTS/fcdm/V3xil/fcdmregxil.jed. Unfortunately, the firmware needs to be in a writeable directory, since jtagpgmr wants to save a logfile. Up until now I've been copying it to my local (writeable) area, which is pretty horrific. Other fast control modules are in similar directories.

    At the time of this writing, JTAG programming uses the xilinx "pod" plugged into the 9-pin serial port "B" on bbr-dev15. Attached to this pod there are (at the time of this writing) is a custom cables that ray has to JTAG fast control boards. Execute the following:

    
    source /afs/slac/package/ecad/custom/util/cae_aliases
    set_xact_33i
    cd <wherever the .jed file is> (see above)
    jtagpgmr &
    

    Empirically, this JTAG program seems to have problems jtagging filenames with upper-case letters.

    jtagpgmr wants to write a log file in a writeable directory. maybe easiest to copy the .jed file to a writeable directory, or softlink it.

    
    Edit --> Add device
    Select correct .jed file for desired device
    
    Click on the device to highlight it.
    
    Operation --> Verify or Program
    
    There will be about a 25 second wait while the software attempts
    to "connect" to the cable.
    
    Move the cable to the next device.
    On UNIX you have to reset the cable by.....
    Output --> Cable reset
    

    For the FCDM, the following chips get the following .jed files (in the schematic-archive area described above).

    
      U11 = pd_cntl1.jed
      U9  = pd_delay.jed
      U39 = fcdmRegXil.jed (this is a "version 3" that fixes register corruption)
      U31 = pd_vme_l.jed
    

    "Version 1" in U39 is only on one-time-programmable XC7300 chips. Empirically, those chips don't appear to corrupt registers. So it's only "Version 2" that is bad. This version number is readable in the ID register of the board.

    For the FCPM, David Hamilton found that the following chips get the following .jed files. These are in the schematic-archive area described above: V1/babar/FCTS/fcpm/fcpm/source (with one exception, noted below):

    
    U7  = csr_id.jed
    U11 = par2ser_new.jed
    U14 = big_cnt2.jed
    U45 = pm_vmell.jed
    U47 = time_ltch.jed
    U56 = pm_cnt_2.jed
    

    Here is one exception. This is FCPM firmware modified by Matt Weaver to support trickle injection "markers":

    
    U25 = V1/babar/FCTS/fcpm/fcpm/cpld/vme_cmdf_mrk/vme_cmdf_v1.jed
    

    I960 Firmware Programming

    The i960 firmware can be programmed similarly to the FCTS firmware, except with a different custom cable that ray has. This cable plugs into the backcard of the ROM. The i960 xilinx firmware is in:

    /afs/slac.stanford.edu/g/cad/archive/babar/odfLabPC/TPC/pMC/pmc_cntm.jed
    

    There are two other versions there (PMC_CNTL.JED, PMC_CNTR.JED) but those don't verify against working boards. There is also a .jed file in /afs/slac.stanford.edu/g/cad/archive/babar/odfLabPC/TPC/pMC/xproj/ver1/rev2/pmc_cntl.jed but that also doesn't verify.

    Note that the cable end that plugs into the xilinx "pod" has two of it's holes filled with solder, to act as a key. The other end has a red wire (VCC) that goes into the bottom right of the connector on the ROM back-card. The green wire (GND) goes into the top right of that same connector.

    Also note that the fibers must be "locked" for this test to work correctly. The event isn't checked if there is damage. This is a bad feature which I should fix.


    TPC Firmware Programming

    For the TPC, the large ORCA chip needs 0506XX (XX is 00 or 01 depending on which half of the firmware it is. The DCH is an exception to this. It has a special version of firmware (10150 and 10151) that gives it 4 larger intermediate store buffers instead of 8 smaller buffers. This is because their front-ends can ship up more data than there is space in the smaller intermediate store, which gave PayloadTruncated damage (which is dangerous because it creates a size bias in the readout). With this version the buffer size is jumper-selectable by TPC jumper JP1, so in principle it could be used in every TPC, but we haven't made that happen. The small ORCA chip needs 0728XX.

    I think the TPC firmware source code is in: /afs/slac.stanford.edu/u/ey/mlf/archived_projects/rom/pc_vhdl.


    CC Firmware Programming

    To program the Controller Card and UPC MACH chips, use the ispvm software installed on afs and the parallel port cable that was purchased from lattice. If the software is lost, it can be downloaded for free at latticesemi.com at the time of this writing. A copy also exists on bbr-dev15 in /dataflow/machprog/ispvm_v12_1_2_unix.tar.gz. "gunzip" this file and "tar xf" it. The file ispVMInstallation.pdf will be in the extracted tree and contains running instructions.

    To use ispvm, you need a .ispvm.ini file in your home directory. cpo's looks like this:

    [AutoScan Options]
    LimitedScan = 
    
    [Column]
    Number Column = 
    
    [Directory]
    Database File = /afs/slac/package/lattice/ispvm_v12.1.2/isptools/ispvmsystem/Database
    Start Directory = /afs/slac/package/lattice/ispvm_v12.1.2/isptools/ispvmsystem
    Working Directory = /afs/slac/u/ec/cpo
    
    [DisplayUES Options]
    ASCII = FALSE
    Decimal = FALSE
    HexaDecimal = TRUE
    
    [Font]
    Family = 
    Italic = 
    Point Size = 
    Strikeout = 
    Underline = 
    Weight = 
    
    [ISPVMSYS GUI]
    Default XCF = /afs/slac.stanford.edu/u/ec/cpo/ccmach.xcf
    
    [Log File Options]
    Clear = TRUE
    Name = /afs/slac/u/ec/cpo/ispvm.log
    Write = TRUE
    
    [Port and Cable Setting]
    Baud = 38400
    Cable = LATTICE Parallel
    ISPEN Connected = FALSE
    PROG Connected = FALSE
    Port = /dev/ttya
    TRST Connected = FALSE
    

    In particular, if you don't have the "working directory" set correctly, ispvm will refuse to run, complaining about a non-writeable directory.

    To use the afs version, execute the following:

    source /afs/slac/package/lattice/ispvm_v12.1.2/setup_env.csh
    (to verify) ispvm /i /afs/slac.stanford.edu/g/cad/archive/babar/DAQ/CC/machverify.dld /t /o /Parallel
    (to program and verify) ispvm /i /afs/slac.stanford.edu/g/cad/archive/babar/DAQ/CC/machprogram.dld /t /o /Parallel
    

    The program can also be run with a GUI by typing only "ispvm".

    A successful verify run looks like this:

    cpo@bbr-dev15 $ ispvm /i /afs/slac.stanford.edu/g/cad/archive/babar/DAQ/CC/machverify.dld /t /o /Parallel
    
    Initialize....
    Check configuration setup: Start.
                    JTAG Chain Verification. No Errors.
    Check configuration setup: successful.
    Turbo Download Operation: Start
    Operation Select:0
    Device1 M4-256/128:Verify Only
    Device2 M4-256/128:Verify Only
    Device3 M4-256/128:Verify Only
    Turbo Download Build ispSTREAM: Successful.
            Processing JTAG chain
            Operation Done. No errors.
    

    Some important notes:

  • There appears to be a bug on the CC jtag connector. The parallel port cable (which we purchased for $60 from lattice) is expecting to receive VCC from pin6 on PL48 on the board. This pin is unconnected by default on the CC, which prevents ispvm from finding the cable (under "Options->CableAndIoPortSetup"). For the moment, when we want to program or verify the firmware, I have been adding a jumper wire, from pin6 to pin1 of one of the resister SIPs near the dip switches. Note that the 10-pin JTAG port has "header" pin numbering, not the "U" numbering of chips. So pin 6 is the "middle-top" pin of the JTAG header. LATER: mike przybylski modified the cable with a clippable jumper that can be connected to the P3 connector "top pin closest to the front" instead of soldering a jumper on the back of the CC.
  • When plugging in the cable into the unkeyed connector, make sure that the red wire on the cable goes to pin 6 (middle-top pin). Pin 1 is in the lower left. I think I plugged it in backwards once on 20000850 and believe I damaged the board electrically.
  • You can read details about how to use ispvm in: /afs/slac/package/lattice/ispvm_v12.1.2/ispvm_install.pdf
  • The program operation and the verify operation both take about 2 minutes to do all 3 chips.
  • On Solaris, I had difficulty reading the programs html help pages (gave me an error message about some java stuff). But after poking around, the following URL provides an index to some rudimentary help: file:/afs/slac/package/lattice/ispvm_v12.1.2/isptools/ispvmsystem/webhelp/toclist.htm
  • ispvm seems to need to be configured the first time you run it. It will complain about not being able to write a log file, but will then give you a dialog box where you can set up some more sensible defaults. It then seems to store these defaults in a file named .ispvm.ini.
  • I tried verifying the CC firmware against the JED/jed files under /afs/slac.stanford.edu/g/cad/archive/babar/odfLabPC/TPC/cc_mach, but failed to find files that would verify against slac ID 20000738. So I uploaded versions from that board, and saved them in /afs/slac.stanford.edu/g/cad/archive/babar/DAQ/CC. These versions verified against slac ID 20000706. Later: for slacid 20001933, found that /afs/slac/u/ey/mlf/archived_projects/rom/cc_abel/Cc1_24_full/cc1.JED and /afs/slac/u/ey/mlf/archived_projects/rom/cc_abel/Cc2_kirkby/cc2.JED verified. Haven't tried to find the third jedec file yet, but it's probably there.
  • I think the CC firmware is in /afs/slac/u/ey/mlf/archived_projects/rom/cc_abel, but one also needs a schematic that shows how the various modules are plugged together (there are paper copies floating around, we don't have a license for the software that produced them).
  • it seems like the programmer doesn't like CAPS in the .jed filename. Worked around this with softlinks.
  • Aug 22, 2006: We have compiled new CC2 firmware with a fix that uses all 4 front-end buffers correctly. ABL modifications to QCTRL.ABL were done by leonid, and tested by cpo. Compilations were done on rdpc out of cpo's Z:/New_CC2_buffer_fix area using a Synario Design Automation CD that was discovered by cpo in building 40 room G223 (old dataflow lab). Mark Freytag performed some magic with the licensing stuff to make it work. The synthesis is archived in /afs/slac.stanford.edu/g/cad/archive/babar/DAQ/CC/New_CC2_buffer_fix/cc2.JED and can be compared to the old synthesis in ~mlf/archived_projects/rom/cc_abel/Cc2_kirkby. The timing reports look very similar, but there is no constraint file that I've seen, so that remains a concern at the time of this writing.

    UPC Firmware Programming

    For the UPC, there are three MACH chip that have to be programmed. This can (and has) been done on bbr-dev15 with the same ispvm software used for the CC MACH chips. The JTAG connector pinout is the same (and VCC is correctly connected on pin 6, unlike the CC).

    As an FYI, once before when we tried, Mark Freytag did it for us using a PC in the old dataflow lab in building 40 room 208. The version mark found on some PC disk didn't verify again the version in one of the UPCs. So we just copied the version in the UPC. We used that to reprogram a mach that had been burned out on a UPC while probing. I don't know what happened to those copies. Today (sep 11, 2003) we copied the version from UPC 20001716. They were copied to:

    /afs/slac.stanford.edu/g/cad/archive/babar/DAQ/UPC

    In the 3 files:

    addrdecoder_mach211.jed
    transcontroller_mach111.jed
    trigpatterngen_mach111.jed
    

    The first one is U26 and is the first scan-path device. The second is U50 and is the second scan-path device. The third is U40 and is the third scan-path device. These files should be used to program the UPC machs.

    Note that one can click "scan" on the ispvm gui to auto-detect the type of chip and the order.

    From the command line the machs can be verified as follows:

    ispvm /i /afs/slac.stanford.edu/g/cad/archive/babar/DAQ/UPC/upcmachverify.dld /t /o /Parallel

    From the command line the machs can be programmed as follows:

    ispvm /i /afs/slac.stanford.edu/g/cad/archive/babar/DAQ/UPC/upcmachprogram.dld /t /o /Parallel

    Note that the "/t" puts it in "turbo" mode, which attempts to do all three chips in the chain in parallel.

    The output of a successful verify run looks like:

    cpo@bbr-dev15 $ ispvm /i /afs/slac.stanford.edu/g/cad/archive/babar/DAQ/UPC/upcmachs.dld /t /o /Parallel
    Initialize....
    Check configuration setup: Start.
                    JTAG Chain Verification. No Errors.
    Check configuration setup: successful.
    Turbo Download Operation: Start
    Operation Select:0
    Device1 MACH211SP:Verify Only
    Device2 MACH111SP:Verify Only
    Device3 MACH111SP:Verify Only
    Turbo Download Build ispSTREAM: Successful.
            Processing JTAG chain
            Operation Done. No errors.
    cpo@bbr-dev15 $ 
    

    The output of a successful programming run looks like:

    cpo@bbr-dev15 $ ispvm /i /afs/slac.stanford.edu/g/cad/archive/babar/DAQ/UPC/upcmachprogram.dld /t /o /Parallel
    Initialize....
    Check configuration setup: Start.
                    JTAG Chain Verification. No Errors.
    Check configuration setup: successful.
    Turbo Download Operation: Start
    Operation Select:0
    Device1 MACH211SP:Erase,Program,Verify
    Device2 MACH111SP:Erase,Program,Verify
    Device3 MACH111SP:Erase,Program,Verify
    Turbo Download Build ispSTREAM: Successful.
            Processing JTAG chain
            Operation Done. No errors.
    cpo@bbr-dev15 $ 
    

    There were 5 UPCs in the system that had outdated trigger-pattern-generator firmware. The symptoms were: the serial-number/bit-pattern would flip randomly during the EMT serial-number calibration. Using Matt's trigger-sum-checking software, the real trigger-sum data was unaffected, fortunately. There is one caveat: with the EMT patch-panel and cables installed in Sept 2003, the skew between the clock and the frame seemed to change, and the 5 UPCs with the wrong firmware had the clock/frame drift too close together, and the result is the output frame on U15 of the EMT TPB boards was not reliable (giving "ERR2" lights on the TPB front-panels). This makes me strongly believe that the more modern version of the firmware delayed the frame (and perhaps the data) relative to the clock by a couple of ns. Indeed, in Jeff's code, it appears that the output of pin 40 is routed back to pin 41. We believe this was to introduce extra delay. Perhaps to account for cable-skew.

    For the UPC xilinxes/alteras, the deformatter/concierge chips on barrel UPCs need SPROM version DC6, and the trigger summers need TS7. On endcap UPCs the deformatter/concierge chip needs DC6E ("E" for endcap). The gate generator needs G17. The version number is indicated by a sticker on the SPROM.


    Xyplex Setup

    I watched Owen Saxton do this once. He used the "Hyperterminal" program with 9-pin serial port COM1. This has an adapter and a phone-jack style connector attached to it. Presumably this cable/adapter came with the xyplex. Ray has a set too, but on first attempt we couldn't make this stuff work with his laptop. Terminal settings were:

  • 9600 baud 8-N-1 (although "autobaud" should work
  • Flowcontrol set to "Hardware"
  • Emulation set to "ANSI"
  • "function,arrow and ctrl keys" as "Terminal keys".

    I found this at http://www.gno.org/~gdr/xyplex. It's also in one of the manuals somewhere (page 83, but I don't remember which manual).

    Setting An MX-1600, MX-1608 or MX-1450 To Factory Defaults
    
       1. Straighten a paper clip and press into the pin-size hole next to console LED on the front panel of the unit. All LEDs on the front of the unit will light up.
       2. Press the paper clip in again and hold it in for 3-5 seconds. The LEDs will light up in a sweeping fashion from right to left, then left to right. When this sweeping stops, there will be 2 or 3 LEDS to the right lit, at this point release the paper clip.
       3. The LEDs will light up in a countdown pattern to 1 (diagnostic test pattern). Then they will all go out and the RUN light will be flashing very fast. You should have a terminal attached to one of the serial ports on the back of the unit. Press the ENTER key several times for the port to autobaud. You will see a text display similar to this:
    
    Terminal Server, Type 97, Rev G.00.00 
    Ethernet address 08-00-87-05-A1-16, port 2 
    Configuration in progress.  Please wait 
    
       4. Type the password "access" (there is no password prompt and it will not display the characters you type) and then press ENTER on your keyboard. The menu below will display. Please select the menu options and answer the questions as detailed below to default your unit.
       5. To Default The Server Load/Dump Parameters:
    
         Welcome to the Configuration Menu. 
     
         Terminal Server Configuration/Maintenance Menu 
     
            1. Display unit configuration 
            2. Modify unit configuration 
            3. Initialize server and port parameters 
            4. Revert to stored configuration 
            S. Exit saving configuration changes 
            X. Exit without saving configuration changes 
     
         Enter menu selection [X]: 2
    
         Initialize configuration to defaults (Y,N) [N]?  Y
    
          Press ENTER on your keyboard at this time...
       6. To Default The Server & Port Parameters:
    
     
    
         Terminal Server Configuration/Maintenance Menu 
     
            1. Display unit configuration 
            2. Modify unit configuration 
            3. Initialize server and port parameters 
            4. Revert to stored configuration 
            S. Exit saving configuration changes 
            X. Exit without saving configuration changes 
     
         Enter menu selection [X]: 3 
    
         When the software has been loaded, should default server and port
         parameters be used (Y,N) [N]? Y
    
       7. Save Configuration Changes And Reboot The Server:
    
         Terminal Server Configuration/Maintenance Menu 
     
            1. Display unit configuration 
            2. Modify unit configuration 
            3. Initialize server and port parameters 
            4. Revert to stored configuration 
            S. Exit saving configuration changes 
            X. Exit without saving configuration changes 
     
         Enter menu selection [X]: S
    
         Save changes and exit (Y,N)  [Y]? Y
    
          The access server will now reboot using factory settings. 
    
    

    To setup the unit, owen did the following:

    2 (modify unit config)
    1 (modify first configuration record)
    N (i forget what the three settings were that took these responses)
    Y
    N
    m (disables these methods of image loading)
    b
    r
    N (no, to question "enable all methods of param loading")
    m (disables these methods of parameter loading)
    b
    r
    N (no, to question "enable all methods of dumping")
    m (disables these methods of dumping)
    b
    r
    <CR> (to use the default image filename)
    ww.xx.yy.zz (to set ip address)
    

    After that he moved up one or two menu levels, and "exitted and saved" which was done by using "s" at some point.

    The next step after this is to install the xyplex and hook up a network connection. Owen has some xyplex tools in ~saxton/IR2AdminTools. To finish the configuration, the script name is xyplex-config in that area. These scripts are written with "expect" which I know nothing about.

    To configure an ir2 xyplex, get on a unix machine which can ping it (bbr-tty4 in this example) and execute:

  • ~saxton/IR2AdminTools/xyplex-config bbr-tty4
  • ~saxton/IR2AdminTools/xyplex-gateway bbr-tty4

    Other miscellaneous xyplex commands that work in privileged mode:

    sho para serv (shows things like last write time to flash)
    sho server ip (shows "current" gateway info and ip address)
    list server ip (shows "permanent database" (what should be on flash))
    set priv
    def server ip primary gate addr 134.79.159.1 (set up gateway)
    logging off and on forces a save to the flash
    

    System Ugliness That Should Not Be Forgotten

  • The controller cards for the UPCs have not had the version of firmware installed that fixes the bug with front-end buffer modelling (although it wouldn't hurt to install it). Be careful when swapping CCs between UPCs and TPCs. The list of boards that were modified is in ~cpo/problems/firmwarefix.txt (it was too painful to put it into the problem database 1 board at a time).
  • InEMC crate 9 slot 2 (or 3, can't remember which) there is a slot-number jumper on the P3 backplane that intermittently fails. As a workaround we have swapped the slot 2 and slot 3 ROMs there.
  • In the master crate the second FCPM slot (counting from 1) intermittently stopped sending some trigger lines to the system. This caused us to lose a few runs. Don't use that slot.
  • In the ELP logical crate, there was some problem with the leftmost slot (I don't remember the details). We moved it to the next slot over.
  • The DCH TPCs have different firmware (for 1 ORCA FPGA) to allow them to have 4 double-size istore buffers instead of 8 regular-size buffers.
  • All EMC IOBs have a problem where the tag bits on fiber C get intermittently corrupted because of a board crosstalk problem. This means that the teststands have to be cabled correctly or you'll see OutofOrder damage (since we only disable the check for fiber C).

    Why Repairs are Frustrating

  • replacing components can cause the pads to lift, especially on glink chips
  • sometimes there is no good place to rest your hands because of the heat on the board
  • there is no good place to clip to a ground, especially on the back of a board
  • it's difficult to count pins as pin densities get larger
  • it's difficult to find all instances of a signal on schematics
  • schematics sometimes not well organized
  • it's easy to have a probe slip or a ground clip come off and blow up other parts. for complex boards like the UPCs, my feeling is that for every 4 problems we fix we create 1 new one. chip clips help, but would be better if the board could be held in a horizontal position so gravity wasn't such a big factor.
  • you need two people: one to hold the scope probes and one to read the schematics in order to avoid having the probe slip
  • sometimes no silkscreen for resistors/capacitors
  • have to re-learn how the board works every time one does a new round of repairs
  • there are known unfixable failure modes that confuse the problem (e.g. turning on/off the light causes even "good" UPCs to fail).
  • we don't have neat access to electronic copies of layouts/schematics for all boards. and several different standards for these.
  • even if we did have electronics copies of everything, need to learn search mechanisms better.
  • to really understand how a board works, have to know both firmware and hardware
  • can't look at some signals (e.g. GHz glink stuff).
  • intermittent problems are hard
  • we don't have good enough test patterns and test software
  • headers aren't "keyed" so it's easy to put on a JTAG connector incorrectly.
  • many different compilers for firmware, that rot with time: either the compiler software isn't supported on newer operating systems, or the licensing daemon disappears over the years. same is true for firmware downloading.
  • can't get at some signals at all (e.g. on PMC card).
  • schematics out of date (e.g. DCH schematics)
  • components (like capacitors) get scraped off because of bad mechanical tolerances