Instructions to run GLT teststand DAQ / Standalone tests ======================================================== You can find the latest version of this document at: /afs/slac/g/babar/trigger/global/docs/teststand.txt Last update: Jan/9/99 Mar/19/99 Major file serving & dataflow change Jul/2/00 Reconnect teststand in dataflow Lab. Mar/6/01 Moved ROC13 serial connection to xyplex Jan/4/02 Teststand files-> bbr-nfs03 + other updates Oct/15/02 Updated DAQ test / UNIX nodes etc. Introduction ------------ The GLT teststand consists of the frontend GLT crate hosting the GLT at slot-9 and the GLINK board DCC at slot-5, and the DAQ crate hosting the fast control distribution module (FCDM) and partition master (FCPM) and the ROM which is ROC13. The test system has a UNIX control terminal communicating to the ROM via ethernet, while the control of ROM download is through a xyplex port (roc32 3000) at dataflow lab. At the moment the tests for memory+data-path etc. are done with a standalone Teststand, while the DAQ tests have to be done with the formal dataflow platform (default platform=27 for test lab. system). You must use a Solaris Sun node in the same subnet (Pub1) as the ROM ROC13. One such UNIX node is DALI (Ultra-5 in test lab). The Environment setup --------------------- The environment setup for running the teststand tests can be done from source /afs/slac/g/babar/trigger/global/test/setup Some of the important directories are assigned symbols for easy access in this setup which are worth remembering: setenv ODF_BASE /afs/slac/g/babar/dist/online_releases/6.3.0/ setenv GLT /afs/slac/g/babar/trigger/global setenv TESTSTAND $GLT/test/v3 setenv ODF_ROOT /nfs/bbr-srv02/dataflow setenv GLTNFS /nfs/bbr-nfs03/detector/trg/teststand/glt setenv GLTCONF /nfs/bbr-srv02/dataflow/constants/glt setenv GLTLAB $ODF_ROOT/1B/18 An important change at end of 2001 is that the /nfs/bbr-srv02/dataflow (ODF_ROOT) disk became mostly readonly to prevent interference of other activities with the datataking. The limited UNIX write access is only possible from IR2 nodes such as bbr-dev20. The /nfs/bbr-nfs03/detector disk is now mountable from the ROM for none IR2 teststands. Most of the GLT teststand utilities have been moved to the new disk for easier access. A summary of what files are kept at where for some important locations: $ODF_ROOT/glt/ contains the official executables for the ROMS such as the libGltApp.o for real DAQ and testserver.o. Only writeable from IR2 nodes. $GLTLAB softlinks for DAQ tests FCGUI, eventDecoder etc. This is th edefault directory for DAQ tests. $GLTLAB/app The ROC13 ROM boot up control directory. The ROM boot is controlled by the script startup which is typically a softlink pointing to alternative startups such as startup.DAQ and startup.TESTSTAND. Switching between different types of tests can be done by deleting startup and softlink it to another. The executables used for the ROM boot are also softlinks which can be controlled similarly. e.g. If built a new version of testserver for testing, just replace the testserver softlink. $GLT The GLT documentations, design files etc. all live under this directory. $GLT/docs This contains the GLT documentations including this file (teststand.txt). $GLTNFS This is where all the work files for the teststand lives. Macro files, temporary I/O files, MC data etc. This used to be at /bbr-srv02/dataflow/constants/glt/ (GLTCONF) in the past. $GLTCONF This contains some macros under /CONFIG_*/ which are still used by some manual DAQ tests. $GLTNFS/O.vxworks A convenient place to hold test executables for the ROM. $TESTSTAND This is the default UNIX teststand operation dir. The UNIX code are under 0_control/ and the ROM code are under 1_source. The UNIX control tcl files are under the tcl/ directory. Another feature worth explaining is which disks are mounted on the ROM and what they are known as there: UNIX disk Mounted on ROM as /nfs/bbr-srv02/dataflow /dataflow /nfs/bbr-nfs03/detector /det So that UNIX operations need to use the full UNIX disk names, while the ROM code in e.g. testserver has to refer to the disks as the ROM mounted names. For the teststand operations, we should use the /det disk whenever possible. GLT test operation startup -------------------------- The teststand operations require two types windows: 1) A window telnet to the ROC13 via xyplex. 2) A window on the a PUB1 subnet Solaris node e.g. DALI. (DAQ operations need at least two window of this type). a) Environment setup: source /afs/slac/g/babar/trigger/global/test/setup b) Display setup: setenv DISPLAY for the UNIX window of class 2) above. c) Terminal connection to ROM: From the window 1) on any node, issue command telnet roc32 3000 (if you already executed the /global/test/setup, an alias is setup for this as tiproc13) If it complains "All ports are busy", it means somebody else has connected up to ROC13 through that port which you'd better find out. It is therefore important to remember quiting the ROM connection when you are done as an effective way of sharing the teststand. To quite the ROM connection, issue the command characters ctrl + ] simultaneously in the ROM xyplex window. To make a manual reboot of the ROM, issue the command ctrl + x in the xyplex window. d) Check ROM setup: cd $GLTLAB/app ls -rtl What is the startup soft link pointing to ? If it is pointing to startup.TESTSTAND, it means the reboot will load the standalone teststand in the ROM, while startup.DAQTEST will load the dataflow DAQ test. If it is not what you want e.g. currently point to the teststand while you want to run DAQ test, then rm startup ln -s startup.DAQTEST startup *NOTE: This directory is write protected and you have to logon to an IR2 machine such as bbr-dev20 to make this softlink change. e) Startup ROM and check running GLT ? Use Ctrl+X on the ROM serial window to get the ROM to reboot. At the end of the message scrolling on the ROM window, check what it sees as platform/detector nmuber: you should get detector=8, platform=1B (27). If detector is not 8 (could happen if some DCT folks were using the teststand then it might be set to = 6), you need to set a dip switch on the FCDM: The "Local-ID" dip switch should be 0 1 |----------- --------- | The top 4 bits = detector 0 | * | | 1 | * | | 2 | * | | XilinX 3 | * | | The bottom 4 bits are the 4 | * | | lowest 4 bits of platform 5 | * | | 6 | * | | 7 | * | | --------- |____________ (** Note: the numbers around the switch are the number printed on top of the dip switch itself. Please ignore the white number printed on PCB - which were reversed in order) After setting the FCDM dip switch and powered on the DAQ crate, check the ROM reboot info to verify the detector setting. Running the DAQ test -------------------- The BaBar dataflow applications typically operate with 3 components. a) The ROM directly connected to frontend electronics with programs operating in its CPU. b) The event logger/decoder on UNIX for receiving DAQ data shipped up by the ROM. The version used for our tests does not do logging but simply dump brief info about the events coming up. c) A GUI on UNIX control the DAQ operation. You need two terminal windows (A & B) logged on to odf03 and execute the setup. First set default directory cd $GLTLAB for both windows. 1) Check the ROM reboot from the XYPLEY serial port window that the ROM was executing the gltTest program and the detector & platform numbers are OK. 2) Now go to UNIX window A to issue command: eventDecoder to bring up the UNIX side event receiver. At the prompt asking for crate list, enter FFFFFFFF and return. When it say "Damn you, son of McBain..." it means it is up and running. You typically don't have to pay much attention to this window after this. 4) Now start the control GUI from Window B with command: fcgui and at the prompt asking for crate list, again enter FFFFFFFF and return. The GUI initialization typically takes a minute and prompt you to hit return "when event level is ready" (i.e. the eventdecoder is running). The GUI should then come up. The dataflow control involve a configuration phase followed by a series of loop levels (begin run,meta,major,minor cycles), while the core phase is the L1 Accept. We don't use much of the intermediate loop level controls so that only the Configuration and L1 Accept controls are relevant. The "GLTTEST" key (=451 decimal as of Oct/02) which switches configuration from normal DB pointers to production full GlobalTCs, a dummy GlobalTC this key points to sets a control flag to tell L1GltConfigure to use a flat file macro on disk. The actual configuration is controlled by a combination of the configuration environment variable and the file $GLTCONF/CONFIG_27/GLT_Configure This is recently separated out from the CONFIG_IR2 directory to avoid any interference with the IR2 operation. The default GLT_Configure points to macro_test, which shares the same style macro command files as the standalone teststand as explained in the next section. There are various command codes for CSR R/W memory R/W and start-playback etc common to both teststand and DAQ tests. However, there are few special commands for the DAQ tests only worth noting: command 201 N1 N2 N3 N1=0 No event dump (Evt dump) N1=1 First N2 events will be dumped on ROM window N1=2 Every N2th event will dumped out N1=3 Every event will be dumped out N3=0 no print lines N3=1 Dump minimal event header listing N3=2 Extended header + TC dump N3=3 Extended header + rawdata + TC dump 202 N N =0 Suppress all printout in configuration N =1 long print out of configuration details 331 N1 N2 N1=0 ignore LUT download (LUT dnld) N1=1 single word LUT download by reading LUT file N1=2 block write LUT downloiad by reading file N1=3 single word LUT download by generating LUT N2=0 force download N2=1 do version check of a few registers to decide whether needing LUT download first 333 [file] Whole trigger line definition file for trig_dec 660 No OP 661 Clear readout 662 Synch 665 Cal-strobe 666 Start-playback Warning: The LUT single fixed location check option for determining the need of download the whole LUT is generally OK to catch "natural diasaters" e.g. the GLT board was powered off. However, if one ran the macro_test pattern test, then followed by a normal event run then it is very much advisable to force the LUT download once. The macro_test overwrites a few locations in the LUT which cannot be picked up by the fixed location check to ensure the whole LUT was really intact. (This is especially important on the IR2 platform: One must do a forced LUT download after a macro_test type of test run.) For running the standard test of input playing driving a test pattern defined by IN_DAQEVTS at 16 clock-8 ticks apart, same for all types of DCT/EMT signals. First check $GLTCONF/CONFIG_27/GLT_Configure is pointing to $GLTCONF/CONFIG/macro_daqtest. If not, use another terminal to change it: cd $GLTCONF/CONFIG_27 rm GLT_Configure ln -s macro_daqtest GLT_Configure Then following the GUI arrows: a) There is no special actions needed for configuration now so that you can simple click through it. The default macro_tpinc setup will ignore LUT download. You can change macro_test to alter e.g. what level of event info dump is needed for the ROM window (you may want to turn them off for timing test, multiple triggers etc.). b) Click all the way down through begin run and various cycles. c) Checking the "enable" environment which should say operation on internal trigger, only the first command is enabled for L1 accept with no delay. The repeatedly clicking on the "enable" L1accept arrow should bring up one event per click. Note that these hand triggers are completely unsynchronized with the playback. In case of transitions get stuck (state turned yellow and never return red arrow) or every transitions ends with an error not moving on to the next level, you need to hit REBOOT and then DISSOLVE to go out and try again. Most of these failures do need to reboot the ROM. The DISSOLVE is important as a ctrl+C to abort abnormally may leave things hanging in the partition master which will cause a stuck yellow state immediately at the startup of the GUI next time. In case of abnormal ctrl+C abort, you can issue a command ipcCleanup afterwards on the unix windows to clear the FCT hardwre states. Running the Standalone teststand -------------------------------- The standalone teststand has two parts: a) A testserver running on the ROM to communicate with GLT directly. b) A teststand GUI running on UNIX to control the tests. First check the XYPLEY window to make sure the ROM is running the testserver program and detector number at ROM reboot is correct. You need one window on a UNIX on on the same subnet e.g. odf02: cd $TESTSTAND 0_control/teststand This will bring up the UNIX teststand GUI. First hit the "initialize" button which will connect the teststand GUI to testserver. Then bring up both "Write & read CSR", "Write & Read Memory for GLT" panels. Try a write CSR5 software LED first to check the teststand is indeed talking to the GLT. Most tests are controlled from the "Write & Read Memory for GLT" panel e.g. : 1) Various looped tests: MC event loops, Memory test etc. are on the top row. The full GLT memory tests will need to run 3 separate control files: memtest_std (All registers/memories excluding LUT) memtest_lut1 memtest_lut2 which is best done with default walking one pattern option=2. 2) "Load/Save from file" subpanel contain a range of generic R/W tests a) For download/read from/to file for a specified memory range. Note that for the readback "Save into user file" operation, it is better to use the default file "OUT" for output, then copy it to different file if want keep with a distinct file name. This is because the ROM cannot open a new file for writing as the default new file access mode restriction, eve if it is an NFS file . b) The macro option allow an arbitrary list of memory operations specified by the macro file. Some useful macros (on $GLTNFS) : macro_tpat5 prepare an AAAAAA/555555 alternate trigger pattern RECIN Read in DR1 & DR2 input record/play memories to two files. Use $GLTNFS/dumpdr program to dump these files. OUTPLAY Load pattern into ODAQ rec/play memory and playback through frontend cable to the GLT inputs and store recording data in DR1,DR2 recording memory. Again use dumpdr program to inspect the result. The macro command lines (all address/values = HEX): {first word=No. of command lines to follow} command=1-4: CSR download [CSR address to download] [value to download] command=0: Download a file to memory [0] [Block-addr] [Start-addr] [end-addr] [format 0/1=16/32 bits] [file] command=10: write a single memory address [10] [Block-addr] [address] [format 0/1=16/32 bits] [value (hex)] command=20: readback a memory range to file [20] [Block-addr] [Start-addr] [end-addr] [format 0/1=16/32 bits] [file] command=666 Start-playback. Reminder: the [file] used in these operations need to be addressed as the ROM file path /dataflow/constants/glt/... etc. 3) The "Write & Read Memory for GLT" panel single word R/W tests: Most of our memories are 32 bits. Always remember to set the block address for any operation, especially after doing something else to come back to this panel. The values in the box only represent what you entered here last and do not necessarily corresponds to the true current state.