USER_DISK_SLC:[TEG.LCLSDOC]MOREBPM.STUFF 14 Sep 2004 rev. 04 May 2005 Very rough outline of BPM data acquisition transactions mainly from micro's point of view: Operator selects some one of many measdef buttons (or does a "Create meas def" to modify one of the existing measdef buttons) on one of SCP's several BPM panels. This action implicitly selects a unit of primary BPMD (usually) and a unit of primary DGRP, the latter of which in turn defines some subset of SLC micros, which you can see by the "DBDUMP" button on the BPM calibration panel, and which the operator can further restrict. To actually perform the acquisition (or calibration), the operator eventually pokes "Start-Stop data" or "One-shot data", or (on BPM calibration panel) "Normal Calib BPM" (with possible variations), or fills in unit boxes on buffered data acquisition panel and pokes "Take buffer data". Less visible possibilities also exist. In any of these possible cases, a unique measdef id number applies, keyed to that particular SCP and to one of buttons along bottom of first-level BPM measurement panel or to one of possible slots that are internally represented. This measdef id defines an 8-bit (actually more restricted than just that) YY value for use by MPG and micros concerned. Term "YY" has lots of quite different meanings, and this particular meaning does not primarily have to do with timing values but just with unique measdef identification. The SCP sends a "prep" request message to each micro in given set of micros, carrying for each micro all the parameters defining which devices are involved in the hoped-for acquisition or calibration and how they are to be programmed (hence contents of "prep" request message are generally different for each micro). "Prep" request message also specifies scan options if any, and number of measurements to be averaged, and base beamcode and "inclusion/exclusion" bitmasks for specifying subset of possible beam pulses on which to acquire data. SCP then waits for "prep success/fail" reply message from each micro. Actual acquisition sequence (or calibration sequence) cant start until after SCP has received or timed out all expected "prep success/fail" reply messages. Upon receiving "prep" request message, micro validates parameters and creates a number of structures which will guide the given acquisition or calibration. At any given time the micro can hold several created measdefs (i.e., several of these created sets of guiding structures), even as many as 20, though it rarely gets that bad. In any micro, generally at most one measdef can be active at one time, though there can be exceptions. Certainly no more than one buffer ring can be serviced on any one 120th of a second. Device calibration generally excludes any other activity. A calibration yields for example relative gain ratios for the three channels of a linac-type BPMP, or measured gain of a toroid charge monitor, or pedestals of a gated ADC. For some BPM devices, calibration requires actual beam, and for some other devices (some of which have internal pulse generators which must be programmed) calibration requires absence of actual beam. The results of a calibration are sent back to the requesting SCP (which stores them as files in a SCP-independent subdirectory tree) and are held in memory by the micro (in a structure whose beginning is defined by CALCONST_beg_ts in REF_RMX_INCLUDE:BPMCALS.HM). The results of any one (successful) calibration operation are identified by a unique 16-bit calibration id. Any calibration is either public or private, distinguished by range of calibration id. A private calibration is available only to the SCP that created it. Whenever a micro creates a BPM hardware readout buffer ring, the micro verifies that all calibration structures that will be needed by that buffer ring are present, and rejects the creation request if at least one is absent. The SCP can retrieve from stored files and download to any micro whatever calibrations are found to be needed. During the "prep" stage preceding a given acquisition, if at least one micro's immediate success/fail reply to the prep request contains status BPMO_NOCAL, then the SCP knows that that micro lacks at least one of the calibrations required for that acquisition, and the SCP downloads all needed calibrations to those micros that failed in that way, and retries the prep for them. If a calibration being downloaded to a micro already exists in that micro and is in use in that micro, the newer version will replace the old in such a way that ongoing data conversion for feeding fast feedback will pick up the new version. Calibrations are identified uniquely by 16-bit calibration id. Apart from requesting data acquisition to be done for itself, a SCP can send a "buffer ring creation" request message (generally when the SCP is booting up a micro), to set up free-running BPM data acquisition to drive fast feedback loops (see procedure Measbufringcre in source file MEASPREP.C86). In the micro, any measdef's created set of structures consists of a buffer ring descriptor (BUFRING_DESC_ts and its RINGBUILDPARMS_ts), a sampling request descriptor (SAMPREQ_DESC_ts), an "M1" buffer (for which no defined structure type exists), and as many "M2" buffers (in which only the beginning part has a defined structure type) as needed for this complete acquisition, and possibly a scan control buffer (whose beginning is defined by SCANBUF_beg_ts). These structure types are defined in REF_RMX_INCLUDE:BPMMBUFS.HM. Each "M2" buffer represents one complete (damping-ring-type) multiplexer cycle, and contains camac "stat/dat's" (each "stat/dat" = status longword & data string read or stored by MBCD for one MBCD command packet) and MBCD command packets needed for completing that multiplexer cycle. Any created BUFRING_DESC_ts is either public or private (i.e., either sharable or non-sharable). A public buffer ring is one sharable by fast feedback loop "connections". If a "prep" request's parameters agree sufficiently with an existing public buffer ring then that acquisition will parasite off that public buffer ring; otherwise (e.g., for a very large scan-type acquisition, or for an acquisition that includes at least one device absent from any candidate public buffer ring) a private buffer ring will be created, and that acquisition may shut out fast feedback for all or part of its duration. Note that each step of a scan may include beam-synchronized camac commands to move wire scanner carriage or to move magnets. For any measdef, all these structures (each of which is an allocated segment in the iRMX sense) generally remain sitting in the micro (possibly across many complete acquisitions by the SCP for that measdef) until the SCP tells the micro to discard that measdef or all measdefs for the given SCP. All structures described here serve the purpose of making the 360-Hz interrupt-driven code as efficient as possible. It seems to me there's no reason why the IOC has to closely imitate these internal structures, but I mention them merely as help in digging through existing iRMXmicro BPM code. If at least one micro sends successful prep reply, SCP sends a "YY prep" request message to the MPG, and waits for a "YY reply" message from the MPG and a "data reply" message from each successfully prepp'ed micro. Requesting SCP is unable to do anything until after all these replies comeback or time out. MPG "YY prep" request message includes of course this YY value, and a bitmask identifying which micros are involved in this acquisition, and all needed parameters by which the MPG will be able to know when the micros should have completed this acquisition, i.e., parameters by which the MPG will be able to blindly track the acquisition. MPG enqueues this "YY prep" request with others (from other SCP's) currently waiting or active. MPG will start the given YY request only after (as far as MPG knows) no micro specified in given YY request has any already active measdef. "YY prep" request message includes same base beamcode and inclusion/exclusion masks as prep request messages sent to micros. To enable the involved micros to start the given acquisition, the MPG sends given YY code in a dedicated 8-bit field in the 360-Hz PNET broadcast message for one beampulse. Thereafter, each involved micro and the MPG independently tests each PNET broadcast message (actually a subset thereof) to determine on each beampulse whether to acquire data on that beampulse. Base beamcode (in a dedicated 5-bit field in the 360-Hz PNET broadcast message) is compared (actually on just the appropriate one of the three 120-Hz subsets of 360 Hz), and if that matches, then masks are tested: inclusion/exclusion masks correspond bit-for-bit with bits 32-127 of the 360-Hz PNET broadcast message, and "1"-bits in the inclusion mask indicate required "1"-bits in the PNET message, and "1"-bits in the exclusion mask indicate required "0"-bits in the PNET message. Each micro knows independently when the entire sequence of that micro's camac device operations for the given acquisition or calibration is complete, and at that time (known as "trickle-down" time) the micro allocates & fills a buffer containing the data reply message and sends it to the requesting SCP (possibly in pieces). From the micro's internal point of view, any complete BPM data acquisition consists of a triply nested loop: BPM multiplexing within data averaging within scan steps. For non-scan-type requests (prep request message function code BPMO_MEAS_PREP as opposed to BPMO_WIRE_PREP), the outermost loop in this nest of loops merely has an iteration count of 1. For scan-type requests, severest limitation on length of scan (number of scan steps times number of measurements to be averaged for each scan step) is determined by availability of memory in the micro. For scans done within PEP2 RINQ BPM processors (acquiring per-ring-turn data), the limitation is availability of dual-port ram in the RINQ BPM processor. Though one could consider the SCP as having relatively infinite space for accumulating data, successive acquisition requests (by the SCP through the MPG) almost always will leave some per-beam-pulse data unacquired between requests. For retrieving PEP2 RINQ beamabort data, no MPG transaction is needed, since MPG played its part when the beamabort if any actually happened, and generally the beamabort prep is part of booting the micro, and LCLS wont have a stored beam anyway. CMS subdirectory given by REF_RMX_BPM: contains iRMXmicro BPM-related source files. A good place to hunt for specific function codes is BPMODRVR.C86. Complete list of BPM function codes is of course in REF_C_INC:BPMFUNC.HC. Here are some of those function codes: BPMO_BUFRING_CRE Create sharable BPM hardware readout buffer ring for feeding fast feedback. BPMO_MEAS_PREP Non-scan-type measurement prep. BPMO_WIRE_PREP Scan-type measurement prep. BPMO_CALB_PREP BPMP calibration prep. BPMO_CALTOR_PREP TORO calibration prep. BPMO_CAL_RESTORE SCP is sending this calibration data for micro to hold (SCP expects status-only reply). BPMO_ACCEPT_PUBLIC Relabel private calibration to public (SCP expects status-only reply). BPMO_SHO_RBUFS Return condensed summary of existing buffer rings for diagnostic display. BPMO_RINQABO_MEAS Retrieve RINQ data from most recent PEP2 beamabort in non-scan format. BPMO_RINQABO_SCAN Retrieve RINQ data from most recent PEP2 beamabort in scan format. BPMO_TIMING_ADJ Update given hardware timing delay values in running acquisitions. BPMO_GETREMDAT SCP is asking micro to send next piece of data reply that was too large for message service to handle as one message. BPMO_MEAS_STOP Delete specified measprep (SCP expects status-only reply). BPMO_CAN_DEF Delete specified calibration (SCP expects status-only reply). BPMO_CLR_SCP Delete all non-public structures built for this SCP (SCP expects no reply). Note that each _PROC function code (in BPMFUNC.HC) is always precisely one more than its corresponding _PREP function code. The micro uses the _PROC function codes to send the data replies after the entire requested acquisition is completed. The immediate success/fail replies (indicating whether the micro thinks the prep itself seems ok) are sent using the same _PREP function as the SCP's request (but of course with the 0x8000 flag set to indicate reply). In subdirectory given by REF_RMX_BPM:, main source files & procedure therein are the following: MEASPREP.C86 and CALBPREP.C86 (more specifically procedures Measprep1, Calblinprep1, Calbpulprep1, and Calbtorprep1) receive "prep" request messages for measurement and calibration respectively, and call procedure Bufringbuild in BUFRINGBUILD.C86 to set up each. To see how incoming request messages are distributed according to function code, look at procedure BPMO_drvr in source file BPMODRVR.C86. BUFRINGBUILD.C86 is sort of the central core of the whole BPM subjob. BPMNMI.C86 contains nearly all 360-Hz interrupt-driven code. BPMFDBK.C86 contains all code for creating, deleting, and driving fast feedback loop "connections". MEASPROC.C86 and CALBPROC.C86 (more specifically procedures Measproc1, Calblinproc1, Calbpulproc1, and Calbtorproc1) receive "trickle-down" events signaled to task Bpmproc_task by 360-Hz interrupt-driven code for SCP-requested beam measurement and device calibration respectively, and they eat through the chain of completed "M2" buffers for the given acquisition, convert the data (generally using conversion constants from the database), and package the results up into a data reply message and send it to the requesting SCP. Just before sending the completed data reply message, Measproc1 again makes the given buffer ring visible to 360-Hz interrupt-driven code, which thereby is made responsive to next possible instance of that measdef's "enable" signal from the MPG. To repeat the same acquisition, the SCP merely sends the same YY request message to the MPG and waits for its reply message and the data reply messages. In source file CALBRSTR.C86, procedure Calbrstr1 receives from the SCP the contents of one or another previously obtained calibration (uniquely identified by 16-bit id), and stores it for possible use at buffer ring creation time or at data reduction time. When reading the iRMXmicro BPM code one must keep in mind the distinction between 360-Hz interrupt-driven procedures and ordinary task-scheduler-driven procedures. Communication between the two kinds of procedures depends on usage by iC386 compiler of certain machine-language instructions. Generally a structure created by task-level code is made visible to interrupt-driven code by means of storing a pointer somewhere (thus making that pointer be nonzero), and this must be done (ultimately) by a single uninterruptible instruction. In source file BPMNMI.C86, procedure Bpmnmi_meas_360 drives measurement, and procedures Bpmnmi_calbnu_360, Bpmnmi_calbpu_360, and Bpmnmi_calbtor_360 drive calibration. Note that each of these is called at 360 Hz (as long as connected by procedure NMIsetcall). Switch-on-case statements in each serve to outline the time stages on which the relevant camac operations must be done. Public array Bufrings_by_PP contains selectors of (abbreviated pointers to) BUFRING_DESC_ts's, indexed by base beamcode. Public array Sampreqs_by_YY1 contains selectors of SAMPREQ_DESC_ts's, indexed by YY-enable-code-from-MPG. These arrays in particular help the 360-Hz interrupt-driven code decide as efficiently as possible what to do next. The whole idea is to optimize the 360-Hz interrupt-driven code, which behaves kind of like a state machine governed by a plurality of state variables rather than by just one. When testing whether to start the (next) step for some acquisition, Bpmnmi_meas_360 calls find_bufring_for_YY1, and the latter procedure might be a good place to look to see how the 360-Hz broadcast governs BPM data acquisition. Procedure Measprep1 receives from any SCP a request message which is either a bpm_measprep_reqh_ts or a bpm_scanprep_reqh_ts (the latter is an open-ended extension of the former; both are defined in REF_C_INC:BPM_REQREPLY_STRUC.HC). Extension bpm_scanprep_exten_ts is present iff function code (in msgheader_ts) is BPMO_WIRE_PREP; member nrpos specifies number of scan steps and hence tells actual size of array magdac (which is present iff at least one element of magctlw specifies nonzero crate address). This message may be sent in pieces (each piece with its own msgheader_ts followed by bpm_largreq_hdrr_ts) if total message is longer than message service's maximum. Here are a few members of structure bpm_measprep_req_ts: i_seg is measdef id (unique across all SCP's and all measdef's and all calibrations that may be in progress or pending). nicals is number of calibrations that will be needed by data reduction for this acquisition. i_cal contains id's of those calibrations. npp is number of base beamcodes to which 360-Hz interrupt-driven code should be sensitive for driving this acquisition. pp contains those base beamcodes. navg is number of measurements (beam pulses) to be averaged to obtain each resulting value. yy is value which MPG will put in "YY" dedicated byte in 360-Hz PNET broadcast message to enable (for all micros) the beginning of the (next complete instance of this) acquisition. measbits contains flag bits defined by *_BIT names in REF_C_INC:BPMPARAM.HC. These flag bits indicate which primaries are to be considered (note that "0"-bits in measbits can cause nonzero elements of ulist to be ignored). umin,umax, are inclusive unit number range limits for units of primaries umin2,umax2 BPMS, TORO, ARRY, WIRE, etc. for which data is to be acquired. Unless overridden by ulist, a unit will be included only if its unit number is within either inclusive range. ulist overrides umin,umax,umin2,umax2 for any primary whose element of nunits is nonzero (sequence of primary names is given by MEAS*_INDEX names defined in BPMPARAM.HC). beamstrength gives expected beam strength for choosing attenuation setting to be programmed into BPM devices for each beam pulse. exclmask specifies bits in PNET broadcast message which must be 0 to allow acquisition on the given beam pulse. inclmask_strt specifies bits in PNET broadcast message which must be 1 to allow acquisition on the given beam pulse (applies only on 1st step of BPM multiplexer sequence). inclmask_nxt same (applies only on steps after 1st step of BPM multiplexer sequence). Procedure Calblinprep1 sets up the calibration sequence for certain BPMP device types, some (but not all) of which lack calibration pulsers and hence require actual beam for calibration. Calblinprep1 receives from any SCP a bpm_calbprep_reqh_ts request message. Here are a few members of that message structure: i_seg is measdef id (unique across all SCP's and all measdef's and all calibrations that may be in progress or pending). i_cal is id of this private calibration (unique across all SCP's and all sets of calibration results). npp is number of base beamcodes to which 360-Hz interrupt-driven code should be sensitive for driving this calibration. pp contains those base beamcodes. navg is number of raw data values (beam pulses) to be averaged to obtain each resulting value. yy is value which MPG will put in "YY" dedicated byte in 360-Hz PNET broadcast message to enable the beginning of this calibration sequence. calbbits contains flag bits defined by *_BIT names in REF_C_INC:BPMPARAM.HC. umin,umax, are similar to above. umin2,umax2 beamstrength gives expected beam strength for choosing attenuation setting to be programmed into BPM devices during calibration. exclmask, are similar to above. inclmask Procedure Calbpulprep1 sets up the calibration sequence for other BPMP types, generally device types for which the calibration sequence does not have to be quite as strictly beam-synchronized. Calbpulprep1 is called upon completion of the calibration sequence set up by Calblinprep1, and obtains parameters from structures built by Calblinprep1. Procedure Calbtorprep1 receives from any SCP a request message of structure bpm_calbtorprep_reqh_ts, and sets up calibration sequence for toroid devices. That message structure is a bpm_calbprep_reqh_ts with an extension which is a bpm_calbtorprep_exten_ts. Here are a few members of the latter: npulvals is number of elements in array pulvals. pulvals is array of amplitude values to be programmed into pulser devices. nunits, are analogous to same-named members of structure ulist bpm_measulist_ts, and apply to primary TORO only. As with beam measurement "prep success/fail" reply messages, calibration "prep success/fail" reply messages tell the lengths which the SCP should expect of data reply messages (for allocating memory before those messages are received). Procedure Bpm_accept_public (in source file REF_RMX_BPM:CALBPUBLIC.C86) receives from any SCP a request message of structure bpm_calb_acc_pub_reqh_ts, specifying id of private calibration which is to be re-identified with given public id, implying deletion of any previous instance of latter. Generally a BPMP (processor = digitizer module) returns three measured values: X, Y, and tmit. Instances of primary BPMP define the digitizer modules; instances of primary BPMS define the "stations" in the beam pipe where beampulse-sensing electrodes are located. At least in the damping rings, more than one BPMS can be wired to one BPMP, with the signals going through single-pole-ten-throw multiplexer modules which themselves also have to be programmed on each beampulse to be measured. Some BPMS's are X-only or Y-only. Sometimes for a given BPMS no tmit value is returned. The software recognizes many different BPMP hardware modules types. The SCP generally displays data (beam measurement or digitizer calibration) from different BPMS/BPMP types in different formats. Beam measurement data reply messages have two different formats: for non-scan prep's the data is packed without any interspersed self-identification; for scan prep's the data includes prefixes specifying primary and unit number; in latter format you dont have to know anything about what you're not looking for. As mentioned above, the immediate success/fail reply to a prep specifies length of subsequent data reply messages for that prep. Calibration data is returned only in non-self-identified format (though different from non-self-identified measurement data reply message format). Non-scan data reply message begins with bpm_alldata_replybegh_ts (preceded of course by fwd_hdr_ts); data structures follow immediately. Each unit's data structure contains a 16-bit status word indicating online/offline status, and succes/failure of the measurement from the hardware if online. For each requested unit of BPMS (including offline units) there is a bpms_data_ts or a rinq_sine_data_ts (defined in REF_C_INC:BPMSTRUC.HC); these are in database BPMS unit order. Then, for each requested unit of TORO there is a toro_data_ts; these are in database TORO unit order. Then, for each requested unit of KLYS there is a klys_data_ts, in database unit order. Then, for each requested unit of GAPM, SLIT, WVFM, ARRY, WIRE there is a similarly-named *_data_ts, in database order within each DUGADC primary. Then, for each requested unit of FIFO there is a fifo_data_ts, in database FIFO unit order. Then, for each requested unit of SBST there is a sbst_data_ts, in database unit order. Then, if list of beam pulse id's is requested, there is a bpm_pulseid_data_ts whose array named pulseid contains navg * nmuxmin longwords. Then, if PTGM data is requested, there is a ws_ptgm_data_ts whose array named item contains navg * nmuxmin ws_ptgm_item_ts's. See procedure Meas_compute_replysize in source file MEASPROC.C86; this procedure effectively determines format of beam measurement data reply message. Scan data reply message begins with bpm_alldata_replybegh_ts (preceded of course by fwd_hdr_ts); the first scanblk_prefix_ts (defined in BPM_REQREPLY_STRUC.HC) follows immediately; each scanblk_prefix_ts's member wc = wordcount of specified unit's data not counting scanblk_prefix_ts itself. If a specific unit of WIRE was specified (in member wun of bpm_scanprep_exten_ts), the first scanblk_prefix_ts is followed by nrpos (see bpm_scanprep_exten_ts) instances of ws_wire_data_ts for that unit. For each online requested unit of BPMS, there is a scanblk_prefix_ts followed by nrpos instances of ws_bpm_data_ts (but slightly different for some modes of acquiring PEP2 RINQ BPM data). Similarly for TORO, KLYS, SBST, FIFO, DUGADC primaries. For each unit of requested beam-synchronized PAU ADC readback data, there is a scanblk_prefix_ts followed by nrpos+1 longwords. If list of beam pulse id's is requested, there is a scanblk_prefix_ts (specifying unit 1 of pseudo-primary "PUID") followed by a ws_puid_ts whose array pulseid contains nrpos * navg * nmuxmin longwords. If beamcode + vetobus data is requested, there is a scanblk_prefix_ts (specifying unit 1 of pseudo-primary "VBUS") followed by a ws_ppyy_veto_ts whose array named item contains nrpos * navg * nmuxmin ws_ppyy_veto_item_ts's. If PTGM data is requested, there is a scanblk_prefix_ts (specifying unit 1 of primary "PTGM") followed by a ws_ptgm_data_ts whose array named item contains nrpos * navg * nmuxmin ws_ptgm_item_ts's. These scanblk_prefix_ts's can of course be in any order. Note that for data reply messages (beam measurement data, device calibration data, beamabort data) large enough to be chopped into several pieces (each with its own bpm_alldata_replybegh_ts preceded of course by fwd_hdr_ts), the SCP must explicitly ask (using function code BPMO_GETREMDAT) for each piece after the first. A data reply message will be chopped into more than one iff its total length (in 16-bit words not counting msgheader_ts or fwd_hdr_ts but counting bpm_alldata_replybeg_ts) is strictly greater than NETMICLONGMSGLEN (defined in REF_C_INC:NETPARM.HC), simply because member datalen in msgheader_ts is only 16 bits unsigned. For new BPM hardware devices created for LCLS, it will be best to define new structures for beam measurement data (in BPMSTRUC.HC) and for device calibration data (in BPMCALSTRUC.HC). Much better than pretending it is "just like" some other device when it isnt quite.