LCLS Controls
SLC Aware IOC
Beam Synchronous Acquisition and
Control (BSAC) Service Design
--- DRAFT --- work in progress
Diane
Fairley and Debbie Rogind
July 12 2005;
Last Modified July
19, 2005
____________________________________________________________
Quick Links
The following document describes the design of the Beam Synchronous Acquisition and Control (BSAC) Service. The document includes a description of the functionality of the BSAC Service, a detailed design of each SLC-aware thread, and the design of the EPICS databases that comprise the BSAC Service.
The BSAC Service of the SLC-Aware IOC performs beam synchronous data acquisition from devices such as beam position monitors, toroids, and gated ADC units. It will also perform beam synchronous control required for the wire scanners (TBD). This service is included in the SLC-aware IOC so that it can provide the beam synchronous data to the SLC Control system high-level applications.
The service is comprised of two SLC-aware threads (the SLC BSAC Facility) and a set of Epics database records (the EPICS BSAC Facility). The Epics database records interface to the hardware device support and control the flow of the data acquisition process. The SLC-aware threads act as a message pass-through to allow the SLC Control System to request beam synchronous measurements. The data can be returned to the SLC control system via the SLC-aware threads, and is also available through EPICS Channel Access to any EPICS client.
The SLC control system at SLAC is
currently used on most of the LINAC. It is the only control system in sectors
20-30, which will be used by the LCLS mostly in tact. LCLS will replace all of
the BPM electronics in these sectors to provide higher resolution. The Injector
for LCLS will use all new control, except for the high power RF components,
which are existing SLC klystrons and modulators. The corrector magnets in the
LINAC that will be used for LCLS will all have new EPICS based controllers.
From the undulator to the experimental stations, all
new controls will be done in EPICS. Note that all SLC data from the existing
LINAC will be available to the EPICS environment, but the time stamp
information that allows data correlation to beam events is not available at the
present.
The motivation to implement an SLC
aware, EPICS IOC, is to allow the new elements of the LCLS control system to
use EPICS, while still taking advantage of the high level applications on the
SLC control systems. These high level applications include:
Correlation plots, energy
management, beam steering, beam based alignment, emittance
measurements, and slow feedback.
Beam Synchronous Acquisition and Control (BSAC) Requirements. By Diane Fairley and Debbie Rogind.
Requirements for High-Level Software Applications Packages by Patrick Krejcik
LCLS Beam Position Measurement System Requirements by Patrick Krejcik
LCLS Timing System Requirements by Patrick Krejcik
Design Specification for the Injector Beam Position Monitors by D. Dowell
Message Service Design by Diane Fairley.
Database Service Design by Debbie Rogind.
See the Beam Synchronous Acquisition and Control (BSAC) Requirements by Diane Fairley and Debbie Rogind.
Also see the SLC-Aware IOC General Functional Requirements by Stephanie Allison.
The BSAC Service is a software facility that runs in the SLC-aware IOC. It includes two parts: the SLC BSAC Facility and the EPICS BSAC Facility.
The SLC BSAC Facility accepts a subset of the SLC Control System request messages with function codes BPMO_* that concern data measurements from BPMs, toroids and other devices. A bsacHdlr thread accepts request messages, passes the request information to the EPICS BSAC Facility, then waits for the data to be measured by the EPICS BSAC Facility and returned. The bsacHdlr thread packages the data into VMS format and returns the data to the SLC Control System.
The SLC BSAC Facility is also responsible for keeping the BSAC related portions of the SLC database and the Epics database synchronized within the SLC-aware IOC. Upon initialization of the SLC BSAC Facility the Epics Database settings are copied to the SLC Database and updated on the SLC Control System. The Epics Database is considered the Master at initialization. From that point (until the next SLC-aware restart) the bsacAsync thread will take the “last database change” made by a SCP, or made by an Epics client, and reflect that change in the other database. In effect the last request wins.
The EPICS BSAC Facility is a set of database records responsible for collecting and processing beam synchronous data according to the request information set up by the SLC BSAC Facility. It can also perform measurements requested by an EPICS client when that client has properly set up the request information records.
The following sections give an overview of the BSAC Service external interfaces. Each external interface is further described in the detailed design sections of this document.
The bsacHdlr thread communicates with the SLC Control system through the Message Service interface. Request messages are received from the Message Service in the bsacHdlr message queue and reply messages are sent to the Message Service to be forwarded to the SLC Control System. See bsacHdlr Detailed Design.
The bsacHdlr and bsacAsync threads utilize the Database Service to access information stored in the SLC Database concerning the acquisition devices. See bsacHdlr Detailed Design and bsacAsync Detailed Design.
Channel Access makes all of the PVs defined in the EPICS BSAC Facility available to EPICS Clients. EPICS Clients may use channel access to initiate a measurement request, and receive the resulting data. Diagnostics may be viewed via channel access as well. See BSAC Epics Database Detailed Design.
The EVR Device support provides the components of the SLC PNET message used to identify which pulses should be collected for each active Measurement Definition. These components include the YY (measurement definition trigger), the PP (beam code) and the modifier bits which indicate when an individual pulse should be included or excluded from the measurement.
The EVR device support also provides the beam synchronous timestamp and SLC pulse ID for beam synchronous measurements. See the BSAC Epics Database Detailed Design.
The BPM Device Support provides the data acquired from the BPMs. See the BSAC Epics Database Detailed Design.
The Toroid Device Support provides the data acquired from the Toroids. See the BSAC Epics Database Detailed Design.
The ARRY Device Support provides the data acquired from the GADC module. See the BSAC Epics Database Detailed Design.
The WIRE Device Support is TBD
See bsacHdlr Detailed Design for a detailed description.
Data for all units of all devices involved in any one measurement (definition) must be collected and processed (averaged and rms values) within 120Hz. Time critical functionality is implemented with EPICS database records. Subroutine records were chosen for speed optimization because they do not cause task switches.
Pulse Id data is collected for every measurement definition, whether or not it was requested. The EVR Device Support must process first to guarantee that the Event Buffer (EB) pattern (the YY, PP, and inclusion / exclusion masks) are available, and that the Pulse Id collection stream is the first processing stream to be initiated. This processing stream includes the CHECKEVR subroutine, which evaluates whether a pulse should be collected for this measurement definition, based on the PNET pattern. All other unit processing streams use the result of this CHECKEVR subroutine as an indicator as to whether the data from that unit should be included in the measurement.
For Pulse Id collection within one measurement definition, one pulse id is collected every navg cycle and sent to the bsacHdlr. The bsacHdlr will return this data to the SLC Control System only if required. The EPICS BSAC Pulse Id collection has the following data flow:
The last subroutine record, PUIDCNT, sends a message to bsacHdlr queue with the pulse id every cycle. bsacHdlr, running at a lower priority, converts and packs the pulse id data into the appropriate place in the data reply message buffer, if the pulse id was requested in the measurement definition.
The general flow of processing for each unit of each device within a measurement definition is similar to the pulse id collection, but includes an averaging loop processed by an additional AVGCNT subroutine record. This AVGCNT subroutine uses the result of the pulse id CHECKEVR subroutine to decide whether to acquire a pulse.
The general unit collection of the EPICS BSAC Facility has the following data flow:
The last subroutine record, MEASCNT$MD, sends a message to bsacHdlr queue with the appropriate data for each unit of each device after navg pulses. bsacHdlr, running at a lower priority, converts and packs the data into the appropriate place in the data reply message buffer.
See bsacHdlr Detailed Design for a detailed description.
See bsacAsync Detailed Design for a detailed description.
The following global data structures are used by the BASC Service. See bsacHdlr Detailed Design for a detailed description of how each is used.
The following structures store the following information about each unit of each device (primary) that can be used to acquire beam synchronous data.
unitStruct_ts structure with:
bsacUnits_ts:
primStruct_ts
bsacPrim_ts:
This structure stores the following information about each MeasDef:
Small and large message buffers are used by the BSAC Service to handle all message traffic. See Message Service Design for a description of these message buffer pools.
Upon start up the bsacHdlr must:
The bsacHdlr thread is a message queue driven thread. It accepts incoming message commands and acts on them. The basic, high level loop is:
· While (slcThreads_as[bsacHdlr].stop !=epicsTrue)
o Read message from queue using the msgQMsgRecv utility
§ Get data word length and function code
o Perform the function based on function code
§ If not a valid function code, drop the message and log and error
o For messages received from the SCP:
§ Allocate buffer for immediate reply message
§ Create reply data, convert for VMS
§ Release the incoming message buffer using msgQFreeBuffer
§ Create large buffer for data reply message
§ Set up pointers and counters into data reply message for participating units
§ dbPut operations to EPICS BSAC Facility records
§ Send immediate reply to msgSend thread using the msgQMsgReply utility.
· (msgheader is in native format)
o For messages received from EPICS BSAC Facility:
§ Identify the prim/unit/md this message came from
§ Do nothing if abort flag for this md is set
§ Get the associated pointer into the large reply buffer(s).
§ Convert and pack the data structure into its place in the large reply buffer(s).
§ Update pointer; increment unit data counter and master mdef counter
§ When done collecting, error check and send first large reply buffer to the msgSend thread
§ When done sending, zero out data collection linked list pointers and counters
The bsacHdlr thread performs the functions related to the following messages received from the SLC Control System, from the slcExec thread, and from within the threads of the BSAC Service itself.
The BPMO_MEAS/WIRE_PREP request initiates the following sequence:
· Assign the measurement definition (MeasDef) number [$(MD) in Epics Records] for this request
· Calculate expected PACKED data size for each requested unit,
· Send BPMO_MEAS/WIRE_PROC immediate reply
· Allocate message buffer(s) where acquired data will be stored
· Set up pointers into message buffers at location of each unit’s data within the buffer in data collection linked list
· dbPut BPMO_MEAS/WIRE_PREP parameters into Epics BSAC Facility MDEF records (except YY)
· dbPut to change Epics BSAC Facility Go records to SCAN=IO_INTR, for each unit involved in this MeasDef.
· dbPut Epics BSAC Facility MDEF:$(LOCA):$(MD):YY for this MeasDef.
· Populate bsacMD timeout variable for bsacAsync
The BSAC_DATA message is sent from the Epics BSAC MEASCNT subroutine and initiates the following sequence:
The BSAC_DONE message is sent from the Epics BSAC MEASCNT subroutine and initiates the following sequence:
If all units have reported data nrpos cycles, then initiate sending data to SCP
TBD
The BPMO_GETREMDAT message initiates the following sequence:
The BPMO_MEAS_STOP message initiates the following sequence:
The BPMO_CLR_SCP message initiates the following sequence:
The BPMO_SHO_RBUF message initiates the following sequence:
The BSAC_ABORT message is sent by the bsacAsync thread when it detects a timeout on a MeasDef. The BSAC_ABORT message initiates the following sequence:
Note: if abort flag is already set in the bsacMD_as structure, incoming data received from the EPICS facility would have been “thrown away” by the bsacHdlr. It would have also reset the MEASCNTMAX record to 0.
The slcExec threads
sends this message to
There is a Measdef abort functionality built into the processing flow of the Epics database records. If, on a currently active Measdef, the MDEF:$(LOCA):$(MD):MEASCNTMAX is set to 0 then the processing will stop on the next MEASCNT loop.
There are four cases where a measurement may be aborted:
Thread termination is initiated by the slcExec thread setting the slcThreads_as[bsacHdlr].stop flag to TRUE, or when the bsacHdlr thread receives a IOC_STOP message.
When the slcThreads_as[bsacHdlr].stop flag == epicsTrue, or IOC_STOP is received:
· Abort any active MeasDefs – send replies indicating abort was done.
· Set slcThreads_as[bsacHdlr].active = epicsFalse
· Release any large message buffers held for currently active measurement definitions using the msqQFreeBuffer utility.
· Destroy the message queue using the messageQRelease utility. This releases any message buffer held in the message queue first.
· Set slcThreads_as[bsacHdlr].tid_ps = NO_THREAD
· Return
The bsacHdlr thread uses the following globals:
The bsacMD_as and bsac collection arrays are initialized by the bsacAsync thread and used by both BSAC threads.
The bsacHdlr receives its messages in a message queue. The queue is initialized at startup, and, when terminating, the bsacHdlr uses the msgQRelease utility to release all pending message buffers and destroy the queue.
The bsacHdlr thread uses both small and large message pool buffers. In particular, the data replies to a measurement request can use several large message buffers. The bsacHdlr ensures that every message buffer it uses is released back to the pool when no longer needed, or when the thread is terminated.
Error messages are logged when:
Test signal generators are developed as needed for testing data acquisition before real devices are available. These Epics databases
Iocsh routines to dump bsacMD_as, bsacUnits_as, other interesting structures TBD
TBD
The BSAC Epics Database is responsible for the acquisition and processing of beam synchronous data from each of the units included in this IOC. The actual database records, as shown with the tool VDCT are used to describe the detailed design of the Epics Database.
Please refer to diagrams in Epics Record Processing above.
Status and Calibration State Records (BSAC_States.db) include calibration information and online/offline/maintenance/checkout states. These records may be used for all device types, including BPMs, Toroids, ARRY, WIRE.
Offset and Tolerance records for BPMS (BSAC_BPMConsts.db). Usage of these records are TBD
The records are initialized upon start up and kept in synch by the bsacAsync thread.
MeasDef initialization (BSAC_MDEF.db) (two pages):
The MDEF records are initialized by the bsacHdlr thread for each new BPMO_MEAS_PREP or BPMO_WIRE_PREP message.
NOTE: The maximum value allowed for AVGCNTMAX is 100, for MEASCNTMAX is 2800.
A unit belonging to a particular measdef is “enabled” by having its $DEV:GO$MDID bo record’s SCAN field set to IO_Intr. The EPICS Device Support routines issue a scan IO request (This is better than the alternative of fanout records and having to process disabled records; it also serves to decouple the device support effectively from the BSAC databases.) which causes records with SCAN = IO_Intr to process.
BSAC_Go.db:
The PUID:$LOCA:$MD:GO FLNKs to PUID$LOCA:$MD:CHECKEVR subroutine, as illustrated below in the BSAC_CHECKEVR.db file. This subroutine compares the PNET fields of YY, PP and modifier bits to the measdef fields of YY, PP and inclusion/exclusion masks. It also accommodates two extra modifier longwords (MODIFIER4 and 5) received by the EVR to be compared with the MDEF INCLUSION4 and 5 and EXCLUSION 4 and 5.
The CHECKEVR processing is performed at 360 Hz; it is expected that the PP match fails for at least 2/3rd of the counts, as the actual pulses for the LCLS project will run at no more than 120Hz.
Pulse id collection is accomplished by $PUID$LOCA:$MD:PUIDCNT in file, BSAC_Pulseid.db. This record counts PUIDCNT.VAL = AVGCNTMAX * MEASCNTMAX number of pulses and feedsback to CHECKEVR when to set the trigger back to 0 (which is after VAL number of pulses). It sends a message to the slc threads, if there, with every pulse id.
BSAC_Pulseid.db:
Each unit of every device processes after PUID:$LOCA:$MD:CheckEVR and PUID:$LOCA:$MD:PUIDCNT, above, have processed. The processing chain for each unit is initiated with the GO$MDID record forward linking to $(DEV):AVGCNT$MDID.
The $DEV:FANOUT$MDID record, above, forward links to each secondary that is defined for a particular unit, causing it to process $DEV:$SECN$MDID, below. This record maintains a running average and standard deviation (RMS) value for its secondary. These are the actual values that are reported to the slc threads after (each) AVGCNTMAX has been reached.
BSAC_ProcAvg.db:
Finally, the data for each unit is output via $DEV:MEASCNT:$MDID, at the end of every navg (AVGCNTMAX) cycles to the slc threads via message queue, with message function code BSAC_DATA, and the data from $DEV:$SECN$MDID as well as $DEV:AVGCNT$MDID (stat). After nrpos (MEASCNTMAX) cycles, when the measDef has completed, the message function becomes BSAC_DONE instead, and the data follows.
File BSAC_MeasCnt.db:
The acquired data is provided by epics device support tasks. These tasks must run at a high priority.
The EVR Device Support is expected to provide data for the following records:
YY, PP,
MODIFIER1, MODIFIER2, MODIFIER3, TIMESTAMP, PULSEID
The EVR Device Support is expected to maintain a buffer of these measurements for each of the 3 previous pulses. A history of 3 must be available to the BSAC Service.
The EVR device support is expected to do a ScanIORequest on the PUID:$(LOCA)$(MD):GO records and interrupt those that have a SCAN = IO_Intr in order to initiate acquisition of pulse IDs for the referenced measDef.
The data records (YY, PP, MODIFIER1, MODIFIER2, MODIFIER3, TIMESTAMP, PULSEID)
must be processed first, so that the data is ready before the ScanIORequest.
The EVR device support must process its data records and do a ScanIORequest to begin Pulse Id processing before all other device support routines process their data records (Primary records) or perform their ScanIORequests. All other unit processing depends on the results of the CHECKEVR subroutine of the Pulse ID processing stream.
The BPM Device Support is expected to provide data for the following secondary (SECN) records:
X, Y, TMIT
The BPM device support is expected to do a ScanIORequest on the BPMS:$(LOCA)$(UNIT):GO$(MD) records and interrupt those that have a SCAN = IO_Intr in order to initiate acquisition of the BPMS secondaries for the current pulse.
The data records must be processed (so that the data is ready) before the ScanIORequest.
The Toroid Device Support is expected to provide data for the following SECN records:
TMIT
The Toroid device support is
expected to do a ScanIORequest on the TORO:$(LOCA)$(UNIT):GO$(MD) records and interrupt those that
have a SCAN = IO_Intr in order to initiate
acquisition of the TORO secondary for the current pulse.
The TMIT record must be processed (so that the data is ready) before the ScanIORequest.
The ARRY Device Support is expected to provide data for the following SECN records:
DATA
The ARRY device support is expected to do a ScanIORequest on the ARRY:$(LOCA)$(UNIT):GO$(MD) records and interrupt those that have a SCAN = IO_Intr in order to initiate acquisition of the ARRY secondary for the current pulse.
The DATA record must be processed (so that the data is ready) before the ScanIORequest.
TBD
Epics database records do not terminate unless the entire IOC is shut down.
There is a Measdef abort functionality built into the processing flow of the Epics database records. If, on a currently active Measdef, the MDEF:$(LOCA):$(MD):MEASCNTMAX is set to 0 then the processing will stop on the next MEASCNT loop.
There are four cases where a measurement may be aborted:
The Epics MEASCNT subroutine uses the following globals:
It is a goal to avoid any message logging from EPICS subroutine records.
All PVs visible through Channel Access
Some specific Diagnostic capabilities so far:
Refer to EPICS Database Detailed Design, Normal Processing, BSAC_CHECKEVR.db for logic details.
Refer to EPICS Database Detailed Design, Normal Processing, BSAC_Process.db for logic details.
Refer to EPICS Database Detailed Design, Normal Processing, BSAC_Process.db for logic details.
Refer to EPICS Database Detailed Design, Normal Processing, BSAC_ProcAvg.db for logic details.
Refer to EPICS Database Detailed Design, Normal Processing, BSAC_Pulseid.db for logic details.
Upon start up the bsacAsync must:
NOTE: There is no checking done at initialization time to assure that the SLC Database and the Epics Databases are consistent as to the types of devices and number of units controlled by a particular slcIOC. Measurement Requests will be returned with error at the time of request if the databases do not match.
While slcThreads_as[bsacAsync].stop!=epicsTrue.
·
Compare HSTAs and
o If changed, update the local area and Epics record
· Compare device offsets, tolerances in SLC DB and local area (TBD)
· For all active MeasDefs, decrement the timeout counter
o If the counter=0 send a BSAC_ABORT message to the bsacHdlr
· Call epicsThreadSleep(1) to sleep for one second.
NOTE: The Channel Access monitors will asynchronously update the SLC Database and the local area with Epics DB values any time those values are changed.
NOTE: the database updates may be done less frequently than once per second TBD. In that case a DBUpdate counter will be kept and SLC-> Epics updates only performed when this counter reaches it’s limit.
Termination is initiated by the slcExec thread setting the slcThreads_as[bsacAsync].stop flag to TRUE.
When the slcThreads_as[bsacAsync].stop flag == epicsTrue:
· Clear all Channel Access monitors
· Release all Channel Access channels
· Set slcThreads_as[bsacAsync].active = epicsFalse
· Set slcThreads_as[bsacAsync].tid_ps = NO_THREAD
· Return
The bsacAsync thread uses the following globals:
TBD
TBD
TBD