SLC CORRELATION PLOTS DESIGN IN OUTLINE Gregory R. White, 12-Mar-1995 SLAC 1. INTRODUCTION AND OVERVIEW "Correlation Plots" is an SLC control system application program for performing accelerator physics and diagnostics experiments that involve the simple statistical correlation of a small number of independent variables, with some fairly large number of dependents (<=160). As implemented, Correlation Plots is mostly in VAX Fortran. It runs exclusively on the central VAX/VMS "mainframe" that performs the high level control tasks for the SLAC Control Program (SCP). At its lower levels, the package is arranged as a suite of general purpose routines for such things as setting up the step and sampled variables in a consistent way, making an acquisition, selecting a plot and fit, drawing the plot etc. A good deal of this is the mechanism for preparing to take an acquisition, such as setting up database query lists, priming timing devices, setting up CAMAC packages etc. A higher layer manages an interactive user session, via GUI panel buttons, to use these routines. In Correlation Plots, a higher layer still provides an API to applications for doing automated correlations such as computing the (1,6) element of the transport matrix of some sector of the machine, the energy spread, or similar. The API subsumes the interactive user interface layer because applications may themselves be interactive. 2. FUNCTIONAL DESCRIPTION OF FACILITIES This section summarizes the more important facilities of the Correlation Plots application from the perspective of users. Touch Panel Interface Users interact with Correlation Plots as an on-line user real-time application, through a touch panel interface. A broadly based generic interface is provided for ad-hoc experiments and for experimental setups that are loaded through configuration files. In addition, the "Correlation Plots Applications" - client programs of the Correlation Plots API - may also have their own touch panel interfaces. The generic touch panel interface consists of buttons for setting up an acquisition, starting it, pausing it, terminating it, displaying data, fitting data etc. In general, there is one GUI panel tree for doing an acquisition and a second for examining the data collected in it. Supplementary panels help to save the data in a number of file formats, control special displays, specify the settle time for magnets, and other options. Assigning a "device name" to a variable (step or sampled) A significant proportion of the code in Correlation Plots defines the subsystem that helps a user with selecting and entering a device name. This help is based on the naming convention used to access the SLC database. The SLC db mirrors almost all of the values in the SLC control system. The db is "hierarchical", with the very simple schema of: "device-name" "device-property", where device-name has the syntax : "type" "region" indexnumber and device-property is an adjective or singular noun, like 'X'. All 4 domain names must be representable in a longword. For instance, the x plane reading of a BPM at the beginning of the linac is BPMS LI00 43 X. Any one of these domains may be given as a wildcard, in which case Correlation Plots will offer valid instantiations. Also having entered some valid devices, the user may enter only the fields to be changed to enter more devices. Furthermore, the user interface recognizes some devices by their "common name". One very important feature of Correlation Plots though, is that its input variables are specifically not limited to the variables in the control system database. Of the 30 or so kinds of variable acquireable by Correlation Plots, only one is the "raw" database device name kind. The others are kinds of variable specially programmed into Correlation Plots for doing more sophisticated sorts of acquisition or stepping, like doing a wire scan, or controling the master oscillator. This facility is very general and reflects strongly the organic development of the SLC control system. Specific Important Functions AP_PARSE() manages the validation of input variables and helps in choosing valid names. AP_PARSE( previously input valid device name, new partial device name) Given that SLC variables have a number of fields, AP_PARSE() attempts to superimpose a partially instantiated variable name onto a valid name, replacing the uninstantiated fields of the partial name with the fields given in the complete name. So, to some extent it guesses the fields it's been given, replaces those fields, and sees if the result is valid. In this way the user can enter only one or two fields of a device and AP_PARSE() will try to fill in the rest. If no match is possible, AP_PARSE() will list the valid instantiations of the field following the most significant validly instantiated field. AP_PARSE() calls AP_VALID_( device_name_token ). There is one such function for each device type, which checks whether a given token is a valid field value for that particular kind of device. Entering a step variable One or two step variables are permitted (in which case the second is nested within the first). A step variable may be a "device" name or a "multi-knob file". A multiknob file contains a list of 1 or more devices and the sensitivity of each to a knob turn unit, so that a number of devices' values can be controlled from a single knob. In this case, one of the step variables varies the values of a number of devices with each step. "PHYS" Variables A very general purpose facility is included for defining sampled variables where some extensive setup or post-acquisition processing is required. For instance, when: * the interesting value must be derived from a complex algorithm, possibly involving factors that are not themselves sampled variables in Correlation Plots * a complex setup procedure must be invoked, like an RPC to a remote service, for instance using VeeTest to acquire data. PHYS variables can even execute a VMS DCL script to perform the acquisition. Math Expression Variables The value of a sampled variable may be an arithmetic expression, in which case each element of the value array is the result of the expression on all congruent elements in the value arrays of the operands. The expressions can include transcendental functions etc - basically any arithmetic function offered by a c rtl function. The expression is compiled into object code in a language implemented specifically for Correlation Plots and the resulting "program" is held in memory, with a pointer to the program held in the variable's ap_descriptor. Applications Programming Interface (API) CRRINTERRFACE.FOR defines functions which can be used by Correlation Plots applications. These include functions to: 1. Set and get a variable to a device name. 2. Deposit to, or read value of, a variable + congruent data such as error. 3. Set up step vars (range, #steps, size of increments) 4. Select a fit type for plot. 5. Retrieve fit results. 6. Select plot type (2d, 3d) 7. Select plot mode (line, dot ) 8. Do plot 9. Set or get number of samples to take or taken. 10. Switch to/from manual to auto acquisition mode. 11. Take manual acquisition step 12. Do auto-acquisition. 13. Get or set sample parameters like BPM averaging. 14. Aggregate scripts for the above, like reset all, set up an entire acquisition etc. Saving Data to File A number of file saving options are included: matlab .MAT files. matlab ASCII files Unformatted ASCII files 20/20 spreadsheet files Saving Correlation Plots Configuration to File. One can save a Correlation Plots configuration. In this case the names of the devices, or the arithmetic expression, that defines each variable, is written to an ASCII file so an experiment can be repeated at some later time. The files are ASCII and in a simple sequential form so that, in theory, a user can set up an experiment by creating its configuration file in an editor. Displaying Results The user interface for displaying results is interesting because it offers a very easy method of requesting the scatter plot of any 2 or 3 variables. Two buttons exist for assigning a variable to a particular axis, x or y, one for each axis. Then, when any other variable's button is hit, a display is generated with that variable on the other axis. For instance, having assigned the independent variable to the x axis, only one button push is needed to draw its scatter-plot against each other of the 160 or so possible variables. In addition though, there's a "plot pairs" button. This makes use of the specific arrangement of the names of the variables on the interface panel. 2 variables, juxtaposed in a column can be plotted, one on x the other on y. Putting the right variables on the buttons that will facilitate this arrangement is left to the user when entering the variables. Also a 3d plot is offered for viewing the plot of both step variables (remember one is nested within the other) against a third, dependent, variable. Fitting A number of fitting routines are offered, some obvious like linear, average, polynomial, sinusoidal; some proprietary like "beam-beam deflection", "eigfit". Status of an Acquisition and Numerical Errors Some Sampled Variables acquirable by Correlation Plots include some "status" to indicate how successful was the acquisition. Additionally some variables may be returned with a numerical error (for instance if the acquisition of a single value was actually the result of some averaged acquisition, like say 4 readings of a beam monitor). Correlation Plots uses, if available, information from both the status, and error, in fitting and in displays. It also defines a class of Sampled Variables called "errorable" (carrying some numerical error), and if the error was too small(!), like <1.0e-15, from such a sampled variable, Correlation Plots will conclude that the sampled reading was probably in error, and it marks that acquisition, of that variable, as bad. 3. ABSTRACT DATA TYPE DESCRIBING A DEVICE Correlation Plots does embody some data "encapsulation", by holding all of the information pertinent to a single variable (device) in a single instance of a union (actually a structure with union members). The values in that union are entirely managed through 2 access routines, one to get the info, the other to set it: and in fact client routines that handle devices don't know that the device is described by a union. Also, there is a single definition of that union - which is general enough as to hold sufficient information to deposit or acquire data for any kind of device understood by Correlation Plots. An instance of this single union is sufficient both to name the device and describe all the features of it necessary to make an acquisition. This idea is similar to that of the TAG in cdev. The include file APDESC.TXT defines the union that describes all facets of a device: APDESC.TXT C=================================================================== C Abs: Include file to define the general device descriptor data C structure. C C The general device descriptor is a simple data structure which C can be used to describe all the different types of devices C currently used by the CORRELATION PLOTS and Multi-device Knob C Utility. It is hoped that future device inclusions can be C accommodated easily with minimal code additions to to the AP_ C support routines, rather than major changes to each application. C=================================================================== C C Access indices into a device descriptor, to obtain C the various parts of a device's name. There are a maximum of 6 elements C that define a device's name. C PARAMETER (AP_PRIM = 1, ! Primary > AP_MICR = 2, ! Micro > AP_EXPR = 2, ! Ptr to NUL terminated expression > AP_UNIT = 3, ! Unit > AP_PROG = 3, ! Ptr to STOP terminated expression program > AP_SECN = 4, ! Secondary > AP_SECN2= 5, ! Additional information for 8 > ! character secondary name. > AP_BEAM = 5, ! Beam number (When Secn = TDES) > AP_ELEM = 5, ! Element name for Fast Feedback > AP_SYMP = 5, ! If Sampled Variable is on LHS of expression, ! ptr to SYMBOL_TS symbol describing that SV. > AP_ELEM2= 6) ! Additional information for 8 ! character element name. C Access indices into a device descriptor to obtain the "system attributes" C of a device. There are a maximum of 5 system attributes. C PARAMETER (MAX_XDESCSIZE = 5, ! Number of "control attributes" > X_DBPTR = 0, ! Database Pointer > X_BPMPTR = 0, ! BPM array index > X_ASTSDBNOD = 0, ! ASTS node into dbnode > X_KLYS_FTP1 = 0, ! part of FTP VM pointer > X_INJT = 0, ! comm block index > X_TOFF = 0, ! T Matrix offset value > X_CAMAC = 0, ! comm block index > X_INPT = 0, ! INPT variable serial number > X_DTIZ = 0, ! secn index in returned data > X_BSCN = 0, ! comm block index > X_WSCN = 0, ! comm block index > X_PHYS = 0, ! physics parameter type > X_TCOL = 1, ! T Matrix Column > X_WSCN_SKEW = 1, ! use skew context instead of wscn > X_ASTSUBLK = 1, ! ASTS Sub Block ptr. > X_BPMDMODE = 1, ! data word difference mode > X_KLYS_FTP2 = 1, ! part of FTP VM pointer > X_FBCK = 1, ! offset for Fast Feedback > X_PCDTYPE = 2, ! Timing device PCD type > X_BPMRMS = 2, ! rms data pointer > X_BPMRMSMODE = 3, ! rms word difference mode > X_BPMHDR = 4) ! Extra BPM header word. C Define a structure for the descriptor. This must agree with the C parameters defined above. Note that only the device name parts are named C in the structure - all the X_ system attributes are defined only by C virtue of the last 5 elements in the .d member array. structure / ap_descriptor/ union map integer*4 d( 6+5 ) ! Last 5 elements are the system attributes endmap map integer*4 prim ! "type" union map integer*4 micr ! "region" endmap map integer*4 loop endmap map integer*4 expr_c endmap endunion union map integer*4 unit ! "ordinal" endmap map integer*4 loop2 endmap map integer*4 prog_p endmap endunion integer*4 secn union map integer*4 secn2 ! "property" endmap map integer*4 beam endmap map integer*4 elem endmap map integer*4 sym_ps endmap endunion integer*4 elem2 ! "Channel # of analog devices" endmap endunion end structure 4. FUNCTIONS MANAGING THE ADT DESCRIBING A DEVICE An array of the /ap_descriptor/ structure is defined, with one element for each device, step or sampled, in Correlation Plots. Another name for this "union" (apart from ap_descriptor) is "X descriptor". Interactions with instances of this "Abstract Data Type" are entirely through the routines X_GET() and X_PUT(). The lower level Correlation Plots functions that actually get or put the data for a device are then, pretentiously, "polymorphic" - they examine that part of the contents of the union that describes the `kind' of device they've been handed, and then act to process the device given the information in the remaining `control attributes' members of the ap_descriptor. They recover the values of the control attributes through the ADT access functions X_GET() and X_PUT(). The functions that get or put the value of a device are AP_GETSNGL() and AP_PUTSNGL(). Encapsulation of a device's description INTEGER*4 FUNCTION X_GET (x descriptor, attribute index, value, n_devs) INTEGER*4 FUNCTION X_PUT (x descriptor, attribute index, value, n_devs) eg X_GET( db variable name ap_descriptor, X_DBPTR, returned database address, 1 ) Given a device name, which in the SLC naming convention includes its type, plus the index of the attribute of interest, these routines permit the value of that attribute to be assigned or retrieved. A translation can be made between a variable name as given by the user, and the device name and property that formally describe that variable (a device name is like an EPICS record name; the property of the device is like an EPICS channel name) - we will call a device name and its property of interest a "variable", which may be "sampled" or "step". Additionally, there may be a number of system attributes necessary to describe how to manage a variable. These two problems are addressed by X_SET_DESCR() INTEGER*4 FUNCTION X_SET_DESCR ( modifies an ap_descriptor ) This function modifies the given ap_descriptor, placing in it values of all the system attributes needed to manage the device's property, in the appropriate members. Given a formal device name and property, it gets the values of the various system attributes needed to interact with that device to get or put the property. There may be >1 such attribute for a device, for instance for an SLC Analog Device there's the database node on which the device resides (X_ASTSDBNOD), and the address of a shared memory sub-block in which its value is held (X_ASTSUBLK). At the level of the user interface, a function named "CRR_SET_DESCR()" translates the variable name a user entered into a device name and property understood by Correlation Plots. For many variables this is a no-op - the variable name is the device name, but for variables that do not map to database channel the translation will not be that simple. Also, the user can enter a wildcard for either the device name or the property of a device; this function will standardize those wildcards into a format that X_SET_DESCR can understand. Encapsulation of a device's value management There are two functions that handle all deposits to, and acquisitions from, the sampled variables: These functions are used in the actual data acquisition loop of Correlation Plots. They provide a single point of contact with all kinds of device. AP_PUTSNGL() Given a device name, and required value, this function retrieves the system attributes for managing the device's value from its "ap descriptor", using the x_get() access routine (see above). It knows which attributes to get because it knows the device's type from its name. From there it deposits the required value directed by the device's type and the values of its attributes. It will be called once for each step of each independent variable. AP_GETSNGL() Given a device's name, this function retrieves the system attributes for managing the device's value from its "ap descriptor", using the x_get() access routine (see above). The attributes tell it where to look for the value. It then retrieves and returns the value. 5. BENEFITS OF THIS OO NATURE 1. Allows all information about how to acquire or set the value of a device to be held in one place, accessible in only one way (that is, privatized). 2. The accessing mechanism to the description of how to interact with a device is polymorphic. X_GET and X_PUT handle all deposits and acquisitions of the attributes for any kind of device, and therefore offer a single interface. 3. The accessing mechanism to the device values themselves are polymorphic. AP_PUTSNGL and AP_GETSNGL handle all data deposits and acquisitions, and therefore offer a single interface. 4. There may be any number (actually <=5) hidden attributes of a device but all are handled within the scope of a single function call. 5. A single interface for setting up and doing a Correlation Plots acquisition. 6. The user can use the common name of a device, and use a wild-card for either or both of the name or the property. This ADT is central to the bulk of the correlation plots package, namely the preparation for an acquisition. 6. PREPARATION FOR DATA ACQUISITION Often in the SLC, the method or location from which a device's value can be acquired, is not straightforward. Only 30% or so of the devices taking part in a correlation plots acquisitions are simply central data-base variables. The remainder engender more effort. For instance to get a klystron phase shift advance from a fast time plot requires taking the time plot acquisition, reducing the data and uploading it to Correlation Plots. This is an example of a "context" - see below. From a variable name, as entered by the user, Correlation Plots does the following to determine how it is to acquire the data: 1. At the time that the user enters the variable name: Members of the variable's descriptor union are setup so that acquisition of the variable will be expedited in the data acquisition cycle. For instance, the address of the variable in the SLC database is copied to its x descriptor. When a user enters a device name as a variable then, it's validated, and then the following functions (described above ) are called in order to set up a correlation Plots variable: Touch panel code CRR_INPUT_VALUE() Checks var (possibly incl. * legal), if so sets it up. . CRR_SET_XDESCR() Var transl. to device name + property, either may be * . . X_SET_DESCR() Gets system attributes (X_) needed to recover value. . . . X_PUT() X descriptor for all devices in list populated. Called once for each property found by X_SET_DESCR(). 2. Just before the data acquisition cycle is started: 2.1 Regional information is assimilated. In the SLC, some kinds of devices are grouped into regions, or, "sectors". Example devices are magnets and klystrons. Operations on devices in these sectors go for all the devices in the sector at once. For instance, even if one wants to know the value of only one device in the sector, some setup may have to be done for all of such devices in the sector, or the acquisition operation itself may have to be done for all such device in the sector, and these operations may be quite lengthy. In the magnet case, the SLC control system magnet utility routines take lists of regions on which to perform required operation on all the magnets in the regions specified. So, if just one Correlation plots variable is say magnet "LGPS LI04 45 BVAL", the current B-field value of all magnets in sector LI04 would have to be uploaded to the SLC database, and the function to do that would have to be called with a sector list that contained one element, "LI04". The sector information is assimilated for each kind of device which requires it. 2.2. List of all pertinent acquisition "contexts" are assimilated. Contexts are closely related to sectors and are used by Correlation Plots variables of type "PHYS". A context's value is a set of constants that describe what data is necessary to perform a PHYS variable calculation for a particular sector, and where to get it. PHYS variables are those that require a calculation or some other coded processing, to compute their value. There are a number of kinds of PHYS variable, one for each calculation type, like wire-scan, even one for performing CAMAC operations. For instance, a PHYS variable that retrieves some moment of the Fast Time Plot of a klystron's phase advance, needs to know which region the klystron is in and the unit number of the klystron, plus the memory address that the FTP data for that klystron is kept in its controlling micro. All of that info will be in the klystron context for that region. As part of the data acquisition preparation, the list of contexts, one for each pertinent sector, for each PHYS variable type is constructed. 7. DATA ACQUISITION Having done the preparation, the acquisition itself is easy. The call-tree in outline is: CRR_DATA_ACQ Initialization: CRR_DATA_ACQ_INIT Builds new "context" if an SV has changed. . CRR_DATA_ACQ_DONE Frees last context before the next is built. . CRR_SCAN_SV Scans through the list of sampled variables defining which types are "present". Also constructs region lists for klystrons, magnets etc.. These lists are used by MSG_SYNC for ensuring the data base values are update before sampling. . . X_PUT Puts context, camac packets etc in X descriptor for SV. CRR_SAVERESTORE Saves and restores values of step variable(s) (. CRR_GET_SV) -if before acquisition to save step var value (. CRR_TRIM_STV) -if after acqu. to restore previous value. Acquisition: For each acquisition cycle, for each step variable, set its value, then make the acquisition from each sampled variable. CRR_TRIM_STV Sets values of step variable(s). . CRR_PUT_STV . . AP_DEVPUT Interface routine for PUTting > 1 device at a time. . . . AP_PUTSNGL Puts value into 1 step variable. . . . . X_GET Says where to put values etc. . . . . {DBLPUT}Put to db or other proactive operation. CRR_PREP_DATA Performs operations to update values of variables, like doing camac, sync db, doing wire scan etc. . BPM_ACQ Sends msgs to micros to prepare to read bpms and tells MPG to broadcast beamcode telling micros to read bpms. . CRR_DB_SYNC Makes sure db has been updated before values are read. . CAMGO Sends camac. CRR_SAMPLE_ALL Manages getting values of all sampled variables. . CRR_GET_SV . . AP_DEVGET . . . AP_GETSNGL Gets a single sampled variable. . . . . X_GET Says where to get values from, eg dblist ptr. . . . . {DBLGET}Get value from data source given by x_get(). End loop over number of samples CMTH_EVAL_ALL Evaluates all arithmetic expression variables. 8. DISPLAY RESULTS The data collected by the acquisition is held simply as a 2d array of floats, sized as "number of variables by number of samples taken". From this simple structure, plotting routines in Handypak (developed at SLAC) are called to draw charts. If some fitting routine is selected, the locus of the fit is superimposed and fit parameters written on the chart. There are a variety of fits available: average, linear, polynomial, sinusoidal, Gaussian. There are also some proprietary fits used only by "the few": beam-beam deflection fit used for optimizing collisions and estimating beam size, "eigfit" which uses a "principle axis transformation" to make an unbiased linear fit when there are errors in the independent variable. Only one full-sized chart can be plotted at a time. There is some facility for plotting up to 4 devices against a step variable, but the charts are really too small to be useful. Also, the facilities for 3d plots are limited. 9. CONCLUSIONS Correlation Plots has proved to be a very powerful tool, used for commissioning, accelerator physics and diagnostics. Much of this popularity is attributable to it its exhaustiveness - Correlation Plots can correlate almost any control point or interesting value with any other, and this property is now an accepted feature of the control system. The reason Correlation Plots has been able to keep pace with new devices is in great part because of the very simple task of adding devices. Correlation plots "Applications" can be developed very rapidly, and have facilitated many new experiments without the need for specialized code. While Correlation Plots does provide some data reduction and validation, its weakest aspect is in the analysis of data.