LCLS Controls

SLC-Aware IOC Design

---

Async Utilities Design

Quick links:

• Introduction
• Async Utilities

1. Scope

This document describes the software design for the SLC-aware IOC Async Utilities. These utilities are for use by cstrAsync, cstrHdlr, and mgntHdlr.

2. Introduction

2.1 Background

Job Global data (including slcJobFunc_as) in file scoped within the .c file containing these utilities. That data is protected by being accessible only by calling these utilities. Again, access to the global area by async handler threads is always through an Async utility.

Much of the functionality here is implemented for the SLC Micros in utilities whose source lives in REF_RMX_ASYNC.

The following table shows the names of Functions on the SLC RMX micros and the Functions on the SLC IOC which accomplish roughly the same functionality.

SLC Micro Function(s)	SLC IOC Function
ASYNCDBINIT INIT_CYC_GLBCOM CYCLE_INIT	slcAsyncInit
INIT_MICR_STS	cstrInitMicrStsInit
GET_CYCLE_TIME	None
GET_CYCLE_FCN	slcAsyncGetCycleFcn
METER_DBSEND	slcAsyncMeterDbsend
PROC_MICR_STS	cstrProcMicrSts
LOCK_STATS	slcAsyncLockStats
UNLOCK_STATS	slcAsyncUnlockStats
SET_FCN_STATS	slcAsyncSetFcnStats

2.2 References

1. SLC Asynch Database Update Design Spec by T Lahey, N Spencer 1989

2. Improving Control of Auto-Checking Functions by T Lahey,N Spencer, R Hall 1990

2.3 Requirements

See the SLC-Aware IOC Functional Requirements by Stephanie Alison.

3. Async Utilities Design

General sketch for all Async Handler Threads that perform cycling

This is the general pattern followed by All Async Handler Threads including (but not limited to) cstrHdlr and mgntHdlr.

Prior to an Async Handler thread running, slcAsyncInit() is called by cstrHdlr at it’s initialization time to set up the Async Global area once for all async Job/functions.

These are the calls that Async Handlers should make for handling cycling functions (Handlers may do other things like receiving VMS messages. Those things are not listed):

LOOP:

· If slcThreads_as[thisHdlr].stop ==epicsTrue, clean up and return.

· Receive an IOC_ASYNC message from cstrAsync which indicates the cycling function to perform. The function index is part of the message.

· Call slcAsyncGetFuncIndexByName to convert index to name.

· Call a local cycling function based on that name.

· Get current epicsTime

· If the cycling function did any dblputs, Call slcAsyncMeterDbSend(funcIx, requestType=slcAsync, forceUpdate=False). It does dbupdate().

· Call slcAsyncSetFcnStats(funcIx, epicsTime, requestType=slcAsync) to update CSTR related statistics in job global.

Exists Flag and Mutex Used by Async Tasks

1. Boolean flags.

· asyncExists is a flag in slc ioc global (see util/slcGlob.c). It is set to False by the executive before cstrAsync is run.

At the end of it’s processing, slcAsyncInit() sets the asyncExists flag to True to indicate that the slcJobFunc global table creation is complete. If slcAsyncInit fails, it sets the asyncExists flag to “false.

Public slcAsync utilities check if the async global area has been set up successfully by calling slcAsyncReady() which checks the asyncExists.

slcAsyncExit() sets the asyncExists flag to “false”.

· The other flags of interest are the active and stopped flags in slcThread_as.

2. Mutex

There is a single mutex (asyncRWMutex) which is used to protect access to the Global Data by utility functions which write to that area. The mutex insures that the utilites are thread-safe. slcAsyncLockStats() and slcAsyncUnlockStats() are utilities for use obtaining and releasing the mutex.

Message Logging

Messages to be logged are stated below under the individual utility descriptions.

Function Return Status

Most utilities return epicsBoolean type status set to epicsTrue (for success) or epicsFalse (for failure).

Resource Management

Writes to the SLC database will be metered in a manner similar to the SLC Micro Test Job by calling the utility function slcAsyncMeterDbsend().

Global Data

slcJobFunc_as is an array of structures in file scope, with one structure (array element) per cycling function across the entire IOC. Each Job may have zero or more cycling functions. The array is calloc'ed by slcAsyncInit() to have one entry per cycling function across the current slc ioc. The pointer to this array is static and file scoped.

slcJobFunc_as and the associated set of dblist pointers are often referred to as “Global Data” within this document. It is called “Job Global” to correspond with the SLC Micro code.

There is a file scoped Boolean flag called asyncExists and a mutex that both provide protected access to Job Global (see descriptions above).

It is important to note that in the CSTR database, all functions for a given Job must be grouped together (example: CSTR:micr:1,CNAM may NOT indicate a mixed grouping like this: KLYS-TRMP CRAT-WATC KLYS-FCHK...)
slcAsyncInit issues an error message and returns bad status if the functions are not all grouped together for all jobs.

Two new entries will be added to slcJob_ts.

· The first is "first cycling function index". This is an integer index into the array slcJobFunc_as. This integer is passed to async functions as an input argument.

· The second is "number of cycling functions". This is also an integer.

The following table describes the association between slcJob_as and slcJobFunc_as.

Async Utilities below look at all cycling functions for a given job.
That is done by looking in slcJobs_as[jobEnum] for the current job, and picking up the first cycle function index and number_of_functions. Those two values are used to inspect the slcJobFunc_as and associated arrays of secondary values.

The following structure is slcJobFunc_ts. To the right is the initial value for each element as set by slcAsyncInit.

INITIAL VALUE

--------------

/* Async cycling function information */

typedef struct {

slcJob_te slcJobId_e; /* enum into slcJob_as for this job */ 9999

char funcName_c[5]; /* null terminated 4-character func name */ READ FROM CSTR:CNAM

unsigned long cyclTimeSec; /* cycle time from CSTR:CYCL */ READ FROM CSTR:CYCL

unsigned long mtrlSec; /* dpupdate meter length in seconds */ READ FROM CSTR:MTRL

unsigned long mtrcSec; /* dbupdate meter max # dbupdates in mrtl */ READ FROM CSTR:MTRC

unsigned long mtrtSec; /* dbupdate meter max time between dbupdates */ READ FROM CSTR:MTRT

epicsTimeStamp meterStartTime_s; /* time metering started */ current time

unsigned long meterCount; /* metering counter */ 0

unsigned long lastDbupdateStatus; /* last database update status */ 0

epicsTimeStamp updateTimeStamp_s; /* timestamp of last dbupdate */ current time

epicsTimeStamp cycleTimeStamp_s; /* last time this cycl function was executed */ current time

unsigned long betweenCycle; /* time between last 2 cycles */ 0

unsigned long numFuncExe; /* total # function executions in calc interval */ 0

unsigned long numVmsMsg; /* # of function executions from vms msg interv */ 0

unsigned long numDbUpdates; /* # of dbupdates (for metering and %) interval */ 0

unsigned long numDbUpdateFail; /* # of dbupdates that failed in calc interval */ 0

epicsTimeStamp sendTimeStamp_s; /* time when db update sent */ current time

unsigned long sendErrCount; /* number of failures in send */ 0

} slcJobFunc_ts;

Diagnostics

Write Async job/function table for an input job name (can be ALL*) to the console.

Provide a way for the user to change a subset of cycling parameters for a job/function pair. Use utilities slcAsyncSetXXXX described below to set them. They must lock the global area before writing it.

Async Utility Function descriptions

The Async utilities all exist in a single file called libsrc/util/slcAsync.c. Prototypes are in slcAsync.h

Utilities are thread-safe. This is accomplished by using a single mutex to protect access to the global data.

It is necessary to restart the SLC IOC in order to add a new cycling function. But, all other cycling parameters (besides CNAM) from the database like CYCL and MTRx are kept in job global and used by Async utilities. Every time those parameters are used, they are read from Job Global. Thus, for an existing cycling function, those items are changeable from the ioc console.

1. epicsBoolean slcAsyncInit(void)

Called by cstrAsync at initialization time to set up the Async Global area before any Handler threads are run. If slcAsyncInit returns bad status, cstrAsync exits. The global area is actually a job/function called slcJobFunc_as in file scope (see above for more on this table). slcAsyncInit is not called from anywhere else.

slcAsyncInit corresponds to several functions on the slc micros as shown in the table above.

slcAsyncInit returns epicsTrue if successful. It returns epicsFalse (error) if there are errors returned by memory allocation, database calls, or the CSTR data fetched from the SLC database.

Call slcAsyncInitVerify() to get cycling parameters from the database, check them, and allocate/initialize slcJobFunc_as. If it returns bad status, goto egresss and do the flag processing at the end of this function which will cause us to return bad status and cause the thread to exit..

Allocate the mutex (asyncRWMutex_ps) for use in accessing the Global Area by calling epicsMutexCreate

The asyncExists global flag is set to true also.

If everything above went OK,

Set slcThreads_as[cstrAsync].active = epicsTrue. This tells the executive that the thread has started.

Set asyncExists local flag to “true”

Else

Set asyncExists local flag to “false”

2. epicsBoolean slcAsyncInitVerify(void)

Get cycling parameters from the database, check them, and allocate/initialize slcJobFunc_as for all job/functions.

Return epicsTrue for success, epicsFalse for failure.

A failure is considered fatal and the caller should cause cleanup and exit to occur. Any of the following cases (and probably more) are considered fatal.

· Invalid job name in CNAM

· Invalid ordering in CNAM

· Any error from any db utility.

· Any error from getting the mutex. Or memory allocation error.

· Sizes of dblists and dbdatas are not as expected.

· Problem initializing a timestamp

This is a static function (called only by another function within this .c file).

The following secondaries values are read (one time only) from the CSTR SLC database primary.

Call dblistalloc(), dblist() and then dblget(CSTR,[this_micro],1,SECN) which, in one call (per secondary) returns lists (arrays) of each value of size MAX_NUM_FUNCS. Calloc slcJobFunc_as based on how many functions are in the CNAM list. These are the secondaries obtained from the database:

§ CNAM - contains Job Name and Cycle Function.

§ CYCL - is the cycle length in seconds.

§ MTRL, MTRC, and MAXT - database write metering parameters.

If any of the DB calls fail, then return bad status.

Loop through the dblists and do the following steps

1. As explained above, the slcJobFunc_as array is calloc'ed by this function to have one entry per cycling function across the current slc ioc.. Cycling functions for a given Job are grouped together in slcJobFunc_as Check that the job name in CNAM is valid and matches a job name in the slcJob_ts array. Log a fatal error if there's no match and break out of loop.

2. Store CNAM (job and function) into slcJobFunc_as. Compute the index to slcJob_as for each functions job and store it in slcJobFunc_as[].slcJobEnum.

3. Check CYCL, MTRC, MTRL, MAXT (dblist/dblget was already done ),
against reasonable limits. Any value which is outside reasonable limits is reset to an appropriate default (see the supertype I table in the requirements doc for defaults). Store values in slcJobFunc_as using one of the slcAsyncSetXXX() functions below. Log a message if the values are set to defaults stating DB value and default it was set to. Return good status in that case.

4. Increment function counter in slcJob_as for this job. Set function index in slcJob_as if first function for job.

Call dbListFree() to free the list obtained above for all the DB lists.

3. epicsBoolean slcAsyncGetCycleFcn(1stCycleFuncIx, numCycleFunc, * cycl2Exe);

This is a public function called by cstrAsync. It returns (in cycl2Exe) the slcJobFuncIx (index) of the cycling function which is “most ready to run” for the current job. It does this by going through all entries in Job Global for the current job as indicated by 1stCycleFuncIx and numCycleFunc.

Call slcAsyncReady() and return if it returns an error.

Call slcAsyncUpdSt2() to dblget() the latest values from the database for CSTR: CMSK, HSTA, MMSK, and SCAN (you’re really getting values for all functions across the ioc). Note: Some common cycling functions (like Magnets) can have CSTR:CMSK changed from the SCP Cluster Status Panel. However, there are other cycling functions that are not supported for that toggling. For the purposes of this code, we'll just always support it being changed for all cycling functions as far as code in this function goes.

Call slcAsyncLockStats() to get a mutex on the Global Area. Return if it returns an error.

Go through entries in slcJobFunc_as (using the input arguments to index it) and see if there an async function for this job that is ready to run based on current time, CSTR: CTIM, CYCL, SCAN, HSTA, CMSK, and MMSK. Return an indication as to which of these async functions is “most ready to run” to the caller using cycl2Exe.

For a cycling function to be ready to run, the following conditions must be true.

§ Current_time – last_execution_time > cycling_period

§ Current_time – time_of_last_msg_send > cycling_period

§ The cycling_period is either CYCL or SCAN, if the bit for the function is set in MMSK and SCAN is less than CYCL.

§ The function must be enabled as determined by checking the appropriate bit in CMSK.

§ The job must be enabled as determined by checking the appropriate bit in HSTA.

Egress: Call slcAsyncUnlockStats() to release the mutex on the Global Area.

4. epicsBoolean slcAsyncSetCYCL (jobFuncIx, value) set CYCL in slcJobFunc_as[jobFuncIx] to the value passed in.
5. epicsBoolean slcAsyncSetMTRC (jobFuncIx, value) set MRTC
6. epicsBoolean slcAsyncSetMTRL (jobFuncIx, value) set MTRL
7. epicsBoolean slcAsyncSetMAXT (jobFuncIx, value) set MAXT

8. epicsBoolean slcAsyncProcChk1(CTIMdblist_ps, CTIMdbdata_ps, UTIMdblist_ps, UTIMdbdata_ps, ELPSdblist_ps, ELPSdbdata_ps, )

This is a public function called by cstrHdlr only. It processes UTIM, CTIM, and ELPS.

These secondaries are always sent (updated) without checking if they changed because UTIM and CTIM change when this function runs.

Call slcAsyncReady() and return if it returns an error.

Call slcAsyncLockStats() to get a mutex on the Global Area. Return if it returns an error.
For each cycling function in the Global Area:

Put the last database update time from Global into the UTIM dblist.
Put the last cycling function update time from Global into the CTIM dblist.
Put elapsed time from Global in the ELPS dblist.

Timestamps are converted to VMS format using an slcioc utility function before putting them into the slc database.

cstrHdlr will do the dlbput.
Call slcAsyncUnlockStats() to release the mutex on the Global Area.

Any failures in the calls above logic cause a epicsFalse (error) return status. Otherwise epicsTrue is returned (success).

9. epicsBoolean slcAsyncProcChk2(NRUNdblist_ps, NRUNdbdata_ps, FAILdblist_ps, FAILdbdata_ps, PUPDdblist_ps, PUPDbdata_ps, PVAXdblist_ps, PVAXdbdata_ps, epicsBoolean dbUpdateNeeded)

This is a public function called by cstrHdlr only. It processes NRUN, FAIL, PVAX, and PUPD.

dbUpdateNeeded is an output parameter.

Call slcAsyncReady() and return if it returns an error.

Call slcAsyncLockStats() to get a mutex on the Global Area.

For each cycling function in the Global Area:

If Number of Executions (NRUN) has changed in Global vs what’s in the previous used dblist, put the new value from Global into the NRUN

dblist and set dbUpdateNeeded flag that something changed

If number of failures (FAIL) has changed in Global vs what’s in the previous used dblist, put the new value from Global into the FAIL

Dblist and set dbUpdateNeeded flag that something changed.
Calculate Percent executions triggered by Vax Messages (PVAX). If it has changed, store it in the PVAX dblist and set flag.
 Calculate Percent executions triggering successful database updates (PUPD). If it has changed, store it in the PUPD dblist and set flag.

Zero the counters for NRUN, FAIL, DBUPDATES, and VAXFUNCS in global.
Call slcAsyncUnlockStats() to release the mutex on the Global Area.

Any failures in the calls above logic cause a epicsFalse (error) return status. Otherwise epicsTrue is returned (success).

10. epicsBoolean slcAsyncUpdSt2(void )

This is a static function (called only by another function within this .c file).

Call slcAsyncReady() and return if it returns an error.

Call slcAsyncLockStats() to get a mutex on the Global Area.

Get MMSK, CMSK, HSTA, FMSK and SCAN from database using dblget()

Pointers to the dblget lists are already set up in file scope.

Return epicsFalse if a dblget failures. epicsTrue otherwise.

Call slcAsyncUnlockStats() to release the mutex on the Global Area.

11. epicsBoolean slcAsyncReady(void )

This is a static function. It returns epicsTrue for success. epicsFalse for an error.

slcAsync utilities check if the async global area has been set up successfully and is available by calling this function.

This function checks the asyncExists slc-global flag.

If (asyncExists == False)

return epicsFalse to indicate an error (this can be caused by slcAsync exiting).

ELSE

return epicsTrue (success)

12. epicsBoolean slcAsyncGetFuncIndexByName(const char * const name_a)
    
    

This public utility is used by Handlers and finds the slcJobFunc_as[] index that matches the first 4 chars of name_pa. Returns index into slcJobFunc_as if found, returns epicsFalse (INVALID_FUNC) if no matching name is found.

The first thing it does is to call slcAsyncReady() to wait on for the global area to be ready to be accessed. If it returns an error, return with error from here.

13. epicsBoolean slcAsyncMeterDbsend (int asyncFuncIx, epicsBoolean forceUpdate)
    
    

· asyncFuncIx is an input parameter indicating the index in slcJobFunc_as for the check function which needs to be checked to see if a dbupdate needs to be done. NOTE: It is not necessary for the caller to pass jobId, since that can be obtained here from slcJobFunc_as[asyncFuncIx].slcJobIx.

· Force_update is an input Boolean indicating if a dbupdate is to be forced. Caller will set this to false for IOC_ASYNC requests.

This is a public function for use by all async threads. It does a dbupdate for a given async function if it’s time to do so or if force_update is set. Dbupdate() actually updates the whole job, not just one async function’s data, but that works well.

There are entries in slcJobFunc_ts (listed above) for keeping track of metering. They include:

· epicsTimeStamp meterStartTime; /* time metering started*/

· int meterCount; /* metering counter*/

· int lastDbupdateStatus; /* last database update status */

If forceUpdate, then, the dbupdate() is always done (forced) even if the metering counters indicate it's not time to do so.

The first thing it does is to call slcAsyncReady() to verify that the global area to be ready to be accessed. If it returns an error, return with error from here.

Call slcAsyncLockStats() and then check the following items in slcJobFunc_as[asyncFunc_ix] to see if a dbupdate() needs to be done.

§ Do an update if the last Alpha update request for this job/function failed.

§ Do an update if no update has happened in the last MAXT seconds.

§ Do an update if the number of successful database updates is less than the allowed number (MTRC) in the last MTRL seconds.

Do an update if the bit for the function is set in FMSK.

Note: dbupdate(jobNameEnum) does an update of ST 3 data for all threads for the indicated job.

Call slcAsyncUnlockStats() and call dbupdate() if any of the above conditions are true.

Note: It is not necessary to process OMSK as is done on the slc micros. That functionality was never implemented.

Call local function slcAsyncSetDbStats(asyncFuncIx…) to update database related statistics (like database update counters) in the slcJobFunc_as array for the input function if an Alpha update was done.

14. epicsBoolean slcAsyncSetDbStats(int asyncFucnIx, epicsBoolean metered, epicsBoolean MeterJustStarted, int dbUpdateStatus,)

· asyncFuncIx is an input parameter indicating the index in slcJobFunc_as for the check function for which Stats are being set.

· Metered is an input parameter indicating if the last call was metered.

· MeterJustStarted is an input parameter indicating if metering just started (0 if dbupdate failed).

· dbUpdateStatus is a status code indicating the status of the last dbupdate() call.

This function is only called by slcAsyncMeterDbsend. It is a static function.

Get current time using epicsTimeGetCurrent. Store in in File Scope???

Call slcAsyncLockStats() to lock the global area.

Update the following items in the slcJobFunc_as array for the input job and function. Presumably, an Alpha dbupdate() was just done:

§ Metering-started timestamp set to current time if metering has just started.

§ Increment the metering counter if metering is in effect.

§ Increment either the database-update-failed counter or database-update-success counter for this job/function depending on status from the update request.

§ Set the last-database-update timestamp to current time.

§ Set the last-database-update status.

Call slcAsyncUnlockStats() to unlock the global area.

Always return good status (epicsTrue).

15. epicsBoolean slcAsyncLockStats(void).

This public function calls epicsMutexLock(id) to take out mutex on Global Area. It returns epicsTrue for success. epicsFalse for failure.

First, call slcAsyncReady() to make sure global is available. If not, return bad status from here.

Note: That call blocks until the lock is available. If the call fails, an error message is logged and bad status is returned.

16. epicsBoolean slcAsyncUnlockStats(void).
    
    

This public function calls epicsMutexUnlock(id) to release mutex on Global Area. It returns epicsTrue for success. epicsFalse for failure.

If the call fails, an error message is logged and bad status is returned.

17. epicsBoolean slcAsyncSetFcnStats(int asyncFuncIx, epicsTimeStamp *startTime_ps, requestType).

2. asyncFuncIx is an input parameter that is an index to slcJobFunc_as indicating which async function this request is for.

3. start_time is an input parameter
Passed by the caller so that the ELPS (elapsed) time can be calculated (current time - start_time (which is the time right before the function was executed)).

4. requestType is an input argument slcReqType_te enum indicating slcAsync or slcMsg.
It indicates if this is as a result of a vms message request or an async request.
It's an enum with two possible values: slcAsync or slcMsg.

This is a public function for use by all async threads.

It sets statistics in the Global Area for the specified cycling function.

Input startTime is the time right before function began and is set by caller (used in calculating func execution elapsed time (ELPS)).

If it returns an error, return with error from here.

Call epicsTimeGetCurrent() to get current time (WHY?... it’s passed in… TBD).

The first thing it does is to call slcAsyncReady() to verify that the global area is ready to be accessed. If it returns an error, return with error from here.

Call slcAsyncLockStats() to take out mutex on Global Area. If it returns an error, return with error from here.

Store current time into in Global Area for this function (set last execution time to current time).
Increment NRUNS (execution counter) in Global Area for this function.
If slcRequestType indicates the call is from a Handler as a result of VMS message (slcMsg), then, increment number of hndlr functions in last calculation interval (and calculate PVAX).
If slcRequestType indicates the call is from a Handler as a result of Async call (slcAsync), then, increment number of async functions in last calculation interval.
Calculate elapsed time (ELPS) and store in Global Area
Call slcAsyncUnlockStats() to release the mutex on the Global Area.

18. epicsBoolean slcAsyncDump(const char * const jobName_a)

This is a public function. It always returns good status epicsTrue.

Call slcAsyncLockStats()

Print all function tables for debugging if jobName_a is ALL*. Othewise, print the function table for the input job name (call slcGetJobIndexByName).

Call slcAsyncUnlockStats()

19. void slcAsyncExit(void)
    
    

This is a public function called by cstrAsync at it’s cleanup time.

Call slcAsyncLockStats()

Set asyncExists global flag to false.

Zero the number of async functions and the first function index in the slcJob_as array for all jobs.

All resources allocated by Async Utilites are deallocated here. This includes memory (eg slcJobFunc_as) and dblists (call dbfree).

Get rid of mutex like this:

NULL the mutex. (TBD** THIS SHOULD BE AFTER MutexDistroy, right?)

Call slcAsyncUnlockStats()

epicsMutexDestroy(asyncRWMutex_ps);

asyncRWMutex_ps = NULL; (TBD** this should be here instead, right?)

---

SLC-Aware IOC Home Page | LCLS Controls | EPICS at SLAC | SLAC Computing | SLAC Networking | SLAC Home

Contact: Ron MacKenzie,
Last Modified: April 2, 2005. by RONM.