LCLS Controls

SLC-Aware IOC Design

---

Cluster Status and Test Service

Quick links:

• Introduction
• Service Design
• cstrAsync Thread Design
• cstrHdlr Thread Design

1. Scope

The following document describes a software design for the SLC-aware IOC (slcIOC) Cluster Status and Test Service.

2. Introduction

This service consists of the cstrAsync Thread and the cstrHdlr Thread.

The cstrAsync Thread is the Async controller thread for the SLC IOC. This simple thread finds out when each cycling function is ready to run and sends a message to the corresponding Async handler thread which runs that function. The handler threads that receive messages from cstrAsync include cstrHdlr and mgntHdlr.

The cstrHdlr Thread has two jobs.

1. Service TEST JOB messages sent from the VMS Paranoia process by echoing back a response for every message received.

2. Service messages sent from the cstrAsync Thread by running the cycling function as indicated by the message.

The cycling functions in cstrHdlr periodically update SLC database CSTR health and status secondaries. This health and status data is read from Global Memory (it was stored there by this and this and other Async handler threads). IOC health and status data is also read from associated EPICS PVs and written to CSTR.

2.1 Background

This same functionality is implemented in the TESTMAIN job on the SLC Micros.

2.2 References

1. PRIMARY.DBS: List of SLC Database Primaries and Secondaries

2. SLC Asynch Database Update Design Spec by T Lahey, N Spencer 1989

3. Improving Control of Auto-Checking Functions by T Lahey,N Spencer, R Hall 1990

2.3 Requirements

See the SLC-Aware IOC Functional Requirements by Stephanie Alison.

3.0 Cluster Status and Test Service Design

3.1 Service Description and External Interfaces

3.1.1 Service Block Diagram

The Cluster Status and Test Service includes two threads shown in the block diagram below: the cstrAsync thread and the cstrHdlr thread.

The cstrAsync thread initializes the Global area by calling slcAsyncInit. The Global area is described on the asyncUtil Design page.

The cstrHdlr thread services TEST-function messages from Paranoia on VMS by receiving the messages and sending a reply back. It reads data from the EPICS database using Epics Runtime Database Access calls. It reads and writes to the SLC CSTR database primary using SLC database utilities and Async utilities.

3.1.2 External Interfaces

Described elsewhere in this document.

3.1.3 Data Flow

See Diagram Above.

3.1.4 Data Structures

See slcJobFunc_ts described in Async Utilities Design.

3.2 cstrAsync Thread Detailed Design

cstrAsync Thread finds out when each cycling function is ready to run and sends a message to the corresponding Async handler thread which runs that function.

3.2.1 Functional Flow

3.2.1.1 Thread Initialization

· slcAsyncInit() is called (see Async Utility web page) to set up the Async Global area and perform other initialization. This initialization includes reading Cycling parameters from the SLC database for all cycling functions. slcAsyncInit sets slcThreads_as[cstrAsync].active = epicsTrue. This tells the executive that the thread has started. The asyncExists global flag is also set to true also and a mutex is created to protect job global.

If slcAsyncInit() returns bad status, cstrAsync branches to egress and calls cstrAsyncExit() and returns.

3.2.1.2 Periodic Thread Processing

· There will be no handling of Check Functions from Paranoia (or other messages from VMS) in cstrAsync like there is in the SLC micro TEST job. On the SLC IOC, that functionality is provided by the cstrHdlr thread.

· In a Loop, perform the following processing:

1. Check the stop flag slcThreads_as[cstrAsync].stop. If it is set, branch to egress and call cstrAsyncExit() and return;

2. For each JOB in slcJobs_as

3. Call slcAsyncGetCycleFcn(1stCycleFuncIx, numCycleFunc, &cycl2Exe) get the slcJobFuncIx of the cycling function to be run for this job.

4. If slcJobFuncIx is not zero, Call cstrAsyncSendFcnMsg(cycl2Exe) to send a message to a handler to run that function.

5. If either call returns bad status, branch to egress (cleanup processing) and call cstrAsyncExit().

6. Call epicsThreadSleep(1) to sleep for one second.

3.2.1.3 Messages Received by cstrAsync

No messages are received by cstrAsync.

3.2.1.4 Termination

As described above, termination is normally initiated by the slcExec thread setting the slcThreads_as[cstrAsync].stop flag to TRUE. When that happens, call cstrAsyncExit() to release resources and clean up. Then, the main thread simply returns.

Fatal errors in thread processing can also cause termination. In that case, cstrAsyncExit() is called.

3.2.2 Global Data

slcThread_as[cstrAsync_e].active is set.

For a description of Global Data, Flags and Mutexes used by Async Tasks, see the Async Utilities Design Document. None are set directly by cstrAsync.

3.2.3 Resource Management

none

3.2.4 Message Logging

All messages logged are stated explicitly in this document.

3.2.5 Diagnostics

See the Async Utilities Design Document for diagnostic function descriptions.

3.2.6 Major Routines

The following table shows the names of Functions on the SLC RMX micros and the corresponding cstrAsync Functions on the SLC IOC which accomplish roughly the same functionality.

This code exists in ioc/cstr/cstrAsync.c and cstrAsync.h

SLC Micro Function	SLC IOC Function
TESTMAIN (job)	cstrAsync (thread)
EXEC_CHK1	cstrHdlrChk1
EXEC_CHK2	cstrHdlrChk2
EXEC_CPUM	cstrHdlrCpum
None	cstrHdlrChk3

1. void cstrAsyncThread(void)

cstrAsyncThread is the main procedure and loop of the cstrAsync thread. It initializes the cstrAsync resources and globals, and then enters a loop where is figures out the next Cycling function to be executed and sends a message to the Handler that processes it. It cleans up resources when terminating.

Functions called only from within cstrAsync

1. epicsBoolean cstrAsyncSendFcnMsg(cycl2Exe)

cycl2Exe is an index into slcJobFunc_as indicating the cycling function to be sent.

Send an IOC_AYNC message the appropriate handler to execute the cycling function associated with cycl2Exe.

The handler ‘s name is found by looking in slcJob_as entry for this job and finding the associated hander.

Call an async utility to set sendTimeStamp_s if the send worked.

Call an async utility to increment sendErrCount counter in slcJobFunc_as[cycl2Exe] if the send failed.

Return epicsFalse if the send fails. Otherwise, return epicsTrue.

2. void cstrAsyncExit()

Called to do cleanup processing before thread exit.

1. Call slcAsyncExit() to release global area resources.

2. Set slcThreads_as[cstrAsync].active = epicsFalse

3. Set slcThreads_as[cstrAsync].tid_ps = NO_THREAD

4. Log a "cstrAsync Stopped and out of service" informational message to CMLOG.

5. return

3.3 cstrHdlr Thread Detailed Design

The CSTR Handler Thread (cstrHdlr) services all TEST job messages from VMS by sending replies back to VMS. SlcAsyncChk1() cycling function is called after each message received.

It also services messages sent from the cstrAsync Thread by running the cycling function as indicated by the message. The cycling functions periodically update SLC database CSTR health and status secondaries. This health and status data is read from Global Memory (it was stored there by this and this and other Async handler threads). IOC health and status data is also read from associated EPICS PVs and written to CSTR by one of the cycling functions. It then queues an update request to the SLC database service.

3.3.1 Functional Flow

3.3.1.1 Initialization

The cstrHdlr thread performs the following as initialization

· Create the message queue using the messageQCreate utility. Call cstrHdlrExit() and return if this fails.

· Call cstrHdlrInitChk1() to initialize dblist() and do dblget() for CTIM, UTIM, and ELPS (one value per function)

· Call cstrHdlrInitChk2() to initialize dblist() and do dblget() for NRUN, FAIL, PUPD, and PVAX (one value per function)

· Call private function cstrHdlrInitMicrStatus() which initializes micro status data items and calls private function cstrProcMicrSts() to set AMSK and MSTA secondaries into the SLC database per the table below. It also echos CSTR:VTIM in MTIM. Call cstrHdlrExit and return if bad status was returned from cstrInitMicrStatus().

· CSTR:VTIM will not be checked to see if it is 25 minutes older than current micro time as it is on the SLC micro. On the SLC micro, that test is done to see if no SCP was involved in the boot request (in which case fast feedback loops are not to be restarted).

· Set slcThreads_as[cstrHdlr].active = epicsTrue

3.3.1.2 Normal Processing Loop

The cstrHdlr thread is similar to the logic of any message-queue driven thread in the slcIOC.
It accepts incoming message commands and acts on them. The basic loop is:

While (slcThreads_as[cstrHdlr].stop !=epicsTrue)

· Read message from message queue using the msgQMsgRecv utility.
Gets the data word length, function code

· If the message is from VMS:

1. Call the appropriate “do” routine (e.g. cstrHdlrDoEcho) depending on which function code was received in the message header

2. If not a valid function code, drop the message

3. Call Async function cstrHdlrChk1((&dbUpdateNeedeed) to perform a CHK1 function.

4. Update statistics for the CHK1 function in the async function tables by calling slcAsyncSetFcnStats()

· Else, if the message is from cstrAsync (it’s an IOC_ASYNC message)

1. Based on the indication in the message of what to do , call one of the cstrAsync check functions:
cstrHdlrChk1(&dbUpdateNeedeed)
cstrHdlrChk2(&dbUpdateNeeded)
cstrHdlrCpum(&dbUpdateNeeded)
cstrHdlrChk3(&dbUpdateNeeded)
Those functions call dblput() (not chk3) as needed to write to the SLC database.

2. Those functions return an indication if the database needs to be updated.

3. If any of those functions return bad status, call cstrHdlrExit() to exit the task

· Call slcAsyncSetFcnStats(asyncFuncIx) to update statistics for the function in the async function (global) table (this is done inside CHK functions on RMX). If it returns bad status, call cstrHdlrExit() to exit the task.

· If the check function returned an indication that the database update needs to be sent to VMS, call slcAsyncMeterDbSend(asyncFuncIx). That indication is set only if something changed and needs to be updated. If it returns bad status, call cstrHdrlExit() to exit the task.

3.3.1.3 Messages Received by the cstrHdlr

The cstrHdlr thread performs the functions related to the follow messages.

3.3.1.3.1 IOC_STOP

The slcExec sends this message to tell the thread to terminate itself. cstrHdlrDoStop() is called.

3.3.1.3.2 TEST_ECHO

VMS sends this message. cstrHdlrDoEcho() is called..

3.3.1.3.3 TEST_ECHO_MWORD

VMS sends this message. cstrHdlrDoEchoMword() is called.

3.3.1.3.4 FUNC_TEST

VMS sends this message. Existence check from PARANOIA. cstrHdlrDoEcho() is called.

3.3.1.3.5 TEST_ERR_METER_RESET

VMS sends this message to reset cmlog throttling. cstrHdlrMeterReset() is called.

3.3.1.3.6 IOC_ASYNC

cstrAsync sends this message. The appropriate cycling function is called.

3.3.1.4 Termination

Termination is initiated by the slcExec thread setting the slcThreads_as[cstrHdlr].stop flag to TRUE, or when the cstrHdlr thread receives a IOC_STOP message.

In either case:

· Set slcThreads_as[cstrHdlr].active = epicsFalse

· Destroy the message queue using the messageQRelease utility. This releases any message buffer held in the message queue first.

· Set slcThreads_as[cstrHdlr].tid_ps = NO_THREAD

· log a "cstrHdlr stopped and out of service" message to CMLOG .

· Log a MICR_DOWN "Stopped and out of service" informational message to CMLOG. New msg in ref_slcmsg:micrdef.msg.

· Return

See the Async utility slcAsyncExit() for a description of resources relased when the slc ioc is stopped.

3.3.2 Global Data

1. slcThreads_as[cstrHdlr].stop is read.

2. See the asyncUtil design page for a description of the Job Global Area where the following data is stored. All access to Global is through Async Utilities.

3. For a description of Flags and Mutexes Used by Async Tasks, see the Async Utilities Document.

4. The following CSTR Supertype I (read-only) secondaries are read as part of cstrHdlr processing. Much of the processing is generalized for all ASYNC threads and is therefore handled within utilities listed in the Async Utilities Section below.

CSTR Secn	Description
CNAM	Cycling function name (job + function name). Read at initialization time only
CYCL	Cycle length in seconds for this function.
MTRL, MTRC, MAXT	Database-write metering parameters (length, count, and max_time).

5. The following CSTR Supertype II (read-only) secondaries are read as part of cstrAsync processing. Much of the processing is generalized for all ASYNC threads and is therefore handled within utilities listed in the Async Utilities Section.

CSTR Secn	Description
HSTA	Status. Set by Cluster Status Panel. Used to tell if an ASYNC is enabled/disabled. Read by IS_FCN_ENABLED() on RMX.
VTIM	Read at initialization time only. Used in setting MTIM
CMSK	Mask of functions in this IOC which should cycle asynchronously. Set by Cluster Status Panel.. Read by IS_FCN_ENABLED() on RMX.
MMSK	Mask of functions that have CA monitors. Used in calculating when to do database updates (if set, use minimum of CYCL and SCAN).
FMSK	Mask of functions that have CA monitors. Used in calculating when to do database updates (if set, force at SCAN interval).
SCAN	Cycle length in seconds for this async function. Used in conjunction with MMSK, FMSK, and CYCL.

6. The following CSTR Supertype III secondaries are handled by the crate verifier on the SLC micros.
On the SLC IOC, they are set to zero at SLC IOC startup time by cstrAsync and then not set periodically.

CSTR Secn	Description
CRTS	Camac Crate Status Mask
CRTT	Camac Crate Temperatures (degF?)
CRV1 to CRV7	Camac Crate Voltages (V?)
CAM	Available CAMAC memory pool (bytes)
NTIM	Last time was TSTA updated by micro
TSTA	Timing job interrupt status

7. The following CSTR Supertype III secondaries are set to the values indicated in the table below at SLC IOC startup time by slcAsyncInit().
There is no periodic processing for these secondaries.

CSTR Secn	Description	Init. Value
MTIM	Last Successful IPL from micro	VTIM

8. The following CSTR Supertype III secondaries are periodically processed.
dblistalloc()/dblist() are called at processes startup time. Then, dblput is called to write values to the database as part of periodic cycling.

CSTR Secn	Description
UTIM	Last database update timestamp per job
CTIM	Last async cycling function update timestamp per job
ELPS	Elapsed time for last cycling function update per job (seconds)
NRUN	Number of executions in last calculation interval per job
FAIL	Number of times dbupdate failed in last calculation interval per job
PUPD	Percent of executions triggering database update in last calculation interval per job.
PVAX	Percent of executions triggered by VMS message in last calculation interval per job.
CPU	CPU idle time (percent)
RMX	Available memory (bytes)
AMSK/MSTA/JMSK	Masks of Active Jobs/Micro Job Status/Jobs Included (JMSK is SuperType I) At startup and periodically, CSTR AMSK (job active mask) and MSTA(micro job status) are set. All these bit masks are zero referenced (32 jobs are represented by bits 0 – 31). AMSK is set by looking the slc job array (slcJobs_as) and setting a bit for each index where the non-zero threads (async and/or hdlr) have the active flag set. Each bit in AMSK corresponding to a particular job is set if ALL the tasks associated with the job are active AND the correspoinding JMSK bit is set.. Then MSTA is set to 1 if AMSK == JMSK and 0 otherwise. This logic is consistent with logic in ref_rmx_test:procmicrsts.f86.

9. The following CSTR Supertype III secondaries are handled by the Magnet Handler (not by cstrAsync).

CSTR Secn	Description
MAGF	Current Magnet Job Function Code
BTIM	Timestamp when magnet job was initialized

3.3.3 Resource Management

3.3.3.1 Message Queue

The cstrHdlr thread creates a message queue for incoming messages. The message queue is destroyed before the thread exists.

3.3.3.2 Memory Pool

Once it is done acting upon the incoming message the cstrHdlr thread releases the message buffer in which it was stored. The cstrHdlr thread allocates the memory block required to store the reply data for the current reply message.

3.3.4 Message Logging

Many of the error messages logged by the cstrHdlr thread are generated and logged by utility functions. But, for messages logged here, the cstrHdlr thread will log all status and error messages using the slcCmlogLogMsg utility (described in the slcCmlog Utilities section of the General Purpose Utilities document).

The cstrHdlr logs messages for the following conditions:

· Service availabilty change

· Invalid function code

· Unable to queue a reply

· Queue full

· Probably more

3.3.5 Diagnostics

See the requirements document for a list of diagnostics that are updated and reset upon demand.

3.3.6 Major Routines

This code exists in ioc/cstr/cstrHdlr.c and cstrHdlr.h

1. void cstrHdlrThread(void)

cstrHdlrThread is the main procedure and loop of the cstrHdlr thread. It initialized the cstrHdlr resources and globals, and waits for message at the message queue. It cleans up resources when terminating.

2. epicsBoolean cstrHdlrInitChk1(void)

This is a static function (called only by another function within this .c file).

Call dblistalloc(), dblist() are called to set up lists for later access to CTIM, UTIM, and ELPS (one value per function). Pointers to lists are in file scope for use by other utilities. Dblget() is called to get initial values which may be overwritten later. If any calls fail, return epicsFalse (failed), otherwise epicsTrue.

3. epicsBoolean cstrHdlrInitChk2(void)

This is a static function (called only by another function within this .c file).

Call dblistalloc(), dblist() are called to set up lists for later access to NRUN, FAIL, PUPD, and PVAX (one value per function). Pointers to lists are in file scope for use by other utilities. Dblget() is called to get initial values which may be overwritten later. If any calls fail, return epicsFalse (failed), otherwise epicsTrue.

cstrHdlr Check functions

4. epicsBoolean cstrHdlrChk1(Boolean dbUpdateNeeded)

dbUpdateNeeded is an output parameter.

Call cstrProcMicrSts() to process and update CSTR secondaries AMSK and MSTA.
Call slcAsyncProcChk1() to update dblget data and do dblput() for UTIM, CTIM, and ELPS.
Unlike RMX, don't call slcAsyncSetFcnStats(Function_enum) to set statistics in the Global Area for this async function. It's called in the main routine.

Set dbUpdateNeeded = true (we always do dbupdate for this check function)

Call an async utility to set cycleTimeStamp for this function if there were no failures.

Any failures in the logic above cause epicsFalse (failed) to be returned. Otherwise epicsTrue (success) is returned.

5. epicsBoolean cstrHdlrChk2(Boolean dbUpdateNeeded)

dbUpdateNeeded is an output parameter

Call slcAsyncProcChk2(dbUpdateNeeded) to dblget the data as needed and do dblput() for , FAIL, PVAX, and PUPD.

Do dblput() on all lists where the data has changed only.
Unlike RMX, don't call slcAsyncSetFcnStats(Function_enum) to set statistics in the Global Area for this async function. It's called in the main routine.

Call an async utility to set cycleTimeStamp for this function if there were no failures.

Any failures in the logic above cause epicsFalse (failed) to be returned. Otherwise epicsTrue (success) is returned.

6. epicsBoolean cstrHdlrChk3(Boolean dbUpdateNeeded)
This is a future CHK function. The current values of TBD are fetched from TBD and copied to TBD EPICS PVs. NO SLC database update is done so we returndbUpdateNeeded = False.

7. epicsBoolean cstrHdlrInitMicrStatus().

Initialize data items and dblists associated with micro (ioc) status after startup.

This function is only called by another cstrHdlr routine. So, it is static file scoped.

Call dballoc() and dblist() to set up lists for CSTR,[this_micro],1:AMSK, MSTA.
Likewise, do the same for CPU, and RMX (one per SLC IOC) for use by CPUM check function.
(Those CSTR secondaries are one per SLC IOC)
The lists are in file scope for future use cstrProcMicrSts().

CSTR,[this_micro],1,[CTRS, CRTT, CRVn, CAM, NTIM, and TSTA, ] are set in the database once using dblistalloc(),dblput(), slcAsyncMeterDbSend(forceUpdate),dblistfree(). They are never set again.

Do a dballoc(), dblist(), dblget(), and dbfree() for JMSK. Store JMSK in file scope for later use function.

Epics PV's (IOC::1:CPU and IOC::1:MEM) will be accessed for use in calculating SLC database CPU and RMX CSTR secondaries and the initial values will be written to the SLC database using dblput().
Perform Initialization for access to these Epics PV's as follows:

o call EPICS dbNameToAddr() to get the pointer for each PV and EPICS dbGet() for the the initial value. dbGet requires a dbrType argument. That argument will be hard coded based on what's in primary.dbs for the particular secondary (alternately, it would be possible to determine the proper value using a yet unwritten but handy utilitiy in database utility dbget.c called dbgetDbrType that converts from dbgetFormat and dbgetWidth to an EPICS dbrType).

o A bad status return from any db routine is cause for exit and proper cleanup (dblistfree's, reset "active" flag, etc).

o The PV name format will be:
CSTR:(4 char ioc name):1:(secondary name of interest).
The 4 char ioc name is determined using executive routine slcGetIOCName().

A bad status return from any of these db calls will result in a call to cstrHdlrExit() and return.

Call cstrProcMicrSts(). If bad status is returned, call cstrHdlrExit() and return.

The value of CSTR:VTIM is echoed to VMS in CSTR:MTIM as an indication that the micro's database has been downloaded. This is accomplished by calling an local cstrAsync function which will do a dblistalloc()/dblist() for each secondary, dblget() for CSTR:VTIM, dblput() putting the VTIM value to the database using the MTIM list. Dbupdate() is done below by MeterDbSend().

Call slcAsyncMeterDbSend(forceUpdate=yes) to do a non-metered dbupdate() to the VMS database for all the data we set into the lists above.

Call Dblistfree() to free the pointer and data lists for MTIM and VTIM.

8. epicsBoolean cstrHdlrProcMicrSts()

This function processes AMSK and MSTA per the table above.

This function is only called by other cstrAsync routines. So, it is static file scoped.

Clear local AMSK bit array.

Go through slcThread_as and set a bit in AMSK for each active job. An active job has the active flag set in slcThread_as for all the threads that it has. At this time, that is only one thread (the Hdlr). If it’s not set, then the bit is not set in AMSK for that job.

Call dblput() to put local AMSK into CSTR:AMSK.
If AMSK != JMSK, set MSTA to BAD(0)
ELSE set MSTA to ALIVE(1).
Call dblput() to put local MSTA into CSTR:MSTA
When it is noticed that a thread has exited (ie AMSK is different than JMSK):

o It is desired to have users notice the problem and attempt a restart manually (and if the thread that's dead is not critical, then they could time the restart to be less invasive). So, no automatic restart will be attempted. BTW, this is the way EPICS works - when a task suspends or exits, it doesn't crash the IOC. Instead people start noticing (sometimes subtle) problems and notify controls. Doesn't happen very often.

o A message will be logged indicating the thread that has died. This message is logged every time through as a reminder that repair and restart are needed.

o An automatic SLC restart should not be attempted since the fatal error would probably just happen again.

9. void cstrHdlrExit(void)

This routine is called just before the thread terminates. Call messageQRelease utility to destroy the message queue, which first reads out each message in the queue and releases the message buffer it points to, then destroys the queue.

. It sets the active flag to false and destroys the cstrHdlr message queue.

10. void cstrHdlrDoStop(void)

This routine is called when the IOC_STOP message is received. Call cstrHdlrExit() and return.

11. void cstrHdlrDoEcho(msg_ps)

This routine is called when the TEST_ECHO or TEST_FUNC message is received.

· Allocate buffer for reply message using the msgQGetSmallBuffer utility

· Create reply data; converting for VMS

· Copy the source SCP ID from the request header to the destination SCP ID used by message logging. Reset after reply is queued.

· Copy the source and destination from the request header to destination and source, respectively, in the reply header. Copy the VMS timestamp and function code from the request to the reply header. Set the proper data length in the reply header.

· Release the incoming message buffer back to the pool

· Send message to the msgSend thread using the msgQMsgReply utility.

12. void cstrHdlrDoEchoMWord(msg_ps)

This routine is called when the TEST_ECHO_MWORD is message received. Reply with blocks of repetitions for given word.

· Allocate buffer for reply message using the msgQGetSmallBuffer utility

· Create reply data; converting for VMS

· Copy the source SCP ID from the request header to the destination SCP ID used by message logging. Reset after reply is queued.

· Copy the source and destination from the request header to destination and source, respectively, in the reply header. Copy the VMS timestamp and function code from the request to the reply header. Set the proper data length in the reply header.

· Release the incoming message buffer back to the pool

· Send message to the msgSend thread using the msgQMsgReply utility.

13. void doErrMeterReset(msg_ps)

This routine is called when the TEST_ERR_METER_RESET message is received. It will call TDB routine to reset CMLOG metering. A reply will be sent with throttling status.

---

SLC-Aware IOC Home Page | LCLS Controls | EPICS at SLAC | SLAC Computing | SLAC Networking | SLAC Home

Contact: Ron MacKenzie
Last Modified: Apr, 2005. by RONM.