LCLS Controls
SLC-Aware IOC Design
James Silva
– ESD Software
February 15,
2005
CMLOG Service
The following document describes a software design for the Input-Output Controller (IOC) Common Message Logging (CMLOG) (iocCmlog) service which can be used by both SLC-Aware IOCs and non-SLC IOCs. The document includes a description of the logic and design of the iocCmlog service.
The CMLOG software package was developed by Jie Chen at Jefferson Lab (chen@jlab.org) and is used by several
laboratories, including SLAC.
The CMLOG client is currently used by non-SLC vxWorks 3.13 IOCs to log messages. Messages are sent to a CMLOG server running on one of the UNIX machines via a client daemon running on the IOC, and the message log can be browsed from any UNIX session by starting the CMLOG browser.
CMLOG was chosen to be the message logging service for SLC-Aware IOCs for various reasons. It is already in use by the vxWorks IOCs and, by programming the iocCmlog service, we do not have to port the SLC micro errlog code to accommodate the new SLC-Aware IOCs.
The iocCmlog service replaces cmlogVxLogMsg delivered as part of the JLAB CMLOG package. It is separate from the iocLog service delivered as part of EPICS base, which uses a different mechanism to store and browse messages. Both iocCmlog and iocLog may run on the same IOC at the same time with EPICS-only messages going to both the iocLogServer and the cmlogServer. The SLC-Aware IOC messages will ONLY go to the cmlogServer.
CMLOG Programmer’s Guide
http://www.jlab.org/~chen/cmlog/cmlog-2-1-b/index.html
EPICS App Dev Guide
http://www.aps.anl.gov/epics/base/R3-14/6-docs/AppDevGuide.pdf
SLC-Aware IOC Utility Routines
http://www.slac.stanford.edu/grp/lcls/controls/global/sw/slc_ioc/exec/slcUtil.htm
SLC-Aware IOC Functional Requirements
http://www.slac.stanford.edu/grp/lcls/controls/global/sw/slc_ioc/requirements/funcReqts.html
LCLS C Coding Standards
http://www.slac.stanford.edu/grp/lcls/controls/global/standards/software/codeStdsC.html
The iocCmlog service must be usable by both the SLC-Aware and EPICS IOC tasks. It must also run Solaris, Linux, vxWorks and RTEMS (CMLOG is currently not built for RTEMS, and messages logged on RTEMS platforms will simply be printed to the console until the RTEMS build is complete).
The iocCmlog service uses the EPICS message queue service to store and retrieve messages for logging. The EPICS message queue service is used for concurrency control and platform independence.
Code must be C and follow the LCLS C coding standards.
Please refer to Section 3.2.1 for a description of logical flow of data and control of the iocCmlog service.
iocCmlogStart – start the iocCmlog service
iocCmlogDStart – start the cmlog client daemon (vxWorks and RTEMS only)
iocCmlogHdlrQSize – set the iocCmlog message queue size
iocCmlogStop - stop the iocCmlog service
iocCmlogLogMsg – send log message to iocCmlog message queue
iocCmlogVLogMsg – send log message to iocCmlog message queue
Messages enter the iocCmlog message queue from two sources:
- IOC applications call iocCmlogLogMsg() directly. The iocCmlogLogMsg() function places the message on the message queue, and it is then picked up by iocCmlogHdlr().
- An EPICS applications calls errlogPrintf(), and the EPICS errlog task calls iocCmlogListener() whenever an EPICS message is issued. The iocCmlogListener() function places the message on the message queue, and it is then picked up by iocCmlogHdlr().
Once a message is retrieved from the queue, the iocCmlogHdlr() function calls cmlog_logmsg() to send the message to the CMLOG client daemon running on the IOC, which sends the message to the CMLOG server. Once the message reaches the server, it becomes viewable by the CMLOG browser applications.
The CMLOG client daemon, which receives messages from all IOC CMLOG applications and forwards them on to the CMLOG server, must be started before the iocCmlog service. For vxWorks and RTEMS (hard) IOCs, the daemon is started early in the IOC boot script, before EPICS databases are loaded, hardware configured, and IOC initialization. For hard IOC, the daemon may be started using the function iocCmlogDStart(). For soft IOCs, the daemon is started at machine bootup and remains up across IOC restarts.
The iocCmlog service is started by the function iocCmlogStart(), which can be started in the IOC boot script or at the console command line. Once it is started, IOC applications can log messages directly to CMLOG using iocCmlogLogMsg(), and EPICS applications can log messages via errlogPrintf(). The function iocCmlogHdlr() task registers the function iocCmlogListener() with the EPICS errlog service to log errlog messages to CMLOG.
Messages are then queued by the function iocCmlogLogMsg(). The function iocCmlogHdlr(), spawned by iocCmlogStart(), processes incoming messages and sends them to the CMLOG client daemon, which then sends the messages to the CMLOG server. Messages are optionally printed to the console and the printout includes the process name, status string, and formatted text string.
The iocCmlog service is stopped by calling iocCmlogStop(), which sends a “stop” message to the iocCmlog message queue.
Please refer to the diagram in Section 3.1.1 for a visual representation of this functional flow.
- iocCmlog message queue
- iocCmlog message queue size
- number of missed messages due to full queue
The iocCmlog service logs only 3 messages on its own:
- iocCmlogHdlr startup (may be disabled using a compiler switch)
- iocCmlogHdlr exit
- Messages discarded due to full queue
The following functions will reside in and be documented as iocCmlog.c, iocCmlog.h, and iocCmlog.html under $EPICS/site/src/iocCmlog. A library called libiocCmlog will be built by a standard EPICS Makefile in this directory and installed under $EPICS/site/lib/<target> for all EPICS targets. IOC applications will include this library in their builds. The iocCmlog.html file will be a standard EPICS distribution web page for iocCmlog and will have build and usage instructions plus a link to this design document. The iocCmlog.html file will be installed in the web directory by the Makefile. Once iocCmlog is finished, a link to iocCmlog.html will be added to the EPICS collaboration soft support page:
http://epics.aps.anl.gov/epics/modules/soft.php
At initialization, iocCmlogHdlr:
(a) Creates a message queue by calling epicsMessageQueueCreate. It uses a file-scope value (default to 100, changeable by iocCmlogHdlrQSize) for message queue size. The queue ID is updated in file-scope space which is initially set to 0. If an error occurs or the queue is already created, printf is called and the thread exits.
(b) Logs a startup message (disabled using a compiler switch) by calling iocCmlogLogMsg with facility_c = "EPICS", status = COS_ALARM, severity = NO_ALARM, doPrintf = 1, status_c = NULL, destination_c = null, filename_c = null, lineno = 0, format_c = "iocCmlogHdlr Startup".
(c) Registers the errlog listener callback by calling errlogAddListener with iocCmlogListener as the listener and a null private pointer.
(d) Finds the IOC name and uses it as host for later message logging. The result of getenv("IOC_NAME") is used. If the "IOC_NAME" environment variable does not exist, the host tag is not included in the call to cmlog_logmsg and cmlog_logmsg will set it.
(e) Calls cmlog_open to get a client handle and sets it to 0 if an error is returned.
It then loops waiting on a message using epicsMessageQueueReceive. For each message, it:
(a) Calls cmlog_logmsg with inputs from the message. The format string and arguments depend on which tags are available for this message. At a minimum, the tags include text, status, and severity. If the compiler switch is on that requires the code tag to be a string instead of an integer, then the code tag is included. If host could be determined during initialization, then the host tag is included. Other optional tags are provided if they could be determined for the specific message being logged and include codeI (for VMS style status codes), process, dest, file, and lineno.
(b) If the doPrintf flag is set for this message, a printf is also done with process name, status string, and formatted text string.
(c) If the status is COS_ALARM and the text is “iocCmlogHdlr Exit”, calls errlogRemoveListener, epicsMessageQueueDestroy, and cmlog_close. It then sets the queue ID to 0 in file-scope space and exits.
(d) If there are no messages pending in the queue (epicsMessageQueuePending) and there have been discarded messages by iocCmlogVLogMsg due to a full queue, then a message is logged by calling iocCmlogLogMsg with facility_c = "EPICS", status = WRITE_ALARM, severity = MINOR_ALARM, doPrintf = 1, status_c = NULL, destination_c = null, filename_c = null, lineno = 0, format_c = "iocCmlogHdlr = %ld messages were discarded", and the number of missing messages as the argument. The number of missed messages is then set to zero.
This is called from the startup file before iocCmlogStart and sets the file-scope queue size for the iocCmlogHdlr message queue. It cannot be less than 32 or greater than 10000. Note that if iocCmlogHdlr is already started, it will need to be manually stopped and restarted for the value to take effect.
This is called from the startup file or from IOC shell. If the iocCmlogHdlr queue ID is 0 in file-scope space, it starts iocCmlogHdlr using epicsThreadCreate and calls printf if error. Arguments to epicsThreadCreate are:
(a) Name = "iocCmlogHdlr"
(b) Priority = epicsThreadPriorityLow
(c) Stack Size = epicsThreadStackSizeMedium
(d) Funptr = iocCmlogHdlr
(e) parm = null
This is called from the vxWorks or RTEMS startup file. If the cmlogClientD is not already started (epicsThreadGetId), it starts cmlogClientD using epicsThreadCreate and calls printf if error. Arguments to epicsThreadCreate are:
(a) Name = "cmlogDaemon"
(b) Priority = epicsThreadPriorityLow
(c) Stack Size = epicsThreadStackSizeMedium
(d) Funptr = cmlogClientD
(e) parm = null
This is called from IOC shell. If the iocCmlogHdlr queue ID is not 0 in file-scope space, it calls iocCmlogLogMsg with facility_c = "EPICS", status = COS_ALARM, severity = MAJOR_ALARM, doPrintf = 1, status_c = NULL, destination_c = null, filename_c = null, lineno = 0, format_c = "iocCmlogHdlr Exit".
This is called by the EPICS errlog task for every EPICS message. It calls iocCmlogLogMsg with facility_c = "EPICS", status = NO_ALARM, severity = ALARM_NSEV (to default to INFO), doPrintf = 0, status_c = “INFO”, destination_c = null, filename_c = null, lineno = 0, format_c = input text.
This is called during EPICS and registers iocCmlogHdlrQSize, iocCmlogStart, iocCmlogDStart, iocCmlogMsg and iocCmlogStop using iocshRegister so that they can be called from IOC shell. This routine requires a database definition file called iocCmlog.dbd with one line:
registrar(iocCmlogRegistrar)
IOC applications will need to include iocCmlog.dbd to use the iocCmlog service.
Lower 3
bits of Condition Code
|
Verbosity
|
CMLOG Severity
|
Severity
Tag
|
0 =
warning
|
3
|
1
|
WARN
|
1 =
success
|
5
|
0
|
SUCC
|
2 =
error
|
2
|
2
|
EROR
|
3 =
informational
|
4
|
0
|
INFO
|
4 =
fatal
|
0
|
2
|
FATL
|
>= 5
|
4
|
0
|
INFO
|
If the input severity is positive, it is assumed to be EPICS and verbosity is determined from this table:
Input
EPICS Severity
|
Verbosity
|
CMLOG Severity
|
Severity
Tag
|
0
|
5
|
0
|
NO_ALARM
|
1
|
3
|
1
|
MINOR_ALARM |
2
|
2
|
2
|
MAJOR_ALARM
|
3
|
1
|
3
|
INVALID_ALARM
|
>=4
|
4
|
0
|
INFO
|
Maximum size is 32 characters.
(l) Bit Mask
Bit mask with bits set for doPrintf, EPICS (don’t use the codeI tag) or VMS (do use the codeI tag), process name available, destination available, and file name available.