Review of Epics Libraries and Resource Usage
The LCLS slc-Iocs are being developed with a combination of independently programmed slc threads and Epics Operating System Independent libraries (libCom and libcom OSI). The independent slc threads may be stopped and started repeatedly while the IOC itself remains up. Concern was expressed in controlling and managing resources used by these threads, such as heap memory, message queues, semaphores, socket and file descriptors, and threads; including those allocated within the Epics libraries.
The libCom and libCom OSI libraries are documented in the IOC Application Developer's Guide. The documentation, in most cases, is a quick summary of the library utilities usage and the interface. Generally the documentation does not cover resource usage within the libraries.
This report is an overview of resource management within the Epics Libraries used (or may be used) by the slc-Ioc developers. The report ends with a set of suggestions for resource management within the slc-Ioc threads.
LIbCom Utility Libraries
The epics libraries have been written well in terms of resource management. Those utilities that allocate resources such as heap memory and mutexes usually have a ‘create’ or ‘init’ routine to perform the allocations, and a ‘destroy’, ‘free’ or ‘release’ routine that de-allocates. It would be good practice for the programmer to review the source of any epics library utilized, and ensure that the library routines are used properly and the de-allocation routines are called.
The epics libcom and LibCom OSI sources are in the directories
The following sections give an overview of the utilities that may be useful for the slc-Ioc threads. They are listed by their directory under /libCom.
Hash facility for integers, pointers and strings. Used by Channel access. This library uses the freeList and epicsAssert utilities (see below) which protect the heap memory allocated by the bucketLib routines, and will ultimately suspend the calling thread if any memory allocation errors occur.
memory manager for applications that allocate and free memory frequently. dbmf uses ellLib as list utility, and uses epicsMutex to protect it’s list and nodes during the function calls, and will suspend the calling thread if any memory allocation errors occur.
A general purpose linked list library. This library uses epicsAssert (see OSI below). It DOES NOT use mutexes, or any method, to protect the list or nodes in a multi-threaded environment.
A general purpose freemem list library that uses epicsMutex to protect its list during function calls. The thread uses cantProceed, which will suspend the thread if a memory allocation error occurs.
A general purpose hash utility. It uses ellLib for the list, and epicsMutex to protect the list during function calls. It uses cantProceed and epicsAssert which will suspend the calling thread for memory allocation and assert errors.
It should be noted that the mutex does not protect the memory linked into the list by the calling thread once the gpHash function has completed. This is true for any of the hash, list, or freelist libraries the slc threads may use.
As an example problem:
Some method of protecting the structure must be added, possibly similar to the message queue technique of copying the data, rather than just returning a pointer. See the Resource Management … section below for further suggestions.
A set of miscellaneous include files, functions, and definitions. Documented in the IOC Application Developer’s Guide. Some that look useful for slc threads:
This directory includes the operating system independent libraries (libCom OSI) that ultimately have different implementations depending on the operating system being used. Mutexes, threads, timers etc. are implemented dependent on the capabilities of the underlying operating system. The subdirectories, by OS name, contain the OS dependent implementations.
The following libraries are included:
See documentation in the IOC Application Developer’s Guide. epicsRingBytes and epicsRingPointer do not employ any method to protect the ring buffer in a multi-threaded environment. Calling threads should implement with read and write locking Mutexes, unless there is a single reader and / or a single writer.
Contains test programs for several of the libCom utilities.
a Timer facility. This is very well documented in the IOC Application Developer’s Guide.
Resource management, thread safety, and the C programming language
On most operating systems, in a multi-threaded environment, global variables are available to all threads within a process, unless defined ‘static’. A static variable is global to all functions below the definition point within the file it is defined. Heap memory is shared by all threads within a process. Other resources, such as sockets, file descriptors, mutexes, and message queues are accessible by all threads if their references are assigned to a global structure. The following are some suggestions when programming for thread safety and resource management.
Heap memory allocations
Other recommendation should be added here as the program evolves.