Communication with the SLC Control System passes either through the Database Service or through the SLC Message Service. The SLC IOCs must mimic the iRMXmicros in control and timing, to minimize changes in the SLC Control System. This document discusses the design of the Linac Coherent Light Source (LCLS) Stanford Linear Collider (SLC) Input/Output Controller (IOC) Database Service (DBS). Specifically, the Database Service design, Database Input/Output Utilities, and Console commands are discussed. The database service implementation described herein could be incorporated into any SLAC subsystem that communicates with the SLC Control System and utilizes the EPICS toolkit for device control.

2 Introduction

The SLC IOC emulates many aspects of the SLC iRMX micro (micro), including maintaining its part of the VMS SLC database. Note that “database” is used to describe a “flat file architecture” throughout this document, and does not refer to a commercial or other relational database. Like a SLC micro, the SLC IOC DBS receives a piece of the entire SLC operational database at initialization time whose functionality is configured for it by the dbgen/dbinstall utilites running on the Alpha. Data is transmitted in a series of byte streams, called supertype (ST) blocks, or superblocks, over a TCP/IP LAN from the DBEX process running on the Alpha.

The data sent to the SLC IOC (and micro) is divided into four sections, physically in memory, according to its function and read/write attributes. Supertype 0 consists of pointer information (actually 4 byte offsets into the other ST 1-3 data blocks) by which each primary/unit/secondary name gets translated into memory address. (Primary/unit/secondary names are part of a naming convention which identifies a particular device.) Supertype 0 information gets written on the Alpha only at dbgen/dbinstall time. The other three Supertypes (1-3) of concern to the SLC IOC contain actual data. Supertype 1 is assumed to be stable, only changeable by the Alpha, and consists of configuration values. Supertype 2 data consists of setpoint values; any update of a Supertype 2 item in the control system originates only in VMS and causes both an update in memory in VMS and a message to the micro/ SLC IOC to do the same update in the micro/ SLC IOC copy of same memory. For the SLC IOC (as opposed to a micro), the EPICS database can also update Supertype 2 data, causing the SLC IOC to update both its local copy in its SLC database as well as send the Supertype 2 data to the Alpha. Any update of a Supertype 3 item originates in the micro/SLC IOC and causes an update in the micro's memory. A separate call sends accumulated updates to the Alpha to be stored in the corresponding VMS memory by DBEX. Any update of a supertype 4 item originates only in VMS and is never seen by the micro/SLC IOC (i.e., any micro/SLC IOC’s supertype 4 data resides only in VMS memory for use by application code in VMS).

Those familiar with the LINAC and PEPII iRMX micro computer code will recognize that most all “dbmain” functionality has been ported to the SLC IOC DBS to run on top of an EPICS OSI layer. The dbmain functionality has been split between dbRecv, dbSend, and dbHdlr threads, responsible for receiving, sending and handling data, respectively, from/to the Alpha.

2.1 Background

The Database Service provides most, if not all, of the functionality contained in the iRMX micro’s REF_:[RMX.DBS] DBMAIN.C module.

2.1.1 General DB Service Design Concepts

· Amount of change to VMS should be minimal.

· Use existing VMS include files wherever possible; refer to VMS directory REF_:[RMX.INCLUDE] *

· Port existing iRMX application code (c, fortran, assembly) wherever applicable.

· Goal – generate same amount of network traffic as SLC micros

· More memory is available for use on SLC IOC – use to advantage for database storage to make database access quicker, if possible.

· Higher performance CPUs are being used, compared with micros – use to advantage

· Use EPICS libcom OSI libraries and platform independent libCom facilities to reduce coding time and to benefit from collaboration knowledge

· Account for different platform byte orientation. VMS packs; solaris and vxWorks (and many 32 bit architectures) byte align on 4-byte boundaries.

o The Supertype byte stream header (supertypehdr_ts), shared between VMS and all micros/SLC IOCS, is problematic, as it has three 2-byte words followed by four 4-byte words. Supertype header sent to/from VMS contains no padding.

· Messages from VMS control system are little-endian, VMS format, and packed

· Support ASCII representations for primary, secondary names and units

· SLC IOC needs supplementary data definition:

o primary ASCII to #

o secondary (secn) ASCII to # mappings for given primary

o datum width (for I or Z)

2.2 References

References for various areas are listed below.

DBEX:

(References for handling data traffic between the SLC IOC and DBEX process executing on the Alpha control system):

· Use VMS procedures in REF_:[RMX.DBS] DBMAIN.C and REF_:[RMX.SLCLIB] exnet.c as the design basis for interface to the DBEX Server process on the Alpha.

· Use VMS assembly code REF_:[RMX.DBS] DBLISTU.A38 as the design basis for dblist and dbunits for accessing SLC supertype 0 database (used in dbIPL.c)

· Also, talk to the experts – Tony Gromme, Ken Underwood, Nancy Spencer, Ed Miller, Ron McKenzie, Ron Chestnut, Stephanie Allison, Terri Lahey.

EPICS references :

· IOC Application Developer's Guide, Release 3.14.6

· Where $LCLS = /afs/slac.stanford.edu/g/lcls/vol1/epics/site/src/slc, use $LCLS/msg/dbSend.c as a design reference for a queued thread and $LCLS/msg/msgRecv.c as design reference for a socket reading thread.

Also see SLC Executive and Message Service Design Presentations

Legacy SLC Database:

POOP: Database Internals
BUG: Database
PG: Database
Introduction to the SLC Database by Ken Underwood, 1988
Notes on Existing SLC Operational Database by Tony Gromme (includes notes on TCP Proxy, 3/2004)

SLC-Aware IOC:

SLC-Aware IOC webpage

SLC IOC Message Service Design

SLC IOC Utilities

2.3 DB Service Functional Requirements

Please refer to: SLC Database Service (DBS) Functional Requirements , by Stephanie Allison, for the complete list of Database Service requirements.

2.3.1 DB Service Use Cases

DBEX

“Look like” a tcp micro – SLC IOC has the same functional requirements as an iRMX micro
Same message formats, traffic, use proxy
Req and Accept database (“IPL”)
Accept updated setpoints from Alpha
Provide updated data from IOC

Supertype 3 (ST3) readbacks
Supertype 2 (ST2)setpoints (new)

Accept and reply to diagnostic inquiries
Accept and Process DBEX up/down messages (up contains db version) (new)

Console Users

Utilities to edit, dump, get type of various slc data

Application code

Utilities to find, get, put, update slc data
Coded “data flow dependancies”

“Job” threads updating Alpha with ST data should wait for round trip confirmation that DBEX received it prior to acknowledging back to Alpha via the message service

This dependency surfaced in the design review

??? What else

As each “job” specification (bpm, magnet) becomes available, make sure that DBS meets its needs

3 Design

3.1 Service Description and Block Diagram

The primary function of the DBS is to keep its portion of the SLC database - shared data between the Alpha Control System and the SLC IOC – up to date. It accomplishes this by communicating to the DBEX process on the Alpha Control System. All of the database message communications between the SLC IOC and Alpha Control System is handled by the dbSend and dbRecv threads on the SLC IOC, and the DBEX process running on prod or dev Alpha. The SLC database transactions between VMS and SLC IOC occur through a TCP proxy.

There are two TCP/IP sockets per SLC IOC that handle DBS communications (“dbex” and “ctl”). These sockets are connections to a proxy, which also maintains two socket connections with DBEX. The proxy forwards data from DBEX to the proper SLC IOC after interpreting a proxy message “forward” header. Thus, the proxy allows DBEX to make two single socket connections in order to communicate with many micros/SLC IOCs.

The dbex socket is bi-directional and handles initial database downloads, updated data traveling in both directions, diagnostic messages, acknowledgements, and dbex up/down messages. The ctl socket is uni-directional and exists solely for DBEX to acknowledge back to the SLC IOC/micro that it received an update to the database.

DB Service consists of the dbSend thread, dbHdlr thread, dbRecv thread, database Input/Output utilities, database Console Commands, and DBS Utilities. The DB Input/Output utilities are global to all threads (refer to Specification for Database Input/Output Routines), while the DBS Utilities are used only by the db* threads and generally contain conversion utilities.

The Device Service function is shown in the DBS Block Diagram

3.1.1 DB Service Block Diagram

The functionality contained in file REF_:[RMX.DBS] DBMAIN.C has been split into dbRecv and dbSend threads, while dbHdlr thread handles new data manipulations for the SLC IOC. The dbSend and dbHdlr threads are “queued message” threads, primarily taking direction from messages sent to their respective message queues. The dbRecv thread is considered a “socket” thread, as it primarily takes direction from incoming network messages from DBEX on the dbex socket.

dbRecv’s prime responsibility is to wait at the dbex socket for network messages from DBEX. It responds to incoming data messages by storing data, in VMS format, as well as by sending queued messages to dbSend to acknowledge DBEX back (when DBEX requests, via a flag in the supertype header) on the dbex socket. Note that currently, acknowledges back to DBEX are only requested by DBEX during IPL, and not during ST 2 data updates. After the SLC IOC receives a database download at startup, it sends a message to dbHdlr to convert the data.

dbSend takes direction from queued messages sent from the slcExec, dbRecv, and “device service” threads. As part of carrying out its functions, it sends all data to DBEX via the dbex socket. For outgoing network messages consisting of supertype 2 or 3 updated data, it requests DBEX to acknowledge back and waits for that acknowledgement on the (dedicated) CTL socket immediately after sending the message.

dbHdlr’s prime responsibility is to offload (large amounts of) data handling from dbRecv or dbSend. It initializes DBS globals and waits at its queue for the message from dbRecv to convert data (compressed ST0 data to string-based format). Upon DBS restart or exit, it resets globals and releases resources. It is desired to be able to stop/restart the DBS and redownload the database while the EPICS application code, co-existing in the SLC IOC, continues to run. dbHdlr coordinates these efforts.

3.1.2 External Interfaces

The following sections describe the ways in which to access the SLC IOC DBS.

3.1.2.1 DBS Sockets

The SLC IOC communicates with DBEX, the Database Executive process that runs on the SLC Control System under VMS on the Alpha. It uses a byte stream composed of a proxy header (intended for proxy use) followed by data service header (or supertype header, interpreted by DBS) followed by data (for data-carrying messages). These messages are tcp/ip, and are managed by a proxy host. The proxy host establishes TCP/IP connections with DBEX in order to communicate with many micro/SLC IOCS using their socket connections. The two DBS socket connections on each SLC IOC are:

Dbex socket
Ctl socket

Both dbRecv and dbSend threads are responsible for the bi-directional communications on the dbex socket, while dbSend communicates with the uni-directional Ctl socket.

3.1.2.2 dbSend Queue

When any job thread or other utility decides to update the Alpha with a job’s updated data, it calls the reentrant utility, “dbupdate(job)”, which sends the “DB_UPDATE” message to dbSend message queue.

dbRecv also sends messages to dbSend Queue in order to have it send DBEX acknowledgements for supertype data, as well as diagnostic requests.

3.1.2.3 DBS Input/Output (I/0) Utilities

General database input/output utilities direct the reading and writing of data from / to the slc database and application code (other job threads). See Database I/O Utilities.

3.1.2.4 DBS Console commands

Console commands entered at the EPICS iocsh, RTEMS Cexp shell, or vxWorks shell, access the database (eventually via the DBS I/O utilities above) for the user.

See Console Commands API.

3.1.3 Data Structures

For the SLC IOC, it is desired to be able to stop and restart the SLC-only threads while the EPICS applications continue running. Furthermore, it is desired to redownload the SLC database upon command, after which device threads reinitialize and clear their dblists. In order to prevent heap memory fragmentation, most data structures which are allocated by DBS will be under control of a memory pool manager, namely the EPICS freeListLib.

3.1.3.1 Headers : Proxy and Supertype

The byte stream messaging between DBEX and the SLC IOC always contains a fwd_hdr_ts proxy header which the proxy interprets, and a dbsuptypehdr_ts supertype header, which the Alpha DBEX process and SLC IOC db* threads use to communicate back and forth. (The supertype header is equivalent to the Message Service message header.)

Name	Data type	Represents
fwd_hdr_ts
ip_port_u len user cmd crc	ip_port_tu int4u user_field_ts int1u int1u	Lower half of the ip address, and the port number Message bytecount - this fwdheader User defined; chunk count for large buffers; Fwd_server command, e.g. 8 bit crc over header; currently set to 0x55
dbsuptypehdr_ts	or	dbsuptype_tu = union
network netlsn id <<<<< Note len micr blkbeg blkend	int2u int2u int2u potential byte int4u Int4u int4u int4u	Type of data alignment mistmatch here >>>>> Size of superblock Microname Copy to location in superblk Copy from location in superblk
datw[ ]	Array of int2u	type-specific data; packed; max size is NETBUFLEN minus sizeof(dbsuptypehdr_ts)/2

The format of the data (datw[]) following the supertype header is dependant upon flags in the supertype header. The dbsuptype_tu definition makes a union of the supertype header (dbsuptypehdr_ts) and the data, such that any element of the supertype header or the data following can be accessed as a 16-bit data word. (This is important when unpacking the packed VMS formatted supertype data into native, structured format.)

3.1.3.2 Supertype 0 Byte Stream

The legacy VMS ST 0 byte stream, comprised of pointers to primary/unit/secondary data, contains a hash table followed by a series (X number) of unit/primary/secondary (UPS) blocks. Each UPS block is followed by its associated nsub number of secondary blocks (secn). In order to save memory on the legacy micros, numbers were associated with each primary, unit, and secondary name in lieu of storing the 4-char ASCII representations for each. In the SLC IOC, ASCII names will be stored instead of numbers. The ASCII representation facilitates programmers and is necessary for console users when typing in primary/unit/secondary names. The ASCII name representations, along with its associated secondary data information (from ST0 block and file on NFS) is translated into a “dictionary” for the SLC IOC (refer to Dictionary).

fwd_hdr_ts proxy header

Dbsuptypehdr_ts supertype header

hash length

array listhead ptrs

array node counts

UPS block 0

secn block [0 – nsub]:

supn | subn

fmt | slen

sptr =

offset in supn

…

….

UPS block X

secn block [0 – nsub]

Where a UPS block contains:

3 links – next UPS, unit,

collided nodes

len - ups + nsub’s

catn - prim #

unit - unit #

nsub - # of secns

3.1.3.3 Supertype 1-3 Byte Stream

Supertype 1-3 byte stream data consists of actual data values (pointed to by ST 0).

fwd_hdr_ts proxy header

Dbsuptypehdr_ts supertype header

Data

data

Data

data

….

3.1.3.4 Send/Receive Buffer Structure

The dbRecv receive buffer and the dbSend send buffer, both allocated in their respective threads, have similar structure to the overall byte streams. They maintain both headers and contain data, all of which are in VMS format. Upon receiving the data stream into the receive buffer, dbRecv converts the incoming dbsuptypehdr_ts into native format so that it can check the contents of the message. The native format is stored in the thread’s queued message variable, the structure of which is shown in the next section.

Name

Data type

Represents

Dbsndrcvbuf_ts

Receive and send buffers

proxy_hdr

supblk

Fwd_hdr_ts

dbsuptype_tu

Proxy header

Superblock header + data

3.1.3.5 DB Queued Message Structure

The dbmsgmail_ts contains all information required when communicating from db thread to db thread. The msgheader_ts contains the message ID (function). The nativeSB contains the native supertypehdr_ts, converted in dbRecv. The diagnostic data field is to service the diagnostic message received by dbRecv from DBEX; the reply is formatted and sent to DBEX from dbSend.

Name

Data type

Represents

Dbmsgmail_ts

Q messages

Msgheader

nativeSB

dbdiag

msgheader_ts

dbsuptypehdr_ts

dbdiagmsg_ts

Msg header

Native superblk hdr

Diagnostic data

3.1.3.6 Dictionary

The dictionary is a string based hash table whose entries contain exemplary information (meta data) about each primary/unit/secondary (PUS) data, namely:

Supertype number
Format (I,A,R,Z,S,T)
Length, or number of data elements (V= variable)
Data width (1,2,4,8 bytes)
Offset into supertype block (when added to that supertype block’s memory pointer, this becomes the address of the data locally)
Secondary number (secn), this is no longer used by SLC IOC, but is copied from the ST0 subblock and stored anyway

3.1.3.6.1 PRIMARY.MAP file

The PRIMARY.MAP ASCII file is created as part of the dbGen/dbInstall process on the Alpha. It is then released to a TBD production area on NFS in order for the SLC IOC to read it. On each line it contains primary name, primary number (catn), secondary name, secondary number (secn), format, width of data (bytes), and number of data items. Note that for each primary, secondary numbers are relative to their associated primary. A secondary of one primary may have a different number than the same secondary name of another primary.

The PRIMARY.MAP file is created so that the SLC IOC can a) convert primary and secondary numbers into their ASCII name representation, and b) have access to the data width information for each secondary. Primary names that do not exist in the SLC IOC database are still entered in the dictionary for name validation.

The native dictionary structure is created from a) the converted number-oriented, compressed, VMS formatted superblock 0, and b) by reading the PRIMARY.MAP file, available on an nfs mount point at location pointed to by environment variable.

PRIMARY.MAP format:

Primary catn Secondary secn format width number

QUAD 1 BMON 135 R 4 1

QUAD 1 KTIM 31 T 8 1 …

NOTE: file lines containing any given primary name must be contiguous in the file; the algorithm used by dbST0CreateDict() depends on this contiguousness.

3.1.3.6.2 Dbhash from gphash

For hash facilities, the dictionary uses EPICS libCom gphash (general purpose hash) as a base for its own dbhash facility. gphash sets up an internal hash table of ‘tablesize’ and returns a pointer to it. It also maintains a mutex which is not needed during a) single-threaded write-only initialization while other threads are locked out waiting for downloadEvent, and b) read-only access during operations. (This mutex requires less than 2 usecs when the mutex is available, so it doesn’t warrant modifying the gphash facility just for selectable mutex locking, unless we call it more than 100 times / second.)

However, gphash doesn’t use memory management facilities, thus it was decided to copy gphash to dbhash and modify it to use routines available in freeListLib. Dbhash will use memory pools for the hash array (ELLIST[tablesize]), and for each GPHENTRY. It will remove the mutex locking and it will substitute the strcmp call with strncmp such that hash strings will not need to be null-terminated.

3.1.3.6.3 Four use cases

When organizing the way data is accessed in the dictionary, there are four possible ways to arrange the data because there are four use cases that fall out of the various database input/output and shell routines. A primary (P) must always be specified, while the other two arguments, unit (U) and secondary (S), may be any combination of ALL*, or specified.

The four use cases are outlined in the bullets, and figure below.

PUS - prim, unit, sec all known; ie dblist(lp, p, u, s)
PU* - prim, unit, all secs; ie dbdump(p,u)
P** - prim, all units; ie dblunits(lp, prim) or dbdump(p)
P*S - prim, all units, sec; ie dblist (lp, p, ALL*, s)

To implement all four optimizations would be costly in memory and may be overkill. At a minimum, the P** case should be implemented, as it can handle all four use cases. Shell commands do not need to be optimized, as they occur at “human speed”, so PU* does not need to be implemented. The P*S case can be handled by P** (not quite as efficiently), and will be re-evaluated for implementation when we address how often the magnet and bpm jobs receive SCP requests with ALL* for units. For those threads that respond to a specific SCP request, they will be implementing the PUC use case during runtime. This means that the P** and PUC cases will be implemented, at a minimum. Every “primary” in PRIMARY.MAP will be entered into the hash table, regardless of its presence in this particular SLC IOC for primary name error checking. Each “primary secondary” doublet from PRIMARY.MAP could also be entered for ease of secondary name error checking; at the moment this is a TBD.

3.1.3.6.4 Planned Dictionary Structure

The following figure shows the minimum Dictionary implementation:

3.1.3.6.5 DBHENTRY structure – dictionary entry

Each dbhash table entry is of type DBHENTRY. The DBHENTRY is defined as:

Name

Data type

Represents

DBHENTRY

Hash table entry

node

name

userPvt

ELLNODE

Const char *

Void *

Link node

“primary unit secondary”

Pointer to either:

dbdictPUS_ts or dbdictP_ts

‘name’ in the above structure maintains our “hashing string” and points to a dictNameTriplet_ts (for PUS name) or points to a dbdictNameSgl_ts (for a single P, U, or S name), while userPvt points to either each dbdictUPS_ts secondary meta data entry or each dbdictP_ts primary entry.

3.1.3.6.6 dbdictNameTriplet_ts

For PUS, the “hashing string” consists of the unique “primary unit secondary” string. Memory is allocated for the dbdictNameTriplet_ts and a const char * pointer is established with the following:

Name	Data type	Represents
dictNameTriplet_ts		Triplet PUS name string
Name[12]	Char	Non- null-terminated “primary unit secondary” string name; length = 4-4-4 = 12 bytes

Note that gphash uses “strcmp” which requires use of null termination for the name string. This will be changed to “strncmp” in dbhash, such that we have char[12].

3.1.3.6.7 dbdictNameSgl_ts

For the hashing string used with primary, unit, or secondary name only, the following structure is used:

Name	Data type	Represents
dbdictNameSingle_ts		Single P-U- or S name string
Name[4]	Char	Non null-terminated “primary” or “unit” or “secondary” string name; length = 4 bytes

3.1.3.6.8 PUS entry – dbdictPUS_ts

Each DBHENTRY entry that points to a dbdictNameTriplet_ts also has its userPvt pointer pointing to a dbdictPUS_ts structure that contains the secondary meta data. The “primary unit secondary” meta data has the following structure definition:

Name

Data type

Represents

dbdictPUS_ts

SLC Secondary data type

supn

count

sptr

width

format

short

uns