Message Passing Facility
MPF provides client server message passing. The messages are handled by message routers. Both local and remote message routers are provided.Industry Pack Support
Industry Pack modules are supported via drvIpac, which supports dumb VME carriers as well as the mv162 and mv172. For details see drvIpac.Epics Device Support for the MPF
A base class DevMpf is provided for writing epics device support that communicates with MPF.All code is written in C++. It is written to minimize dependencies between various components. For example the IP support can be used without MPF, MPF without the IP support, etc. The current implementation uses the standard C library but not the standard C++ library.
When MPF is used between processors, communication is via a tcp connection. Thus the two processors can reside anywhere as long as it is possible for them to communicate via tcp.
.../support/mpf/
gunzip <file>.tar.gz tar xvf <file>.tar
.../support/mpf/mpfX-Ywhere X-Y is the release. For example.
.../support/mpf/mpf1-5
The inet address must be replaced by the address of your test mv162.iocepicsclient/st.cmd iocepicsserver/st.cmd iocmpfclient/st.cmd iocmpfserver/startServers ioctcpclient/testClientWaitBuffer100 ioctcpclient/testClientWaitNoBuffer ioctcplocal/tcpTestWaitLocalBuffer100 ioctcplocal/tcpTestWaitLocalNoBuffer ioctcpserver/testServerWaitBuffer100 ioctcpserver/testServerWaitNoBuffer
Now issue the commands:
cd X-Y gnumake
Now proceed to the next two sections.
MV167
MV162boot device : ei processor number : 0 host name : <host> file name : <full path>/vxWorks inet on ethernet (e) : <IPmv167>:<subnet mask> inet on back plane (b): <IPgateway> host inet (h) : <IPhost> gateway inet (g) : user (u) : <user> ftp password (pw) (blank = use rsh): <password> flags (f) : 0x0 target name (tn) : ioc167 startup script (s) : <full path name>/st.cmd other (o) :
boot device : sm=0x80000600 processor number : 1 host name : <host> file name : <full path>/vxWorks inet on ethernet (e) : inet on back plane (b): <IPmv162> host inet (h) : <IPhost> gateway inet (g) : <IPgateway> user (u) : <user> ftp password (pw) (blank = use rsh): <password> flags (f) : 0x0 target name (tn) : ioc162 startup script (s) : <full path name>/st.cmd
mpf/epicsDevApp/epic/TestSrc/DevMpfInt32Test.c | This is an example of device support. |
mpf/mpfServerApp/testSrc/serverInt32.cc | The echo server for Int32 messages. |
mpf/mpfServerApp/epicsDevApp/localSrc | The subdirectory localSrc shows how to build for a system on which the client and server reside on the same processor. |
mpf/mpfServerApp/epicsDevApp/clientSrc
mpf/mpfServerApp/epicsDevApp/serverSrc |
The subdirectories clientSrc and serverSrc show how to build on a system where the client and server reside on different processors. |
mpf/epicsDevApp/Db/*.db | The databases for the example. |
mpf/iocBoot/iocepicslocal/st.cmd | The startup file for the local test. |
mpf/iocBoot/iocepicsclient/st.cmd
mpf/iocBoot/iocepicsserver/st.cmd |
The startup files for the remote test. |
Studying the above will show how to write servers, device support, build servers and client, and boot servers and clients.
In RELEASE add the line:
If you are using the config rules like those makeBaseApp release R3.31.2 or later thenMPF=<full path name>
If you are using older rules then you must edit CONFIG_APP to add:gnumake
ifdef MPFIn Makefile.Vx statements like
USR_INCLUDES += -I$(MPF)/include
MPF_BIN = $(MPF)/bin/$(T_A)
USER_DBDFLAGS += -I $(MPF)/dbd
endif
can be added.LIBOBJS += $(MPF_BIN)/mpfLib LIBOBJS += $(MPF_BIN)/ipLib
Note that ipLib also includes ipacLib. Thus an application using mpf does not have to also reference the ipac support.
MPF does not include support for specific ip modules. Such support must be obtained elsewhere.
<top>/ config/ RELEASE utilApp/ utilSrc/ mpfApp/ kernelSrc/ messageSrc/ libSrc/ testSrc/ ipApp/ src/ mpfServerApp/ epicsDevApp/ Db/ adl/ epicsDevSrc/ devMpf.* epicsTestSrc/ DevMpfInt32Test.c clientSrc/ localSrc/ serverSrc/ iocBoot/ iocepicsclient/ iocepicslocal/ iocepicsserver/ iocmpfclient/ iocmpflocal iocmpfserver/ iocmpflocal/ ioctcpclient/ ioctcplocal/ ioctcpserver/
config | This is almost exactly like <top>config described in the 3.13.1 version of makeBaseApp |
RELEASE | EPICS_BASE must be defined correctly. |
utilSrc | DLList, DataFreeList, FreeList, and WatchDog; Builds mpfUtilLib. |
kernelSrc | Contains the MPF code: Message, Routers, Tcp support; Builds mpfKernelLib |
messageSrc | Contains the code to build specific message types. If new messages type are added this is the place. If it proves necessary to support an endless number of message types a better way of building new message types must be developed. Builds mpfMessageLib. |
libSrc | Builds mpfLib which contains mpfUtilLib, mpfKernelLib, and mpfMessageLib. |
testSrc | Contains some low level test programs. Not of interest except for MPF core development. |
ipApp/src | The ip support. Builds ipLib. |
mpfServerApp | Contains serverInt32 and serverChar8Array which just echoes the messages sent to them. They are intended for testing and examples. clientInt32 and clientChar8Array are MPF clients. They are good examples of an MPF client and are also used for testing. |
epicsDevApp/Db and adl | Db contains the epics databases for the example in iocrstclient. adl contains the example medm screen. |
epicsDevApp/epicsDevSrc | This contains DevMpf, the base class for mpf epics device support. |
epicsDevApp/epicsTestSrc | MpfInt32Test.cc is a simple example of device support derived from DevMpf. |
iocepicsclient
iocepicsserver |
Epics (client) side and server side of example described previously. |
iocepicslocal | The example where client and server run on the same processor. |
iocmpfclient
iocmpfserver |
Tests for remote MPF communication |
iocmpflocal | Test for local MPF communication |
ioctcpclient
ioctcpserver |
Test for TCP communication. |
ioctcplocal | Local test for TCP communication. |
whereINP("C1 S0 @Int32")
Messages are sent through a router. Two routers are provided: localRouter and RMR (Remote Message Router). The localRouter passes messages between a client and server residing on the same processor. RMR sends messages between a client and server on different processors via a tcp connection.
typedef void (*clientCallback)(Message *message,void *clientPvt); class MessageClient { public: MessageClient(clientCallback,void *clientPvt); int bind(char *server, int location); int send(Message *message); void *getClientPvt(); private: ... }A client of the message passing system must:
class MessageServer { public: MessageServer(const char* name); void waitForMessage(); Message *receive(); Message *allocReplyMessage(Message *clientMessage,messageType type); int reply(Message *); void setQueueSize(int size); const char *getName() const; void report() const; private: ... }A message server must
Message instances are kept on a free list. Thus malloc is called only if the free list is empty. Free is never called. Each message type has it's own free list.
For servers it provides a report like the following:
iocrstserver> mrr clientRouterList serverRouterList 1 RMRServer stateConnected queueSize 100 inQueue 0 replyQueueFull 0 sendPerSec 240 receivePerSec 240 tcpSendPerSec 10 tcpReceivePerSec 10It provides a report for each server router. It shows the connection state. The meaning of the other fields are:
For clients it produces the following type of report
iocrstclient> mrr clientRouterList 1 RMRClient stateConnected queueSize 100 inQueue 0 sendQueueFull 0 Server Int32 has 4 clients. bindState connected sendPerSec 40 receivePerSec 40 tcpSendPerSec 10 tcpReceivePerSec 10 serverRouterListIt produces a report for each client router. It shows if the router is connected. The queueSize, inQueue, and sendQueueFull provide statistics about the send queue. It also shows a list of each server attached to the router.
iocrstserver> msr "Int32" Int32 queueSize 10 inQueue 0 queueRequests 132171 queueFullResponses 0 replyRequests 132171The meaning of the fields are:
Each vxWorks system using the message passing system needs the following command in the startup file:
routerInitIf local communication is desired then the following command must appear:
localMessageRouterStart(location)The parameter is the location of the local server. Note that the client specifies a location when it binds to a server. Each cpu can start a local router.
If tcp communication is desired then for each remote system the client must have the following command:
tcpMessageRouterClientStart(location,port,"address",bufSize,queueSize)
tcpMessageRouterServerStart(location,port,"address",bufSize,queueSize)
<message class> *pmessage = new <message class>;for example an Int32 message can be allocated as follows:
Int32Message *pmessage = new Int32Message;
Int32Message *pmessage =(Int32Message*)pMessageServer-> allocReplyMessage(preceive,messageTypeInt32);where preceive must be a message received from the client. The reason is that the receive message contains information describing where the client is located.
delete pmessage;Thus the server must delete any message received from a client and a client must delete reply messages from the server. The definition of class Message is:
Client and server code normally only call new, delete, the methods described in the following table, and message specific methods. The other methods are called by MPF itself. A message is allocated via a call to new and released via a call to delete. When delete is called the message is placed on a free list. Only one of the message types derived from Message can be created, i.e. it is not possible to create an object of type Message.class Message { public: messageType getType(); int32 getClientType(); void setClientType(int32 type); int32 getClientExtra(); void setClientExtra(int32 extra); virtual ~Message(); virtual int toBuffer(char **buffer); virtual int fromBuffer(const char **buffer); virtual int fromBufferSwitch(const char **buffer); virtual void print() const; static Message *allocate(messageType type); protected: Message(messageType type); .... }
The virtual methods must be implement by any class that derives from Message. The toBuffer and fromBuffer methods are called to put and take messages from a buffer that passes over a communication link.
method | Recommended Meaning |
getType | Returns the message type |
getClientType
setClientType |
These methods are for use by a client to access field clientType. When the server allocates a reply message via a call to allocateReplyMessage, the clientType from the clients message is copied to the reply message. In general a server should not use these methods. They are intended for clients that send multiple messages to a server and need to match reply messages with messages sent. Classes derived from DevMpf MUST NOT use these because DevMpf itself uses them. Use clientExtra instead. |
getClientExtra
setClientExtra |
These access a field clientExtra which is similar to clientType. They can be used by classes derived from DevMpf. |
MPF provides a header file "mpfType.h" which defines typedefs for int16, uint16, int32, uint32, float32, and float64. MPF provides static methods to transfer the following to/from network buffers:
Field | Recommended Meaning |
timeout | The timeout the server can use it it has to wait for a device. It should be in seconds. |
cmd | A command. Servers can supply a header file defining an enum for the commands. |
status | A status value. |
address | For anything that is an int32 and can be interpreted as an address. |
extra | An extra int32 word that client/server can use for anything they want as long as they agree on the meaning. |
Field | Recommended Meaning |
value | The address of a char8 array. The characters are not interpreted or translated by the message class, i.e. they are just a stream of octet values. The standard C library routines such as ::strcpy and ::memcpy can be used on the value field. |
numberRetrys | A useful field for servers. It should be the number of retrys if a command fails. |
eomLen | A useful field for servers. It should be the length of an end of message string. |
eomString | A useful field for servers. It should be a 1 or 2 character end of message string. |
Char8ArrayMessage provides the following methods for allocating and
freeing the string value.
method | usage |
allocValue | This is a static method which looks for the smallest freelist that can provide the needed space. If the requested size is larger than the size for the largest freelist, it calls new to allocate space. |
setSize | Sets the current size for the array. This size MUST be less than that allocated by allocValue. |
getSize | Gets the current array size. |
getMaxSize | This gets the size allocated by the call to allocValue |
freeValue | This puts storage obtained by allocValue back on the appropriate free list or calls delete if the storage was obtained via a call to new. This method is called automatically when a Char8Array message is deleted. |
Field | Recommended Meaning |
value | The Int32 value being passed. |
method | usage |
allocValue | This is a static method which looks for the smallest freelist that can provide the needed space. If the requested size is larger than the size for the largest freelist, it calls new to allocate space. |
setLength | Sets the current length for the array. This length MUST be less than that allocated by allocValue. |
getLength | Gets the current array length. |
getMaxLength | This gets the length allocated by the call to allocValue |
freeValue | This puts storage obtained by allocValue back on the appropriate free list or calls delete if the storage was obtained via a call to new. This method is called automatically when an Int32Array message is deleted. |
Field | Recommended Meaning |
value | The Float64 value being passed |
method | usage |
allocValue | This is a static method which looks for the smallest freelist that can provide the needed space. If the requested size is larger than the size for the largest freelist, it calls new to allocate space. |
setLength | Sets the current length for the array. This length MUST be less than that allocated by allocValue. |
getLength | Gets the current array length. |
getMaxLength | This gets the length allocated by the call to allocValue |
freeValue | This puts storage obtained by allocValue back on the appropriate free list or calls delete if the storage was obtained via a call to new. This method is called automatically when an Float64Array message is deleted. |
field | usage |
baud | Baud rate. 0 means don't change |
stopBits | Stop bits. Normally 1 or 2. 0 means don't change. |
bitsPerChar | Bits per character. Normally 5,6,7, or 8. 0 means don't change. |
parity | Single character. Normally 'E', 'O', or 'N'. 0 means don't change. |
flowControl | Single character. Normally 'H' or 'N'. 0 means don't change. |
DevMpf is an abstract base class for implementing EPICS device support.
// return codes for startIO and completeIO. #define MPF_OK 0 #define MPF_NoConvert 2 enum replyType {replyTypeNone, replyTypeCompleteIO, replyTypeReceiveReply};
class DevMpf { public: DevMpf(dbCommon*,link*,bool iointValid); // Following must be implemented by device support modules virtual long startIO(dbCommon*)=0; // start async IO virtual long completeIO(dbCommon*,Message *)=0; // end async IO // the following can be implemented by device support modules virtual void receiveReply(dbCommon*,Message *); virtual void connectIO(dbCommon*,Message *); // connection message virtual void outOfBandIO(dbCommon*,OutOfBandMessage *);// outOfBand virtual long convert(dbCommon*,int pass); // do linear conversion // send a message to MPF server with no reply expected int send(Message*); // send a message to MPF server and wait for reply via completeIO int sendReply(Message*); // send a message stating the type of reply int send(Message*,replyType); // This routine gets a pointer to the user portion of the parm field const char* getUserParm() const { return((const char*)userParm); } long getStatus() const { return(status);} // Following are DSET routines static long read_write(void*); // generic DSET read/write routine static long ioint(int cmd,dbCommon*,IOSCANPVT* iopvt); // DSET i/o intr static long linconv(void*,int); // calls convert IOSCANPVT ioscanpvt; bool iointValid; bool isConnected() { return((connectState==connectYes) ? true : false);} private: ...
}Before describing the methods a few comments may be helpful.
If the support is for a simple device, i.e. a device that can be supported via the following:
For devices that are not simple, e.g. devices that must take special action when connecting to the server, additional methods are available: In particular:
ioscanpvt is for provided for convenience. If a derived
class wishes to support io interrupt processing it can request that the
base class perform the necessary initialization. The derived class must,
however, call scanIoRequest.
method | Usage | meaning |
DevMpf | Constructor | The constructor must be given the address of the record and link (INP or OUT). iointValid specifies if the base class should initialize ioscanpvt |
startIO | Called by DevMpf | When a request is made to process a record this is called only when
the following conditions are all true: The server is connected, PACT is
false, and no reply message from the server is available. This MUST be
implemented by any class derived from DevMpf.
This method is always called as a result of a record being processed and the record is not active. If startIO issues a send(message,replyTypeCompleteIO) then PACT is set TRUE, i.e. the record will not complete processing until the reply message is received and completeIO is called. |
completeIO | Called by DevMpf | DevMpf calls this when the reply to a send(message,replyTypeCompleteIO)
is received. This MUST be implemented by any class that derives from DevMpf.
This method is always called as a result of a record being processed |
receiveReply | Called by DevMpf | DevMpf calls this when the reply to a send(message,replyTypeReceiveReply)
is received. This call does not involve record processing.
The default method just prints a message and deletes the message it received. |
connectIO | Called by DevMpf
Default supplied by DevMpf. |
DevMpf calls this when it receives a ConnectMessage. DevMpf provides
a default implementation.
If device support implements this method it should call DevMpf::connectIO (just before returning) to ensure correct behavior. |
outOfBandIO | Called by DevMpf
Default supplied by DevMpf. |
DevMpf calls this when it receives an OutOfBand message. For example
if server receives a send (no reply) message that can not be processed,
the server can send an OutOfBand message to the client.
The default method just prints a message and deletes the message it received. |
convert | Called by DevMpf, Default supplied by DevMpf | This is the DSET convert routine for ai, ao type records. The default version does nothing. Device support derived from base can provide it's own version. |
send(Message*) | Equivalent to send(message,replyTypeNone) | |
sendReply | Equivalent to send(message,replyTypeCompleteIO) | |
send(Message*,replyType) | Called by derived class | Send a message to the server. SetClientType is called to specify
the replyType. If the message is successfully sent and replyType is replyTypeCompleteIO
pact is set true.
NOTE: Only one send(message,replyTypeCompleteIO) should be outstanding, i.e. after calling send(message,replyTypeCompleteIO) wait for completeIO to be called before issuing another send. |
getUserParm | Called by derived class | Get the portion of the parm of the INP or OUT link that follows the server name. |
getStatus | Called by derived class | Get the status. A 0 value is success. Any other value is device dependent. |
read_write | Called by record support | This is the read or write DSET routine. Handled automatically by DevMpf |
ioint | Called by record support | This is DSET get_io_intr routine. Handled automatically by DevMpf |
linconv | Called by record support | This is DSET linr_conv routine. It calls convert. |
The INP or OUT field MUST have the format:
field(INP or OUT, "#C<location> S<signal> @<server>,<deviceSpecific>")
parameter | meaning |
location | Location of the server. Must be integer which determines which message router handles messages. |
signal | For optional use by device support derived from DevMpf |
server | Name of the server which receives messages. |
deviceSpecific | For optional use by device support derived from DevMpf. Either a blank or comma can separate the server name from the device specific information. |
The first is used for record types that support linear conversions, e.g. ai and ao, and the second for other record types.MAKE_LINCONV_DSET(<dset name>,<dev init>) MAKE_DSET(<dset name>,<dev init>)
FreeList provides the same functionality as the freeList facility provided with epics base, in fact it is just freeList redone in C++.
DLList is a double linked list class that does not require nodes to be embedded in objects placed in a list. This is different than the ellList facility provided by epics base.
WatchDog is a class that provides functionality similar to the vxWorks wdLib. The major exception is that the user supplied callback is called by a WatchDog supplied task rather than being called at interrupt level.
DataFreeList is a class for data free lists. It provides free lists of sizes ranging from 16 bytes to 4096 bytes. It is used by the array message types.
Reboot can be used to turn off interrupts when a soft reboot is being performed.
These classes are not described in this document. If you want to use
them in new code look at examples in existing code.
IndustryPack Support consists of the following components:
IndustryPackCarrier
The base class for an IndustryPack carrier. It allows access to IndustryPack carrier configuration and control independent of the specific carrier hardware board.IndustryPackSite
The base class for a specific site on a specific carrier, i.e. the place where an IP module resides.IndustryPackIpac
Implements the IndustryPackCarrier and IndustryPackSite classes which actually pass requests on to the ipac software to implement.IndustryPackModule
Provides IndustryPack Module configuration independent of the specific IP carrier. This class is used by specific module support such as OctalUART and GpibGsTi9914.
class IndustryPackCarrier { public: IndustyPackCarrier(); int registerName(const char *carrierName); static IndustryPackCarrier *find(const char *carrierName); IndustryPackSite *findSite(const char *siteName); IndustryPackSite *addSite(IndustryPackSite *pSite); const char *getCarrierName(); virtual volatile void *allocMemMapIP(int size) = 0; private: ... };
class IndustryPackSite { public: IndustryPackSite(); const char *getSiteName(); virtual volatile void* allocMemMapIP(int size) = 0; virtual volatile IP_ID_PROM* getMemBaseID() = 0; virtual volatile void* getMemBaseIO() = 0; virtual volatile void* getMemBaseIP() = 0; virtual int intConfig(int num) = 0; virtual void intEnable(int num) = 0; virtual void intDisable(int num) = 0; protected: const char *siteName; IndustryPackCarrier *pIpCarrier; private: };
Method | Implementor | Usage | Description |
registerName | IndustryPackCarrier | Derived class. | Register the carrier name. |
find | IndustryPackCarrier | Public use. | Find and return IndustryPackCarrier. |
findSite | IndustryPackCarrier | IndustryPackModule | Find and return site. |
addSite | IndustryPackCarrier | Called by derived class. | Add a site. |
getCarrierName | IndustryPackCarrier | Public use. | Return carrier name. |
allocMemMapIP | Derived Class | IndustryPackModule | Allocate memory. |
getSiteName | IndustryPackSite | Derived class. | Get site name |
allocMemMapIP | Derived class. | IndustryPackModule | See IndustryPackModule |
getMemBaseID | Derived class. | IndustryPackModule | See IndustryPackModule |
getMemBaseIO | Derived class. | IndustryPackModule | See IndustryPackModule |
getMemBaseIP | Derived class. | IndustryPackModule | See IndustryPackModule |
intConfig | Derived class. | IndustryPackModule | See IndustryPackModule |
intEnable | Derived class. | IndustryPackModule | See IndustryPackModule |
intDisable | Derived class. | IndustryPackModule | See IndustryPackModule |
ipacAddCarrier(&ipmv162,"A:l=3,3 m=0xe0000000,64;B:l=3,3 m=0xe0010000,64; C:l=3,3 m=0xe0020000,64;D:l=3,3 m=0xe0030000,64") initIpacCarrier("carrierName", 0)
class IndustryPackModule { public: static IndustryPackModule * createIndustryPackModule( const char *moduleName, const char *carrierName, const char *siteName); static IndustryPackModule *find(const char *moduleName); static IndustryPackModule *find( const char *carrierName, const char *siteName); volatile void *allocMemMapIP(int size); int intConfig(int num); int intEnable(int num); int intDisable(int num); volatile IP_ID_PROM *getMemBaseID(); volatile void *getMemBaseIO(); volatile void *getMemBaseIP(); int getMemSizeIP(); unsigned char getManufacturer(); unsigned char getModel(); unsigned char getRevision();
private: ... };All public methods are implemented by IndustryPackModule and used by module specific code.
Method | Description |
createIndustryPackModule | Creates an instance of IndustryPackModule. It prints an error message and returns 0 if carrierName or siteName does not exist or if moduleName already exists. |
find | Two find methods are provided. One finds modules by module name. The other finds modules by carrier and site name. |
allocMemMapIP | Allocate IP memory. |
intConfig | Interrupt Configure. The argument must be 0 or 1. |
intEnable | Enable interrupt. The argument must be 0 or 1. |
intDisable | Disable interrupt. The argument must be 0 or 1. |
getMemBaseID | Get the address of the module ID space. |
getMemBaseIO | Get the address of the module IO space. |
getMemBaseIP | Get the address of the module IP space. |
getMemSizeIP | Get the size of allocated IP memory. |
getManufacturer | Get the manufacter ID. |
getModel | Get the model ID. |
getRevision | Get the revision |
The local tests were run on an mv162. The remote tests were run on an mv167 client and mv162 server. I think they were both 25 MHz processors. msgPerSec and tcpPerSec are round trip, i.e. a send and receive. The message in all cases was just an Int32 value. The no buffer case means that a value was sent and the client waited for that response before sending the next message. The buffer case means that the specified number of values were sent before waiting for the responses.
Test | msgPerSec | tcpPerSec | client idle | server idle |
tcpLocalNoBuffer | 237 | 237 | 15% | - |
tcpLocalBuffer100 | 6050 | 60 | 5% | - |
tcpRemoteNoBuffer | 485 | 485 | 54% | 39% |
tcpRemoteBuffer100 | 10050 | 100 | 52% | 40% |
Test | msgPerSec | tcpPerSec | client idle | server idle |
mpfLocalNoBuffer | 3705 | - | 0% | - |
mpfLocalBuffer100 | 7966 | - | 0% | - |
mpfRemoteNoBuffer | 331 | 331 | 52% | 40% |
mpfRemoteBuffer100 | 1440 | 14 | 52% | 43% |
mpfRemoteBuffer1000 | 2100 | 16 | 31% | 18% |
Test | msgPerSec | interrupt | idle | total |
local | 159 | 76% | 0% | 89% |
remote | 156 | 65% | 10% | 91% |