Information about the PAD protocol ================================== The 'PAD protocol' runs on top of UDP. The PAD embedded controller acts as a 'PAD protocol server' reacting to 'requests' issued by a 'PAD protocol client' (usually a VME IOC or a linux box). PAD Request ----------- A PAD request contains a timestamp (so that e.g., current EVR time may be communicated to the PAD), a 'transaction ID', protocol version number and commands for one or multiple PADs. Note that the protocol itself does not use the timestamp information itself and doesn't define the semantics. The data are simply returned to the sender. If a single request contains commands for multiple PADs then the request is either broadcast or posted to a multicast group. This is the normal mode of operation since it also provides a means for the PAD to 'learn' the client's MAC and IP addresses without any need for ARP. The layout of a PAD request is defined in the 'padProto.h' header. Note that all members of a PAD request are in network (AKA big-endian) byte order. PAD Addressing - - - - - - - - Since commands contained in a single request may be directed to multiple PADs the PAD protocol implements a simple addressing scheme: Each PAD subscribed to a single multicast group or residing in the same LAN broadcast domain is assigned by the system engineer a unique 'channel' ID which is a small integer. When the 'padProtoHandler()' loop is started on a PAD the channel ID must be passed as an argument. When the client wants to address a specific PAD then it must know its channel ID. There are three ways of addressing PAD(s) with a request: a) Broadcast. The request contains a single command struct and the 'nCmds' member is set to PADREQ_BCST. All PADs receiving the request will execute the same command. b) Single PAD. The request contains a single command struct. The 'nCmds' member is set to the negative channel ID or zero if the ID is zero. E.g., if the channel ID is '5' then 'nCmds' would have to be set to '-5'. Only the addressed PAD would execute the command. c) Multiple PADs. The request contains multiple command structs (which may identify different commands but are all padded to the same size). In this case the PAD processing the request uses its channel ID to index the array of command structs to extract the command it should execute. Note that if a command is to be issued to a specific channel using method c) then there must be at least 'channel ID + 1' command structs in the request. PAD Reply --------- By default, the PAD sends a reply back to the client upon completion of the command. Optionally, the client may set a flag (in the command sent embedded in the PAD Request) instructing the client that no reply is wanted. This saves CPU cycles and network bandwith. The Reply contains a copy of the timestamp and 'transaction ID' that were embedded in the request. Also, the reply identifies the PAD instance ('channel') sending the reply. The reply to some commands may contain 'payload' data. The layout of a PAD reply is defined in the 'padProto.h' header. Note that all members of a PAD reply are in network (AKA big-endian) byte order. PAD Command ----------- A PAD Command is an aggregate of varying size, depending on the 'command type' since a specific command may require additional data to be communicated to the addressed PAD. If multiple commands are sent in a single request then all command 'struct's are padded to the size of the biggest command present in the request. The 'cmdSize' member in the request is set to the size of the individual commands (= size of the biggest one). Supported PAD Commands: PADCMD_NOP No operation. This command has no effect. Note that no reply is sent either (even if PADCMD_QUIET flag is not set). This command can be used as a placeholder when sending a request using method c) above. PADCMD_NOP can be sent to channels that should remain idle. PADCMD_ECHO Send the reply back. The intended use is for testing/debugging. PADCMD_STRM Start PAD 'streaming' mode. This is the main operating mode. See the section on 'streaming' below for more information. PADCMD_STOP Stop PAD 'streaming' mode. PADCMD_SPET While in 'streaming' mode just update the timestamp and transaction ID (which are embedded in the streamed information). PADCMD_SIM Update simulation parameters and (optionally) send a packet with simulated stream data. See the sections on 'streaming' and 'simulation' below. Example for sending a PAD request #include #include /* for htons/htonl & friends */ #include /* Send NOP to channel 0 and ECHO to channel 1 */ struct { PadRequestRec req; PadCommandRec cmds[2]; } myreq; struct { PadReplyRec rep; uint8_t payld[100]; } myrep; struct timeval now; gettimeofday(&now, 0); myreq.req.version = PADPROTO_VERSION2; myreq.req.nCmds = 2; myreq.req.cmdSize = sizeof(myreq.cmds[0]); /* timestamp and XID are returned in the ECHO reply */ myreq.req.timestampHi = htonl(now.tv_sec); myreq.req.timestampLo = htonl(now.tv_usec); myreq.req.xid = htonl(0xdeadbeef); myreq.cmds[0].type = PADCMD_NOP; myreq.cmds[1].type = PADCMD_ECHO; write(socket_sd, &myreq, sizeof(myreq)); if ( read(socket_sd, &myrep, sizeof(myrep) > 0 ) { printf("Received reply to command type %u from channel %u\n", PADCMD_GET(myrep.rep.type), myrep.rep.chnl); printf("Status: %u\n", myrep.rep.stat); printf("Timestamp was %u.%u\n", ntohl(myrep.rep.timestampHi), ntohl(myrep.rep.timestampLo)); printf("XID was %u\n", ntohl(myrep.rep.xid)); printf("%u bytes in payload\n", ntohs(myrep.rep.nBytes)); } Streaming --------- The PADCMD_STRM and PADCMD_STOP commands are sent by a client to switch the PAD into and out of 'streaming mode'. In 'streaming mode' the PAD sends unsolicited packets containing e.g., digitizer samples whenever new data are available. In most cases sending 'stream' data is triggered by an external event, e.g., an 'ADC acquisition done' interrupt. The task processing such an interrupt calls int padStreamSend(PadStreamGetdataProc getdata_cb, int type, int idx, void *uarg); to send the data. The 'getdata_cb' is a user-provided routine (see below) that is responsible for formatting and copying the data. Note that 'padStreamSend()' fails without doing any processing if the PAD is not in streaming mode. This is a possible scenario for an application that uses a PAD stream: 1. Client fills a PadStrmCommandRec struct with desired parameters: - command type (PADCMD_STRM) - UDP port number where it listens for streamed data - number of 'samples' (array elements) *per channel* that the PAD should acquire and stream back. Note that the PadProtocol always assumes a channel number of four, i.e., a stream ships two-dimensional arrays with 4 x nsamples data items. As of protocol version 3 there is an option for only shipping a single channel (rather than 4) in a stream packet. - a set of flags describing the desired data format: o endian-ness (little-endian if PADCMD_STRM_FLAG_LE is set, big-endian otherwise). o array element word-length (32-bit if PADCMD_STRM_FLAG_32 is set, 16-bit otherwise). o layout (column-major if PADCMD_STRM_FLAG_CM is set, row-major otherwise). In column-major layout array element (i,j) is stored at linear index i+j*4. In row-major layout the element (i,j) is stored at linear index i*nsamples+j. E.g., in C, array elements 'X[i][j]' are stored in row-major layout whereas FORTRAN's X(i,j) uses colum-major. o single-channel option. If PADCMD_STRM_FLAG_C1 is set then the client instructs the stream server to only send data from a single channel (instead of the default, four). The client and server must agree upon a scheme to distinguish channels (e.g., implicit: only a single channel exists or explicit: use the data 'kind' identifier to label a particular channel). 2. Client sends a PadRequest containing the PadStrmCommandRec to the PAD. 3. PAD switches to streaming mode. 4. On each trigger the PAD receives it calls 'padStreamSend()' which in-turn executes the user-supplied callback for formatting and copying the data to memory or directly to the transmitter. 5. Client sends a PADCMD_STOP 6. PAD switches streaming mode off. An application that wishes to implement support for the PAD streaming mode needs to implement two callback routines: int start_stop_cb(PadStrmCommand start, void *uarg); and void * getdata_cb(void *pktBuf, int idx, int nchannels, int nsamples, int d32, int endian_little, int col_major, void *uarg); The 'start_stop_cb()' is executed whenever the PAD receives a PADCMD_STRM or PADCMD_STOP command, respectively. In the former case the 'start' argument points to the PadStrmCommandRec that contains additional information (flags, layout, etc.) and in the latter case (PADCMD_STOP) the 'start' argument is NULL. The 'start_stop_cb()' may perform any actions required by the application to switch to and from streaming mode. The 'start_stop_cb()' must return zero if these actions were successful. Under any condition that prohibits changing the mode as requested this callback must return nonzero in order to reject the request. In particular, the start_stop_cb() must check the requested parameters (nsamples, flags) and reject the start request if the requested parameters/format are not supported by the application by returning a nonzero status. E.g., if your 'getdata_cb' does only support 4-channels, 16-bit, row-major layout and no byte-swapping then 'start_stop_cb()' must return nonzero if any of the flags PADCMD_STRM_FLAG_C1, PADCMD_STRM_FLAG_CM, PADCMD_STRM_FLAG_32 or PADCMD_STRM_FLAG_LE (assuming that the pad protocol server executes on a big-endian machine) are set. ************************************************************* It is MANDATORY for the start_stop_cb() to check ALL parameters and return nonzero if any unsupported parameter is solicited. It is also highly RECOMMENDED for the start_stop_cb to check the protocol version and reject the request (by returning nonzero) if an unexpected protocol version is detected since newer versions of the protocol may add more features and a request for such features may go undetected by the current implementation of start_stop_cb(). ************************************************************* The 'uarg' parameter passed to the 'start_stop_cb' is an arbitrary pointer that is supplied by the user when the stream facility is initialized, i.e., it is a copy of the 'uarg' parameter passed to int padStreamInitialize(void *if_p, PadStreamStartStopCB start_stop_cb, void *uarg); The 'getdata_cb()' is responsible for converting the data to the requested format as indicated by the 'd32', 'endian_little', 'col_major', 'nchannels' and 'nsamples'. Note that all of these parameters *MUST* be honored by this callback or the client will receive corrupted data. Hence it is very important that the 'start_stop_cb()' performs a proper check and rejects starting the stream if an unsupported format is requested. The callback can either write the correctly formated data itself to the FIFO / hardware (this is ugly but quite a bit faster on the coldfire platform) or it can write to a memory buffer. In the former case the callback returns NULL, in the latter a pointer to the memory buffer. E.g., the 'pktBuf' area provides enough space to store data that fits into an ethernet frame (minus the ethernet, IP, UDP and PadProto headers). It is important that *exactly* nsamples * (d32 ? 4 : 2) * nchannels bytes are written (or contained in the buffer). If 'endian_little' is set then words of the requested size (32-bit if 'd32' is nonzero, 16-bit otherwise) must be written in little-endian byte ordering. If 'endian_little' is zero then big-endian byte ordering is to be used. Note that it is the application's responsibility to perform the necessary (compile-time or run-time) checks to find out under what circumstances byte-swapping must be performed. If 'col_major' is set (nonzero) then the data array has the (C-language) layout (nchannels is normally 4) data[nsamples][nchannels] otherwise the layout is data[nchannels][nsamples] Note that the callback has no way of ignoring any of those flags. If you don't want to implement a given flag then you can check for it in the start_stop_cb() and reject starting the stream from there. However, I recommend to always implement at least endian_little since that allows you to use the 'padProtoHost' tool from a linux-x86 host for development. The 'idx' parameter can be used as a sequence number. It is passed to 'padStreamSend()', handed to 'getdata_cb()' and eventually stored in the stream packet. It can e.g., be used if an array is to be streamed which is bigger than what fits into a single datagram. Such an application could repeatedly call 'padStreamSend()' for different 'idx' numbers. 'getdata_cb()' would then copy the sub-part of the large array that corresponds to 'idx'. Finally, the receiver of the stream would re-assemble the big array from the individual parts based on the received 'idx' numbers. E.g., the application could declare /* 4*64 32-bit numbers fit into one datagram */ uint32_t my_array[N_CHUNKS][4][64]; void * my_getdata(void *pktBuf, int idx, int nchannels, int nsamples, int d32, int le, int cm, void *uarg) { return uarg; /* 'idx' not required here - 'uarg' already points to chunk 'i' */ } for ( i=0; istrm_cmd_idx], rply->data, sizeof(uint32_t)*4*64); The callback prototype has changed in the recent past. I recommend to pay attention to compiler warnings which would bring this change to your attention. Example start_stop_cb(). Assume that we support two modes, a 'raw mode' where we ship raw 16-bit ADC samples and a 'processed mode' where we ship a few 32-bit numbers that we compute from the raw samples. Use the D32 flag as an indication from the client whether it desires 'processed' vs. 'raw' data. int my_start_stop_cb(PadRequest req, PadStrmComand scmd, void *uarg) { if ( scmd ) { /* Start Stream */ if ( PADPROTO_VERSION3 != req->version ) { fprintf(stderr,"Unsupported protocol version 0x%02x: (expected 0x%02x)\n", req->version, PADPROTO_VERSION3); return -ENOTSUP; } if ( (scmd->flags & PADCMD_STRM_FLAG_32) ) { /* assume processed mode */ if ( (scmd->flags & (PADCMD_STRM_FLAG_CM | PADCMD_STRM_FLAG_C1)) ) { fprintf(stderr,"Neither col-major nor single-channel operation is supported\n"); return -ENOTSUP; } /* OK. Could verify that * ntohl(scmd->nsamples) matches what we expect */ } else { /* assume raw mode */ } } else { /* Stop Stream */ } return 0; } Example getdata() callback: void * getdata(void *pktBuffer, int idx, int nchannels, int nsamples, int d32, int endianLittle, int colMajor, void *uarg) { if ( d32 ) { /* Processed mode */ /* Cannot be colMajor since we rejected that in start_stop_cb */ if ( endianLittle ) { /* Copy byte-swapped data (nsamples * 4 32-bit words) to * lan9118's FIFO and return NULL to let the caller know * that the work is done. */ return 0; } else { /* If our data is already in a memory buffer we can let the caller * do the copying (only more efficient to write to TX FIFO ourselves * if we compute data and hold it in a register already, e.g., * because we byte-swapped or read from ADC FIFO.) */ return (void*)memBuffer; } } else { /* "Raw mode" */ if ( colMajor ) drvPadReadFifosColMajor( LAN9118_FIFO_HACK, nsamples*4, endianLittle); else drvPadReadFifosRowMajor( LAN9118_FIFO_HACK, nsamples*4, endianLittle); } return 0; } drvPadUdpComm ============= >>> this is from an email-conversation, I don't recall with whom <<< It's not the default value. You define the number of samples in the DrvPadUdpCommPrefsRec struct (on the IOC/VME side). If the 'nsamples_dynamic' flag is set then you indicate that it is OK for the application (again on the IOC/VME side) to change the number of samples at run-time (via drvPadUdpCommStrmSetNSamples()). If the flag is clear then the number of samples is fixed (which probably is what you want). Again: the number of samples is 'per channel'. The PadStream protocol assumes a 4-channel digitizer. Of course, you can use the protocol for your own purpose (that's what you do when you ship processed data) but the protocol still assumes 4 channels. Hence you transmit 4*nsamples numbers and the 'getdata' callback MUST exactly copy that amount to the Fifo (or hold it in a memory buffer if it wants the caller of 'getdata' to write to the fifo). If the 'd32' flag (again in the DrvPadUdpCommPrefsRec struct) is set then the numbers/samples are 32-bit, 16-bit otherwise. > The getdata callback in the LLRF PAD only put 48 byes into the hardware > fifo. It should *always* honor 'nsamples' (and pad if necessary). > But, the drvLan9118TxPacket gave more than 1k (1088 bytes > exactly). > So, the fifo could be screwed up. > Yes. > I think my packet size is 112 bytes. > 44 byes (udp header) + 20 byes (data header) + 48 bytes (PAD data). > Never mind the packet size. You just *have to* honor nsamples that's all you have to do. > So, I want to adjust the nsamples. > I tried to use drvUdpCommSetNSamples() API. See above. You should be able to just set it in the 'PrefsRec'. > 1. Is it correct one to adjust the nsamples variable? > Probably not. > 2. When try to use the drvUdpCommSetNSamples, I got symbol name problem. > Actually, the API has been defined in the libpadUdpCommSupport.a. > Unfortunately, this module contains some EVR symbols. > But, we don't have EVR library for the coldFire. > This routine is called on the VME/IOC side. The PAD has no say in this. VME requests a stream of given - nsamples - word-size (32/16) - endianness - matrix layout (row-major/col-major) The PAD can only reject a request from the 'start_stop_cb' (letting it return nonzero). The PAD has no way to modify the request. Finally 'getdata' ALWAYS MUST provide the requested data EXACTLY as requested (nsamples/d32/littleEndian/colMajor) HTH -- Till You don't have to do anything (which is the advantage of using this shared code). The 'padUdpHandler' is smart enough to deal with the case of the peer being a multicast address. However, you do need to somehow figure out what MC address to use. This is an administrative decision and we came to the conclusion that we use the startup script and DNS for this. We should at some point have a DNS name for each group and your startup script could do e.g., setenv("PADMCGRP", "dns_name_of_group", 1) Your code (look at Bpm/padBpmApp/src/devWfPadBpm.c) can e.g., do arg->mcaddr = 0; /* => use broadcast */ if ( !(mcgrp = getenv("PADMCGRP")) ) { epicsPrintf("Warning: 'PADMCGRP' envvar unset; expecting to use BROADCAST for IOC communication\n"); } else { if ( hostToIPAddr(mcgrp, &ina) ) { epicsPrintf("Error: Lookup for '%s' failed -- unable to start UDP listener\n", mcgrp); goto bail; } arg->mcaddr = ina.s_addr; epicsPrintf("UDP Listener uses IP Multicast (%s) -- reception of broadcast traffic DISABLED\n", mcgrp); drvLan9118BcFilterSet(lanIpBscIfGetDrv(lanIpIf), 1); } And later the padUdpHandler is called: padUdpHandler(arg->mcaddr, ...) Note that as a precaution the code disables reception of any FNET broadcast traffic at the EIOC's 2ndary NIC hardware. This is to protect the NIC and CPU from excessive traffic (which in theory shouldn't happen but you never know...) This means that it will be IMPOSSIBLE to use ARP for finding the EIOC's (2ndary) IP address (primary networking is not affected, of course). Note that this code works if you don't have a DNS name (e.g., for development) - you just use a string with and address in 'dot' notation, e.g., "239.255.254.40" WATCHDOG: -------- drvPadUdpComm (if you're using that) already supports/implements such a watchdog: A watchdog task periodically executes (by default every 10s but there is a global variable you can change) code which checks which channels are 'on-line' AND have received any kind of data during the last 20s. When you execute drvPadUdpCommStartReq( channel ), i.e., when a PAD is requested to start streaming data then the 'channel' is marked 'on-line'. Similarly, when you call drvPadUdpCommStopReq( channel) then the channel is marked 'off-line'. Whenever your 'cook' callback that processes the raw data returns a value >=0 then a flag is set ( multiple flags are maintained, one for each 'kind'/flavor of data) indicating that 'data arrived' (it is this flag that is checked by the watchdog task). Note that for now your cook callback should only return the macros defined in since other values may have special semantics in the future: PAD_UDPCOMM_COOK_STAT_OK: (OK) -> set 'data arrived' flag and request record IO/scan-list to be scanned. PAD_UDPCOMM_COOK_STAT_NOSCAN: (got data but e.g., there is no beam) -> set 'data arrived' flag but omit scanning the record list PAD_UDPCOMM_COOK_STAT_NOSCAN_NOPOST: -> do NOT set 'data arrived' flag, do NOT scan record list. When the watchdog task finds that no data of any 'kind' have been received (i.e., the 'cook' callback has never returned >=0 for any 'kind ' of data then your 'watchdog' callback is executed. You can use this callback to take appropriate action. Here some more (un-ordered) info from another email: The major new features are as follows: 1) Support for up to 8 'kinds' of data: drvPadUdpComm no longer restricts you to use only three kinds of data. Also, the special return values defined for the 'cook' callback: PAD_UDPCOMM_COOK_STAT_DEBUG_NOSCAN PAD_UDPCOMM_COOK_STAT_DEBUG_DOSCAN have changed. These return values are used for special applications to indicate that the data buffer should not be posted at the 'message-board' in the slot # 'kind' but in some different slot. Up to R2-0-0 this different slot was encoded implicitly as WAV_BUF_NUM_KINDS - 3 + 'kind'. As of R2-0-0 you can/must explicitly specify the alternate 'kind' value as a parameter to the macros PAD_UDPCOMM_COOK_STAT_DEBUG_NOSCAN( alt_kind ) PAD_UDPCOMM_COOK_STAT_DEBUG_DOSCAN( alt_kind ) Note that you are not normally supposed to use these macros anyways. They are intended for special debugging purposes. However, the new version simply allows you to use up to 8 different 'kinds' of data. One thing is important to note: Data are posted at the message board in slot # 'kind' after the 'cook' callback is done. If you e.g., have 4 'kinds' then packets of these 4 kinds arriving in quick succession are posted in 4 different slots. Waveform records pick up the latest buffers when they are scanned. As already described in an earlier message, there are three ways to scan waveforms: a) ordinary, periodic scanning. The waveform just picks up an arbitrary 'wavbuf', i.e., the last posted before the scanning happened. b) 'wbscan'. If the waveform record's INP field contains the string 'wbscan' then posting a wavBuf on the message board (done by the high-priority drvPadUdpComm driver task after executing 'cook') automatically requests the waveform to be scanned (but this happens in a different, lower-priority task context). Note that the scan-rate is identical to the post-rate, i.e., most likely 120Hz. c) I/O-Intr scanning. If the waveform is set to I/O-Intr scanning then scanning is triggered by arrival of 'kind-zero' data (note that you cannot change these semantics). Note that records associated with *all* 'kinds' that are I/O-IntrI scanned are on the *same* scan list that is triggered by arrival of kind-zero data. Hence, you would normally arrange the timing of your streamed data so that kind-zero arrives *last*. Also, this is where the PRIO field of the records matters. If PRIO is not HIGH (i.e., medium or low) then the scan rate is decimated to no faster than 1Hz. If PRIO is HIGH then the scan rate equals the arrival of kind-0 data. If this seems weird then please remember that this was developed for BPMs. Here, two calibration cycles (kind == CalRed and kind == CalGrn) precede a BPM reading cycle. Hence, the condition that kind-0 arrives last is met. Also, for BPMs, no waveforms are ever set to PRIO==HIGH. Only the final BPM readings which go into BSA and which are AI records are PRIO==HIGH and are thus scanned with every kind-0 data arrival. The waveforms are PRIO==LOW and thus scanned synchronously with kind-0 data but at a much decimated rate. This is the intended scenario for the I/O-Intr type scanning. 2) Support for multiple references to PadReply packets. The semantics of drvPadUdpComm are as follows: - driver receives a udpComm/padReply packet. This is 'owned' by the driver. If any error is encountered then the driver releases the packet. - driver executes 'cook' callback. The callback may inspect the packet but does *not* take over ownership. - driver allocates a wavBuf and attaches the packet to it (along with a proper 'free' method so the wavBuf code knows how to eventually release the packet). It then 'posts' the wavBuf on the message-board and this effectively transfer 'ownership' to the wavBuf and associated waveformRecord devsup code. If you want your 'cook' routine to share ownership of a packet (this is read-only, you must never modify packet data) then you can - under udpComm-R2-0-0 - create an additional reference to a packet: padReplyRef(PadReply rply) and thus hold on to it independently from drvPadUdpComm. When your code is done with the packet it must release it: padReplyFree(PadReply rply) If you transfer ownership to wavBuf then you must set the wavBuf's 'free' member to padReplyFree. Note that most of this (except for creating the reference) is done by the (new + public) courtesy routine drvPadUdpCommPostReply(). 3) Support for number of PAD ADC channels other than 4: The padProtocol was modified so that it is possible to request streaming of a single channel instead of 4 channels. Unfortunately, there were not enough free bits to allow an arbitrary number 1..4. A new stream option/flag PADCMD_STRM_FLAG_C1 was added, which - if set - indicates that the client request streaming of a single channel only (the channel number could be communicated via the stream 'kind' that can be set in the request and is replied back). Note (and please read README.drvPadUdpComm) that your stream 'start_stop_cb' *must* check all the requested features (and the protocol version) and reject starting the stream if unsupported features are requested or an unrecognized protocol version is detected. HTH -- Till