The EPICS on RTEMS @ SSRL How-To

Introduction

At SSRL, we use VGM series VME single board computers by Synergy microsystems (featuring a G3/G4 PowerPC CPU) for the SPEAR3 electron storage ring control system.

On the software side, we use EPICS (Experimental Physics and Industrial Control System), running on the RTEMS (open-source) hard real-time OS. Various software components of crucial importance are involved:

1. The RTEMS System

   • The RTEMS OS along with a BSP (board support package) for the VGM boards. The BSP was developed at SSRL. An important feature of this BSP is its support for the VME bus (i.e. the Tundra Universe driver).

     Development for RTEMS is done using the GNU tool chain.

   • A BSP extension library (which also supports similar BSPs, such as the MVME23xx). This library provides lowlevel functionality such as address probing, shared PCI interrupts and more.
   • The CEXP library. CEXP is a portable C expression interpreter and symbol-table utility. It builds on top of the GNU BFD (for object file access) and the TECLA (sleak re-entrant line editor) libraries. CEXP also implements a run-time object loader/linker with C++ support. CEXP acts as an interactive shell.
   • "GeSys", an RTEMS 'main' application containing all the relevant parts of RTEMS, the core libraries (such as libc), CEXP and the RTEMS configuration tables (which define static resource limits such as the maximum number of tasks etc.). The 'main' application also takes care of  initializing the system. This 'main' application is also referred to in this article as the 'system application' or 'GeSys' (generic system).
   • Several utilites such as 'telnetd' which are (optionally) loaded into the running system.
   • NFS client support.

2. Application Software

   • An EPICS application, consisting of an object file and several text files ("dbd", "db" and scripts) are loaded into the running RTEMS system and subsequently initialized / started.
3. Bootloader
   • The RTEMS BSP for the VGM computers features a bootloader (which is itself a small, statically linked RTEMS application). The bootloader resides in on-board FLASH memory. The SMON firmware can be used to download and "burn" a flash image.
   • The bootloader supports a set of configuration parameters (such as the board's IP address or the remote file which is to be downloaded and booted). The bootloader can be instructed to fetch the boot parameters either from NVRAM or from a DHCP server.
4. Remote Filesystem Environment
   • The bootloader and the running RTEMS system need to remotely access files on a server in order to download the boot image and the application code and data files. The bootloader is capable of using the TFTP or RSH protocols for downloading an image.

   • One way for accessing remote files under RTEMS is by using TFTP. There is a TFTP filesystem available providing some abstraction from the TFTP protocol. Files on the TFTP filesystem are accessed simply by using the standard open(),read(),write() and close() routines.

     Nevertheless, the TFTP filesystem has some restrictions which are discussed below. OTOH, the TFTP filesystem has the advantage that files can be served out of AFS.

   • NFS is also available.

The RTEMS System

This section describes how generic (i.e. non-EPICS) RTEMS is done. There exists a lot of information at the RTEMS/OAR website about the subject, hence this page is limited to a brief introduction and issues related to dynamic loading with CEXP.

NOTE: EPICS developpers may skip reading this section. The EPICS Makefile system transparently invokes the RTEMS/GNU tool chain. Generating EPICS applications is discussed below.

Compiling RTEMS Applications

The relevant parts of RTEMS at SSRL are installed in AFS "package" space under

/afs/slac/package/rtems/prod/

where the tools and rtems subdirectories are located (besides the prod directory, there is the src directory holding a checked-out tree of the RTEMS sources). As the names imply, the GNU toolchain is installed under tools (currently, the powerpc-rtems target is supported for linux-x86 and solaris-sparc hosts), RTEMS under rtems has been built for the mvme23xx and the svgm powerpc BSPs.

Traditionally, all parts of RTEMS were contained in a big static library which would be linked against the application code. (This has changed a little bit - as explained below.)

RTEMS comes with its own "Makefile system" which works as follows:

1. Your PATH environment variable must contain the path to the GNU tools which are prefixed 'powerpc-rtems-', i.e. after adding

   /afs/slac/package/rtems/prod/tools/@sys/bin

   to your PATH, you can invoke the cross compiler as

   powerpc-rtems-gcc

2. You must point the RTEMS_MAKEFILE_PATH environment variable to your BSP's Makefile.inc. A slightly customized version of this file (along with customized versions of a few other files) is installed under

   /afs/slac/g/spear/rtemsApplications/<version>/config

   hence you must set RTEMS_MAKEFILE_PATH to that directory. The SSRL customized Makefile.inc results in the SVGM BSP being used.

3. You copy a template Makefile to your application directory and edit it as needed. It is recommended that you use an SSRL application's makefile template (Makefile.leaf under /afs/slac/g/spear/rtemsApplications/<version>/config) as the templates distributed with RTEMS use a different install location. Other details are also slightly changed, most notably, the SSRL makefiles know how to build loadable modules.

   The SSRL template Makefile contains two different rules for the PGMS target:

   • If your target has the suffix ".obj", a dynamically loadable object file will be built.
   • If the target has the suffix ".exe", a statically linked, traditional RTEMS application is built.
4. Some explanation of (original, i.e., not SSRL specific) RTEMS makefile variables can be found in

   /afs/slac/package/rtems/prod/rtems/make/README

5. Your application's objects will be built in a "o-optimize" subdirectory. When you do "make install", the application will be installed to

   /afs/slac/g/spear/rtemsApplications/<version>/powerpc-rtems/bin

   where it will be found by the TFTP server for transmission to the target.

   Note that there is also a

   /afs/slac/g/spear/rtemsApplications/<version>/powerpc-rtems/svgm/bin

   which should be used if your application depends on specific features of the VGM board.

What is the difference between a "traditional" (statically linked) RTEMS application and a loadable object?

A "traditional" application contains the application code and RTEMS itself - all symbol references have been resolved by the linker. Such an application is written (by the linker) into a 'xxx.exe' ELF file. The bootloader does not support loading ELF, however (the complete ELF file is also pretty big which would slow down loading). Therefore, the make process also generates a pure binary "xxx.bin" image of the application which is what must be presented to the bootloader. Also, a much smaller ELF file, "xxx.sym", is generated containing the application's symbol table information which can later be used by CEXP (CEXP is also capable of reading the symbols from the full-blown ELF file).

A loadable object OTOH is a normal ELF object containing lots of symbol and relocation information etc.Loadable objects only contain your application's code; they are not linked against RTEMS or libc (nor any of the libraries present in the "system" application). Note that loadable objects are loaded by CEXP (not the bootloader), which knows how to deal with it. Loadable objects get the "xxx.obj" suffix but any ordinary "yyy.o" file can as well be loaded. An ".obj" file is just a collection of ".o" files, linked together with the relocations preserved.

The RTEMS "GeSys" (Generic System) Application

The /afs/slac/g/spear/rtemsApplications/system directory contains a relatively simple RTEMS application which merely fires up the system and starts CEXP. The "system" application is just "the" only generic application which is statically linked and booted. "Real" applications (e.g. EPICS applications) are built as loadable objects and added to the system at run-time. (GeSys is downloadable from outside of SLAC from this page.)

The "system" application consists of two configuration and one initialization source code files, a couple of linker scripts and a Makefile.

RTEMS Configuration Tables

The configuration files take care of "system configuration" in a old-days UNIX or vxWorks sense, i.e. they define a bunch of static limits for various system resources (such as the maximal number of semaphores, tasks, networking buffers, file descriptors etc.). These limits are stored in so called 'configuration tables' which are consulted by the kernel to know how many resources to allocate for the system. Our configuration is pretty generous as the VGM board has plenty of memory and large EPICS applications might be loaded. However, it might turn out that some of the parameters need to be tuned in the future.

Configuration Linker Scripts

Then, there is completely separate aspect of "system configuration" that only is relevant when run-time loading is involved. A statically linked system always has all symbol references resolved by the linker which guarantees that all pieces of code needed by the application are pulled out of libraries, object files etc. and linked into the executable. A system with a dynamic linker, on the other hand, must be built in a different way: it is not enough to resolve all symbol references when putting the executable together as some symbols from the core libraries may be needed in the future (when loading objects into the running system). Assume, for example, that the "system application" does not use 'printf'. When linking against libc (last step of the 'system build' procedure), the static linker will thus not include the 'printf' routine into the executable and an attempt to loading a module which does call 'printf' at run-time will fail due to the unresolved reference to 'printf'.

Therefore, when building the "system application", the (static) linker must be forced to include vital parts of the core libraries into the system executable. What parts are considered 'vital' is the other aspect of "system configuration". This type of configuration is defined by a series of linker scripts found in the 'config' subdirectory of the "system application". These scripts simply list symbols to be considered 'undefined' by the linker who will then search the system libraries for resolving these forced references. The linker scripts are generated using the auxiliary 'Makefile.symlists' and must be manually edited/tailored by commenting out symbols which should not be included into the system build. Here's some more information.

System Application Initialization

The actual code executed by the "system application" is very basic: after performing some basic initialization, CEXP gains control, attempts to execute a couple of initialization/startup scripts and then waits for user input:

1. The network stack is initialized and the TFTP filesystem mounted on "/TFTP"
2. NTP server (time) synchronization is performed
3. Any 'NAME=value' pairs found in the commandline (passed by the bootloader or via DHCP option 129) are stored in the "environment" for retrieval by the application calling 'getenv()'.
4. chdir() into the directory from where the boot image was fetched
5. attempt to load the symbol file (its name must match the boot image name after substituting '.sym' for its filename extension)
6. attempt to execute the "system script" named 'st.sys' (residing in the same directory as the system binary)
7. Try to 'getenv("INIT")'. The value of the 'INIT' variable is expected to denote the path to a 'user script' to be executed by CEXP.
8. if INIT was found, chdir() to its path component and execute the script.
9. start the interactive CEXP interpreter.

Note that failure to load a symbol file is fatal.

The Bootloader

Booting RTEMS on a VGM board is done in four fundamental steps (FDIAG - SMON - netboot - RTEMS). In more detail, these are:

1. After powering-up the board, the FDIAG firmware (which runs out of ROM, no RAM needed) is in control. If the 'boot SMON' flag is set in FDIAG's configuration (use the 'config' command at the FDIAG prompt), FDIAG automatically proceeds to 2.
2. The higher level firmware 'SMON' is expanded from ROM into RAM and gains control.
3. SMON executes the startup script (AKA 'execution buffer') with the magic name "startup". To modify that script, issue 'vi "startup" ' at the SMON prompt. You should exit vi by ':q' _not_ ':w' and reboot after changing the startup script. I normally use a very simple, one-line startup script which merely invokes another script ('execEB "<script>" ' command). This allows for easy switching between boot configurations (e.g. vxWorks vs. RTEMS).
4. "startup" should consist of the line
execEB "rtems";
   vectoring to the "rtems" startup script which prepares the board and fires up the rtems bootloader from flash.
5. The RTEMS bootloader is itself an RTEMS application, called "netboot". Netboot expands and relocates itself from flash into RAM. It then obtains a basic configuration (IP setup & remote boot-file path) either from NVRAM or by using BOOTP (which variant to use is an NVRAM settable option).
6. Netboot loads the final RTEMS binary image using TFTP or RSH from a server. It also stores the boot parameters in a special memory area where they will be picked up by the 'real' system. Netboot then flushes the caches, stops the network interface and branches into the 'real' RTEMS system.
7. The final system moves itself to low memory and boots.

Note: when the interface between Netboot and the BSP changes (as it happened when support for DHCP/BOOTP vendor option 129 was added to RTEMS), Netboot needs to be upgraded to a version compatible with the BSP.

NOTE: The IP configuration of SMON, Netboot and vxWorks-bootrom are maintained in separate NVRAM regions and are hence completely orthogonal. SMON's IP configuration is only used/needed when reprogramming the FLASH (e.g. for installing or upgrading netboot).

Programming Netboot into VGM Flash

A convenience script exists at

/afs/slac/g/spear/rtemsApplications/powerpc-rtems/svgm/img/reflash.st

which can be loaded into VGM NVRAM by (xxx is to be substituted by the TFTP server path to the script)

loadEB "xxx/reflash.st" "reflash"

By simply executing the "reflash" script ('execEB "reflash" ') Netboot will be transparently upgraded to the latest version. (SMON's IP configuration has to be set up, though - use the 'config' command).

More detailed instructions about programming the VGM flash can be found here.

Working with Netboot

Working with netboot is pretty self-explaining. For changing the NVRAM configuration you have to hit a key (note that some keys invoke specific actions as listed/shown by netboot) before the timeout expires (this timeout is also NVRAM configurable). A good key to use is the space bar. Note that there are two countdowns/timeouts when booting a VGM. The first gives you a chance to abort the SMON startup script (not related to netboot), the second one is netboot's countdown.

By hitting the 'c' key, you can enter the 'edit/change' mode where you are prompted for all configurable parameters. There is command line history and full cursor control available. Some special <Ctrl> Key combinations are available for navigating through / committing / canceling your changes.

There are also keys available for temporarily changing netboots behavior:

• 'm' enforces using the NVRAM configuration (although 'normally' BOOTP is used) for 'this one time'.
• 'd' enforces using BOOTP (although 'normally', the NVRAM configuration is in effect).
• 'b' gives you a chance to temporarily change the boot-file / command line parameters without changing the rest of the configuration.

These options are very practical for doing tests etc. when you don't want to touch the configuration but temporarily switch behavior.

The NVRAM configuration is now also accessible from a running RTEMS/GeSys systems using the 'nvramConfigShow()' and 'nvramConfig()' routines. Note that the actual, 'live' configuration of a board may differ from the configuration stored in NVRAM because the user might have used the 'm', 'd' or 'b' keys at boot-time. To obtain information about the current boot configuration, use 'bootConfigShow()'.

Using an Initialization Script

As already explained above, a user initialization script may be passed to the system application by providing 'netboot' with a 'command line parameter'. Examples:

Specify an absolute path on the 'server' (i.e. the same host where the boot file was loaded from):

INIT=/TFTP/BOOTP_HOST/epics/myioc/st.cmd

Specify an absolute path on any host with a 'tftp' server:

INIT=/TFTP/192.42.249.188/rtems-epics/someioc/myinifile

Use a path relative to the directory where the system was loaded from.

INIT=../../applications/myapp/ini

Please note that vxWorks initialization scripts may need to be modified in order to be compliant with CEXP syntax.

Setting Up a DHCP/BOOTP Server

Here's a dhcpd.conf file snippet...

Links

back to top.