The EPICS on RTEMS @ SSRL How-To

Till Straumann <strauman_at_slac.stanford.edu>, last modified on 2004/5/19

Introduction

At SSRL, we use VGM series VME single board computers by Synergy microsystems (featuring a G3/G4 PowerPC CPU) for the SPEAR3 electron storage ring control system.

On the software side, we use EPICS (Experimental Physics and Industrial Control System), running on the RTEMS (open-source) hard real-time OS. Various software components of crucial importance are involved:

  1. The RTEMS System

  2. Application Software

  3. Bootloader
  4. Remote Filesystem Environment

The RTEMS System

This section describes how generic (i.e. non-EPICS) RTEMS is done. There exists a lot of information at the RTEMS/OAR website about the subject, hence this page is limited to a brief introduction and issues related to dynamic loading with CEXP.

NOTE: EPICS developpers may skip reading this section. The EPICS Makefile system transparently invokes the RTEMS/GNU tool chain. Generating EPICS applications is discussed below.

Compiling RTEMS Applications

The relevant parts of RTEMS at SSRL are installed in AFS "package" space under

/afs/slac/package/rtems/prod/

where the tools and rtems subdirectories are located (besides the prod directory, there is the src directory holding a checked-out tree of the RTEMS sources). As the names imply, the GNU toolchain is installed under tools (currently, the powerpc-rtems target is supported for linux-x86 and solaris-sparc hosts), RTEMS under rtems has been built for the mvme23xx and the svgm powerpc BSPs.

Traditionally, all parts of RTEMS were contained in a big static library which would be linked against the application code. (This has changed a little bit - as explained below.)

RTEMS comes with its own "Makefile system" which works as follows:

  1. Your PATH environment variable must contain the path to the GNU tools which are prefixed 'powerpc-rtems-', i.e. after adding

    /afs/slac/package/rtems/prod/tools/@sys/bin

    to your PATH, you can invoke the cross compiler as

    powerpc-rtems-gcc

  2. You must point the RTEMS_MAKEFILE_PATH environment variable to your BSP's Makefile.inc. A slightly customized version of this file (along with customized versions of a few other files) is installed under

    /afs/slac/g/spear/rtemsApplications/<version>/config

    hence you must set RTEMS_MAKEFILE_PATH to that directory. The SSRL customized Makefile.inc results in the SVGM BSP being used.

  3. You copy a template Makefile to your application directory and edit it as needed. It is recommended that you use an SSRL application's makefile template (Makefile.leaf under /afs/slac/g/spear/rtemsApplications/<version>/config) as the templates distributed with RTEMS use a different install location. Other details are also slightly changed, most notably, the SSRL makefiles know how to build loadable modules.

    The SSRL template Makefile contains two different rules for the PGMS target:

  4. Some explanation of (original, i.e., not SSRL specific) RTEMS makefile variables can be found in

    /afs/slac/package/rtems/prod/rtems/make/README

  5. Your application's objects will be built in a "o-optimize" subdirectory. When you do "make install", the application will be installed to

    /afs/slac/g/spear/rtemsApplications/<version>/powerpc-rtems/bin

    where it will be found by the TFTP server for transmission to the target.

    Note that there is also a

    /afs/slac/g/spear/rtemsApplications/<version>/powerpc-rtems/svgm/bin

    which should be used if your application depends on specific features of the VGM board.

What is the difference between a "traditional" (statically linked) RTEMS application and a loadable object?

A "traditional" application contains the application code and RTEMS itself - all symbol references have been resolved by the linker. Such an application is written (by the linker) into a 'xxx.exe' ELF file. The bootloader does not support loading ELF, however (the complete ELF file is also pretty big which would slow down loading). Therefore, the make process also generates a pure binary "xxx.bin" image of the application which is what must be presented to the bootloader. Also, a much smaller ELF file, "xxx.sym", is generated containing the application's symbol table information which can later be used by CEXP (CEXP is also capable of reading the symbols from the full-blown ELF file).

A loadable object OTOH is a normal ELF object containing lots of symbol and relocation information etc.Loadable objects only contain your application's code; they are not linked against RTEMS or libc (nor any of the libraries present in the "system" application). Note that loadable objects are loaded by CEXP (not the bootloader), which knows how to deal with it. Loadable objects get the "xxx.obj" suffix but any ordinary "yyy.o" file can as well be loaded. An ".obj" file is just a collection of ".o" files, linked together with the relocations preserved.

The RTEMS "GeSys" (Generic System) Application

The /afs/slac/g/spear/rtemsApplications/system directory contains a relatively simple RTEMS application which merely fires up the system and starts CEXP. The "system" application is just "the" only generic application which is statically linked and booted. "Real" applications (e.g. EPICS applications) are built as loadable objects and added to the system at run-time. (GeSys is downloadable from outside of SLAC from this page.)

The "system" application consists of two configuration and one initialization source code files, a couple of linker scripts and a Makefile.

RTEMS Configuration Tables

The configuration files take care of "system configuration" in a old-days UNIX or vxWorks sense, i.e. they define a bunch of static limits for various system resources (such as the maximal number of semaphores, tasks, networking buffers, file descriptors etc.). These limits are stored in so called 'configuration tables' which are consulted by the kernel to know how many resources to allocate for the system. Our configuration is pretty generous as the VGM board has plenty of memory and large EPICS applications might be loaded. However, it might turn out that some of the parameters need to be tuned in the future.

Configuration Linker Scripts

Then, there is completely separate aspect of "system configuration" that only is relevant when run-time loading is involved. A statically linked system always has all symbol references resolved by the linker which guarantees that all pieces of code needed by the application are pulled out of libraries, object files etc. and linked into the executable. A system with a dynamic linker, on the other hand, must be built in a different way: it is not enough to resolve all symbol references when putting the executable together as some symbols from the core libraries may be needed in the future (when loading objects into the running system). Assume, for example, that the "system application" does not use 'printf'. When linking against libc (last step of the 'system build' procedure), the static linker will thus not include the 'printf' routine into the executable and an attempt to loading a module which does call 'printf' at run-time will fail due to the unresolved reference to 'printf'.

Therefore, when building the "system application", the (static) linker must be forced to include vital parts of the core libraries into the system executable. What parts are considered 'vital' is the other aspect of "system configuration". This type of configuration is defined by a series of linker scripts found in the 'config' subdirectory of the "system application". These scripts simply list symbols to be considered 'undefined' by the linker who will then search the system libraries for resolving these forced references. The linker scripts are generated using the auxiliary 'Makefile.symlists' and must be manually edited/tailored by commenting out symbols which should not be included into the system build. Here's some more information.

System Application Initialization

The actual code executed by the "system application" is very basic: after performing some basic initialization, CEXP gains control, attempts to execute a couple of initialization/startup scripts and then waits for user input:

  1. The network stack is initialized and the TFTP filesystem mounted on "/TFTP"
  2. NTP server (time) synchronization is performed
  3. Any 'NAME=value' pairs found in the commandline (passed by the bootloader or via DHCP option 129) are stored in the "environment" for retrieval by the application calling 'getenv()'.
  4. chdir() into the directory from where the boot image was fetched
  5. attempt to load the symbol file (its name must match the boot image name after substituting '.sym' for its filename extension)
  6. attempt to execute the "system script" named 'st.sys' (residing in the same directory as the system binary)
  7. Try to 'getenv("INIT")'. The value of the 'INIT' variable is expected to denote the path to a 'user script' to be executed by CEXP.
  8. if INIT was found, chdir() to its path component and execute the script.
  9. start the interactive CEXP interpreter.

Note that failure to load a symbol file is fatal.

The Bootloader

Booting RTEMS on a VGM board is done in four fundamental steps (FDIAG - SMON - netboot - RTEMS). In more detail, these are:

  1. After powering-up the board, the FDIAG firmware (which runs out of ROM, no RAM needed) is in control. If the 'boot SMON' flag is set in FDIAG's configuration (use the 'config' command at the FDIAG prompt), FDIAG automatically proceeds to 2.
  2. The higher level firmware 'SMON' is expanded from ROM into RAM and gains control.
  3. SMON executes the startup script (AKA 'execution buffer') with the magic name "startup". To modify that script, issue 'vi "startup" ' at the SMON prompt. You should exit vi by ':q' _not_ ':w' and reboot after changing the startup script. I normally use a very simple, one-line startup script which merely invokes another script ('execEB "<script>" ' command). This allows for easy switching between boot configurations (e.g. vxWorks vs. RTEMS).
  4. "startup" should consist of the line
    execEB "rtems";
    vectoring to the "rtems" startup script which prepares the board and fires up the rtems bootloader from flash.
  5. The RTEMS bootloader is itself an RTEMS application, called "netboot". Netboot expands and relocates itself from flash into RAM. It then obtains a basic configuration (IP setup & remote boot-file path) either from NVRAM or by using BOOTP (which variant to use is an NVRAM settable option).
  6. Netboot loads the final RTEMS binary image using TFTP or RSH from a server. It also stores the boot parameters in a special memory area where they will be picked up by the 'real' system. Netboot then flushes the caches, stops the network interface and branches into the 'real' RTEMS system.
  7. The final system moves itself to low memory and boots.

Note: when the interface between Netboot and the BSP changes (as it happened when support for DHCP/BOOTP vendor option 129 was added to RTEMS), Netboot needs to be upgraded to a version compatible with the BSP.

NOTE: The IP configuration of SMON, Netboot and vxWorks-bootrom are maintained in separate NVRAM regions and are hence completely orthogonal. SMON's IP configuration is only used/needed when reprogramming the FLASH (e.g. for installing or upgrading netboot).

Programming Netboot into VGM Flash

A convenience script exists at

/afs/slac/g/spear/rtemsApplications/powerpc-rtems/svgm/img/reflash.st

which can be loaded into VGM NVRAM by (xxx is to be substituted by the TFTP server path to the script)

loadEB "xxx/reflash.st" "reflash"

By simply executing the "reflash" script ('execEB "reflash" ') Netboot will be transparently upgraded to the latest version. (SMON's IP configuration has to be set up, though - use the 'config' command).

More detailed instructions about programming the VGM flash can be found here.

Working with Netboot

Working with netboot is pretty self-explaining. For changing the NVRAM configuration you have to hit a key (note that some keys invoke specific actions as listed/shown by netboot) before the timeout expires (this timeout is also NVRAM configurable). A good key to use is the space bar. Note that there are two countdowns/timeouts when booting a VGM. The first gives you a chance to abort the SMON startup script (not related to netboot), the second one is netboot's countdown.

By hitting the 'c' key, you can enter the 'edit/change' mode where you are prompted for all configurable parameters. There is command line history and full cursor control available. Some special <Ctrl> Key combinations are available for navigating through / committing / canceling your changes.

There are also keys available for temporarily changing netboots behavior:

These options are very practical for doing tests etc. when you don't want to touch the configuration but temporarily switch behavior.

The NVRAM configuration is now also accessible from a running RTEMS/GeSys systems using the 'nvramConfigShow()' and 'nvramConfig()' routines. Note that the actual, 'live' configuration of a board may differ from the configuration stored in NVRAM because the user might have used the 'm', 'd' or 'b' keys at boot-time. To obtain information about the current boot configuration, use 'bootConfigShow()'.

Using an Initialization Script

As already explained above, a user initialization script may be passed to the system application by providing 'netboot' with a 'command line parameter'. Examples:

Specify an absolute path on the 'server' (i.e. the same host where the boot file was loaded from):

INIT=/TFTP/BOOTP_HOST/epics/myioc/st.cmd

Specify an absolute path on any host with a 'tftp' server:

INIT=/TFTP/192.42.249.188/rtems-epics/someioc/myinifile

Use a path relative to the directory where the system was loaded from.

INIT=../../applications/myapp/ini

Please note that vxWorks initialization scripts may need to be modified in order to be compliant with CEXP syntax.

Setting Up a DHCP/BOOTP Server

Here's a dhcpd.conf file snippet...

Links

back to top.