SPEAR EPICS

IOC Console Access

Introduction
Definitions
User's Guide
System Administration

Introduction

The iocConsole program is inspired by the ASD Standard Procedure for IOC Console Access. Although both the SLAC and ASD versions are conceptually the same and both utilize secure shell (ssh) and the screen program, there are enough significant differences in implementation to warrant a separate description here.

The iocConsole program provides a common mechanism for connecting to IOC console serial lines as well as IOCs running as a standalone task on a Unix workstation or server. It uses the GNU screen program which provides useful features for managing IOC console sessions:

The telnet, rlogin, ssh, or iocsh session on the IOC console continues to run even when no users are attached to the screen session. IOC console output is captured, optionally logged to file, and available to the next iocConsole session that attaches.
It maintains a scrollback history buffer so IOC console output can be retreived even if it has scrolled off the top of your terminal window. It is also possible to search the scrollback buffer making it easy to find diagnostic messages that scrolled by earlier.
It allows multiple simultaneous connections so that more than one user at a time can be viewing IOC console output. Anything typed in on one session appears in all of them.

Below is a system diagram showing the elements involved in three different types of IOC console access.

Definitions

There are two kinds of IOCs:

Hard IOC

RTEMS

VxWorks

Soft IOC

Each IOC must have a unique name. The name does not necessarily have to be the CPU node name and could be an alias name.

There are four kinds of IOC consoles:

Hard IOC Connected to a Terminal Server

Hard IOC not Connected to a Terminal Server

Soft IOC with bootparams File

Soft IOC without bootparams File

User's Guide

Before using iocConsole for the first time, the user must generate a public RSA key for their Unix account and the owner of the project's shared Unix account must add that key to the shared account's authorized key list. This is required since all screen programs are run under the shared account ($IOC_OWNER). Consult your EPICS system manager for the procedure.

ONCE YOU ARE ATTACHED TO A CONSOLE, USE C-a d (control-a d) TO DETACH.

Connect to an IOC defined in $IOC_SCREEN/screeniocs

To connect to a hard or soft IOC that is defined in $IOC_SCREEN/screeniocs or a soft IOC with a bootparams file, type at the prompt:
iocConsole.pl <iocname>

Connect to an IOC not defined in $IOC_SCREEN/screeniocs

To connect to a hard IOC connected to a terminal server, type at the prompt:
iocConsole.pl <iocname> <terminal server> <port> <screen host>
where:
<terminal server> is the terminal server node name
<port> is the port* on the terminal server
*if using an ssh terminal server, prefix port number with 'ssh' (e.g. ssh3033)
<screen host> is the host where screen is run

To connect to a hard IOC not connected to a terminal server, type at the prompt:
iocConsole.pl <iocname> <ioc node name> <telnet or rlogin> <screen host>
where:
<ioc node name> is the ioc DNS node name
<telnet or rlogin> is the command used to connect directly to the IOC
<screen host> is the host where screen is run

To connect to a non-bootparams soft IOC, type at the prompt:
iocConsole.pl <iocname> <executable name> sioc <screen host>
where:
<executable name> is the unique IOC executable name, including path
sioc indicates this is a soft IOC
<screen host> is the host where screen is run

Notes on Connecting

For a hard IOC, a screen process is started, if it is not already started, and will remain up until a cleanup is done. For a non-bootparams soft IOC, the IOC will be started, if it is not already up, and will remain up until a cleanup is done. For a bootparams soft IOC, the IOC must already be started.

For most terminal servers, a password must be provided the first time the screen process is started. If someone has an existing telnet connection to the hard IOC terminal server port (or to the hard IOC itself if it's not connected to a terminal server), iocConsole will not be able to connect.

If you ssh as the shared account on the screen host, the ssh connnection is not required and is not made. The console access will be a little faster.

Kill the IOC Console

To remove the IOC console and all sessions attached to it or to kill a non-bootparams soft IOC, type at the prompt:
iocConsole.pl -cleanup <same arguments as connect>

Start IOC Console with Auto-Restart

To add the auto-restart feature to the IOC console or non-bootparams soft IOC so that the console is automatically restarted if the terminal server is rebooted, "quit" is typed at the telnet prompt, "exit" is typed in a soft IOC shell, or the soft IOC dies for some reason, type at the prompt:
iocConsole.pl -stayup <same arguments as connect>

Screen Commands

Everything typed to the screen session is sent to the program running in the current window. The only exception is the one keystroke that is used to initiate a command to the screen session manager itself. By default each of these commands begins with a control-a (abbreviated C-a from now on) and is followed by one other keystroke.

The most important screen commands are:

C-a d : Detach the screen session from this terminal (C-a C-d also works).
C-a ? : Show all screen commands.
C-a * : Show a list of all users attatched to this screen session.
C-a H : Toggle screen logging on and off. This is a good general tool to re-start screenlogging when necessary.
C-a [ : Enter copy/scrollback mode (C-a C-[ or C-a <esc> also works). In copy/scrollback mode you can use 'vi' or 'emacs' commands to move around the scrollback history. Some useful scrollback mode commands are:
- Arrow keys move line by line or column by column.
- h, j, k, l move line by line or column by column (vi-style).
- + - move one line up/down.
- C-u C-d move up/down one half a screen.
- C-b C-f move up/down a full screen.
- g moves to the beginning of the scrollback history.
- % preceded by a number jumps to that percentage of the scrollback.
- / or ? followed by some characters followed by a return searches forward or backwards for those characters.
- <esc> exits copy/scrollback mode.

The entire list of screen commands can be found in the screen man page.

Examine Screen Log Files

If the screen program is configured so that all console input and output is also logged to a file, then these files can be examined without needing to connect to the IOC console. The log file will reside in NFS (defined by $IOC_INFO or $IOC_DATA) and will reside under the directory of the iocname if it exists.

System Administration

Environment Variables

$SCREENBIN : Location of the screen executable. If not defined, iocConsole looks for screen in /opt/screen/bin.
$IOC_OWNER : Shared account used by iocConsole. If not defined, iocConsole uses "cdioc".
$IOC_SCREEN : Location of the screeniocs and screenrc file. If not defined, the location defaults to the directory of the iocConsole.pl executable.
$IOC_INFO or $IOC_DATA : Location of the bootparams file for soft IOCs. Location used for writing the screenlog file. Location from which a soft IOC is started so that it will contain the core dump file if the soft IOC crashes. If not defined, the location defaults to the current working directory. Note that this area must be NFS since AFS requires token refresh every 25 hours and the detached screen process will be up an indefinite period of time.
$IOC : Location of the iocBoot directory used in starting a soft IOC. If not defined, the location defaults to current working directory.
$IOC_LOGSCRIPT: Script invoked from iocConsole to log the new iocConsole session. If not defined, logging is disabled. Suggested logging from the specified script can be directed to unix applications such as an electronic log or cmlog.

Installing Screen

The screen source and description is available at the GNU screen site. For taylored Linux machines, screen is provided by Red Hat. In order to accommodate the multiuser mode required for iocConsole, the screen executable must be installed with SUID access mode and access to the screen program must be done using a shared account. The screen program is installed locally by the system manager on the screen host in $SCREENBIN. It must be owned by root and have the SUID bit set (ie, "chmod u+s /opt/screen/bin/screen").

Socket Directory (/tmp/screens)

Since screen runs with SUID access mode, it keeps its sockets under /tmp/screens/S-<shared-account-name>. The /tmp/screens directory cannot be overridden by the $SCREENDIR environment variable. It must never be deleted and files under it must never be removed while screen is running.

Screen Configuration

Screen can be configured to special actions such as sending all input and output to a log file ("deflog on") and logging timestamps when there is new activity or there's been no activity for a while ("logtstamp on"). The screen man page has an extensive list of configuration options which can be added to the screenrc configuration file. The iocConsole program first looks for the configuration file called $IOC_SCREEN/<iocname>/screenrc and, if it's not there, then looks for $IOC_SCREEN/screenrc. If neither file is provided, screen itself will look at $SCREENRC and finally $HOME/.screenrc of the shared account.

If screen log files are created, periodically deletion or compression of screenlog files should be done using a cronjob. If the IOC is in trouble, it may be logging many messages which could quickly use up space in the NFS area.

$IOC_SCREEN/screeniocs

A file listing all IOCs may be used so that iocConsole can be called with just the IOC name and no further arguments. This file must have one line per IOC:

Hard IOC Connected to a Terminal Server:
<iocname> <terminal server> <port> <screen host>
Hard IOC Connected to an "ssh Terminal Server":
<iocname> <terminal server> <sshport*> <screen host>
Hard IOC not Connected to a Terminal Server:
<iocname> <ioc node name> <rlogin or telnet> <screen host>
Soft IOC:
<iocname> <executable name> sioc <screen host>

*port number is prefixed with 'ssh' (e.g. ssh3033) to identify it as an ssh terminal server

Non-Bootparams Soft IOC Startup and Core Dump

To start a non-bootparams soft IOC, the executable name must be unique. This may be achieved by adding a symbolic link. For example, <$IOC>/<iocname>/bin/<executable file name> can be used as the executable name but requires that the "bin" symbolic link be created.

The startup file is assumed to be <$IOC>/<iocname>/st.cmd. Production soft IOCs should be started as part of bootup of the screen host. To start the IOC, the detached screen calls siocStart.pl. If the stayup option is set, this script will automatically restart the IOC if the IOC went down without an error (ie, somebody typed "exit" at the IOC shell) or if it went down with an error and the number of restarts in a row with an error is less than 100.

If the soft IOC crashes with a core dump, the "core" file will reside in NFS (defined by $IOC_INFO or $IOC_DATA) and will reside under the directory of the iocname if it exists.

Source for iocConsole

Source files are perl scripts which reside in $EPICS/extensions/src/iocConsole (slaconly) and are installed in $EPICS/extensions/bin/<host_arch>. Users may also download the public iocConsole.tar.gz file and build it as an EPICS extension. The scripts assume perl is installed in /usr/local/bin.

Contact: Stephanie Allison
Last Modified: Feb 16, 2010