LCLS Hard IOC Release Procedure

I. Configuring your IOC CPU to run RTEMS and EPICS

II. Releasing a new hard IOC to the production network

III. Adding a new hard IOC to an existing application

I. Configuring your IOC CPU to run RTEMS and EPICS

If it has not already been done, you will need to configure your CPU to run RTEMS and EPICS. Instructions to do this for VME and coldfire CPUs are located on the Controls wiki, at the url below, under the ‘Getting Started’ section. This section also contains NVRAM configuration examples for production and development environments for both rtems-4.7.1 and rtems-4.9.1. You will need to be on the SLAC network to view this page:

http://lcls-dev/tiki-index.php?page=LCLS+EPICS+IOC+Processors&highlight=nvram

II. Releasing a new hard IOC to the production network

These instructions tell you how to release a new hard IOC to the production network. Making a mistake at some point in this process could cause a lot of problems in the production control system. If you are new at this or if you don't fully understand all the steps, you should have somebody helping you to make sure you don't make any mistakes. Be very careful!

Major steps (details below):

1. Set up directories and create the hard IOC in the development environment. Boot it and test it on the development server as much as possible.

2. Import the new IOC into CVS.

3. Go to the production network (lcls-builder). Create a new directory for your new IOC under iocTop. Check out your IOC from CVS here.

4. Create a data directory for your IOC.

5. Create a boot directory under iocCommon for your new IOC. Set it up to point to the new IOC you just set up under iocTop.

6. Add your new IOC to screeniocs file, the configuration file for iocConsole.

7. Use iocConsole to log into your IOC.

8. Set up your IOC to use the channel watcher if needed.

9. Use CVS to create an official tagged release of your IOC.

10. Check your tagged release of the IOC out of CVS into your IOC's directory under iocTop.

11. Change the startup file boot path under iocCommon to point to your new tagged release.

12. Boot your IOC again using iocConsole and test.

13. Copy displays to $EDM and commit to CVS.

14. Ask Judy Rock to add PVs to the alarm handler as needed.

15. Ask Bob Hall to add PVs to the channel archiver as needed.

16. Ask Sonya Hoobler to add your new IOC to the Network section of LCLSHome.

17. Send a message to controls-software-release to announce your new IOC.

Details:

1. Set up directories and create the IOC in the development environment. Boot it and test it as much as possible.

a. Log into development environment:

Log into lcls-dev2.

If your shell default is tcsh, then type bash to change to bash shell.

source /afs/slac/g/lcls/tools/script/ENVS.bash/ (or add this to your .bashrc)

b. Create application directories in the development environment:

* make a development directory for your application:

cd $EPICS_IOC_TOP (/afs/slac/g/lcls/epics/iocTop)

mkdir <moduleName> (for example Alarms)

cd <moduleName>

mkdir Development

cd Development

(to work on an existing application, cd <moduleName> and cvs checkout the desired release)

c. Create the IOC using the makeBaseApp.pl script. Use the "lcls", "ioc", or "example" template. For example (for RTEMS-beatnik architecture):

makeBaseApp.pl -t lcls -a RTEMS-beatnik <appname>

makeBaseApp.pl -i -t lcls -a RTEMS-beatnik <appname>

d. Get a good copy of configure/RELEASE from one of the development IOCs and modify it so that it includes any modules that your IOC needs.

Make sure EPICS_SITE_TOP is set to the development directory:

/afs/slac/g/lcls/epics

e. Modify <appname>App/src/Makefile to load libraries and dbd files from the modules your IOC needs.

f. Rename the directory iocBoot/ioc<appname> to iocBoot/ioc-<area>-<sys>##, following the naming convention

g. Create databases and displays and modify st.cmd as needed.

h. Make displays. These can be optionally saved with the IOC under

<appname>App/srcDisplay.

i. Build the IOC.

j. Now create boot and data directories in the development environment:

* make a boot directory:

cd $IOC

mkdir <dev iocnodename> (for example ioc-bsy0-mg01)

cd <dev iocnodename>

create a startup.cmd file (see example in $IOC on production)

use Dev versions of pre- and post-st.cmd scripts:

# Execute dev pre_st.cmd script

< /afs/slac/g/lcls/epics/iocCommon/All/Dev/pre_st.cmd

...

# Execute dev post_st.cmd script

< /afs/slac/g/lcls/epics/iocCommon/All/Dev/post_st.cmd

* create data and iocInfo directories for your ioc

cd $IOC_DATA/epics/ioc/data

if there is no data directory for your ioc already, create one

mkdir <iocnodename>

cd <iocnodename>

mkdir iocInfo

k. IOCs on the Development network must be given NFS write access in order to write to the data directory. Fill out the following form to request NFS access:

http://www.slac.stanford.edu/comp/unix/nfs-access-req.html

l. Add your IOC to the development section of the $IOC/screeniocs file so that it can be booted from iocConsole.

Now you're ready to test away!

2. Import the new IOC into CVS.

a. Choose a module name for the IOC. Module names are stored in

$CVSROOT/CVSROOT/modules - see the names there for examples.

b. Check out the modules file from CVS into a temporary working directory: cvs

co -d <temporarydirectoryname> CVSROOT/modules

c. Edit your local copy of modules file and add your new module. The location of your IOC in CVS will be epics/ioc/<modulename>.

d. Commit the modules file back to CVS: cvs commit -m "add new ioc <modulename>" modules

e. Import your IOC into CVS. From the top directory of your IOC application, type:

make clean uninstall

cvs import epics/ioc/<modulename> CD_SOFT R1_0

f. Check that your import was successful. In a working directory, type:

cvs co -d <anything> <modulename>. Then cd <anything> and build.

Run the IOC to test.

3. Go to the production network (lcls-builder). Create a new directory for your new IOC under iocTop. Check out your IOC from CVS here.

a. From lcls-prod02, ssh iocegr@lcls-builder.

b. Set the environment variable CVSROOT:

export CVSROOT=:ext:<username>@lcls-prod02:/afs/slac/g/lcls/cvs

c. Have Jingchen or Ernest create a directory for your project under $EPICS_IOC_TOP (/usr/local/lcls/epics/iocTop)

d. cd <modulename>

e. Check out your IOC from cvs: cvs co -d Devl <modulename>

f. cd Devl

g. modify configure/RELEASE so that EPICS_SITE_TOP=/usr/local/lcls/epics

h. make

i. Boot the IOC and test. Your are running your IOC on the production network.

Be careful to avoid PV name collisions!

4. Create a data directory for your IOC.

a. Have Jingchen or Ernest create a directory for your IOC under

$IOC_DATA (/u1/lcls/epics/ioc/data). They will perform steps b-d.

b. Create a directory with your IOC's nodename. This nodename must match the name of the subdirectory under iocBoot in your IOC: mkdir <nodename>

c. cd <nodename>

d. Create a directory for IOC info: mkdir iocInfo

5. Create a boot directory under iocCommon for your new IOC. Set it up to point to the new IOC you just set up under iocTop.

a. Have Jingchen or Ernest create a directory for your IOC under $EPICS_IOCS (/usr/local/lcls/epics/iocCommon). They will perform step b.

b. Create a boot directory for your IOC. The name of this directory must follow the IOC naming convention, and it must match the subdirectory under iocBoot.:

mkdir ioc-<area>-<sys>##

c. Go to your new directory: cd ioc-<area>-<sys>##. Create a startup file for your IOC. Use ioc-in20-bp01 as a template: cp ../ioc-in20-bp01/startup.cmd startup.cmd

d. Modify the startup.cmd file. Replace the date and username with your own. Modify the line beginning with chdir so that it goes to your IOC's boot directory.

6. Add your new IOC to screeniocs file, the configuration file for iocConsole.

a. Have Jingchen or Ernest edit screeniocs to add your new IOC to iocConsole. You will need to provide them with your IOC nodename, the terminal server nodename, and the terminal server port number.

7. Use iocConsole to boot your IOC.

a. Use iocConsole to log into your IOC: iocConsole <nodename>

b. Power up your IOC (if you haven’t already). To reboot your ioc, type bsp_reset() (for rtems-4.9.1 or later) or rtems_reboot() (for earlier versions of rtems).

c. Verify that your IOC's info is being written to the data directory:
ls /u1/lcls/epics/ioc/data/<nodename>/iocInfo. You should see several files with names starting with IOC.

8. Set up your IOC to use the channel watcher if needed.

a. <details to be added later>

9. Use CVS to create an official tagged release of your IOC. (This step should be done in the development environment, if possible.)

a. Go back to the top directory of your IOC:

cd /usr/local/lcls/epics/iocTop/<modulename>/Devl

b. Commit any changes you have made to cvs: cvs commit -m "prepare for initial release"

c. Create a cvs tag for your IOC: cvs tag <modulename>-R1-0-0

10. Check your tagged release of the IOC out of CVS into your IOC's directory under iocTop.

a. Go up one level, to your <modulename> directory: cd ..

b: Check out your new tagged release from CVS: cvs co -d <modulename>-R1-0-0 -r <modulename>-R1-0-0 <modulename>

c. Go to the top of the new tagged release of your IOC: cd <modulename>-R1-0-0

d. Build: make

11. Change the path in startup.cmd under iocCommon to point to your new tagged release.

a. cd /usr/local/lcls/epics/iocCommon/<nodename>

b. Modify startup.cmd to point to the boot directory in your tagged release.

12. Boot your IOC again using iocConsole and test.

a. iocConsole <nodename>

b. To exit iocConsole and leave the IOC running, type ctrl-a then ctrl-d. This is the way you must always exit iocConsole for a production IOC, whether it is a soft IOC or a VME IOC. You must never type exit or ctrl-c or ctrl-y in a production soft IOC, because this will kill the IOC and cause it to reboot. Watch out! This can be very disruptive to operations!

13. Copy displays to $EDM and commit to CVS.

a. All production EDM displays are stored in one of the subdirectories under $EDM. The subdirectory $EDM/lcls is reserved for files related to the top level lclshome display, so don't put your files there. Instead find the appropriate subdirectory for your subsystem and copy the files there: cp *.edl $EDM/<subdirectory>.

b. Go to the subdirectory and add the files to CVS and commit:

cd $EDM/<subdirectory>; cvs add <filename>; cvs commit -m "initial import (or other reason)" <filename>

14. Ask Judy Rock to add PVs to the alarm handler as needed.

If you need to add some new PVs to the alarm handler, send Judy an email with a list of PVs broken down by subsystem and alarm handler region. Detailed instructions are on the wiki page "LCLS Use of the Alarm Handler" http://lcls-dev/tiki-index.php?page=LCLS+Alarm+Handling

15. Ask Bob Hall to add PVs to the channel archiver as needed.

If you want to add PVs to the archiver, send Bob an email with a list of PVs and the name of your subsystem. You also need to specify the method that the alarm handler should use to watch these PVs; typically "monitor" is fine for all PVs. You will also need to specify the rate at which the archiver might expect to see the data change; typically 1 second is fine.

16. Ask Sonya Hoobler to add your new IOC to the Network section of LCLS Home. Send Sonya an email with the name of your new IOC, its location (including building number, rack number, rack elevation), and the type of VME crate it is in. She will add it to the Network pages under lclshome, and to the soft IOC that monitors all the production IOCs.

17. Send a message to controls-software-release to announce your new IOC.

Your release message will automatically be written to the software section of the operations elog.

----------------------------------

You're done!

----------------------------------

===============================================================================

III. Adding a new IOC to an existing application

in Development:

1. Check out the application in dev, and add the IOC directory

a. on lcls-dev2, checkout the default version of the app (e.g. Alarms)

cvs co -d Development <app name>

b. cd Development/iocBoot

c. mkdir <new ioc nodename> (e.g. ioc-li25-bl01)

d. Copy Makefile from an existing IOC into the new iocBoot dir

e. Create (or clone) st.cmd for the new IOC in the new iocBoot dir

2. in $IOC create boot directory and files for the new IOC - see section I above

3. in $IOC_DATA create IOC data directory for the new IOC - see section I above

4. boot the new IOC and make sure that it boots, and that iocInfo files are created in the right place - see section I above

5. Add the new IOC to CVS:

a. cd Development/iocBoot

cvs add <ioc nodename>

cvs commit

cd <ioc nodename>

cvs add *

cvs commit

in Prod:

6. cvs checkout on prod

7. create boot directory and files and symbolic links for the new IOC

8. create data directory for the new IOC

9. boot the new IOC to make sure it works

10. add new IOC to screeniocs, and cvs commit this file (Ernest or Jingchen will commit it to cvs if they make the changes for you.)

11. create ChannelWatcher config file for the new IOC, and edit Master.XX Channel Watcher master file to include the new IOC's config file and data dirs.

12. Make a new tagged version on prod with the new IOC and point boot links there - see section I above for details

===================================================================

last modified 1/8/2010 4:21 PM