CRAM migration guide

Preparing your current IOC deployment for cram

  1. Choose a name for your software package. We should use the same name in all the build and deployment tools
    1. This should be the module name in CVS.
    2. This should be the name used with eco.
    3. This should be the name of the folder in iocTop where the releases are located.
  2. For each applicable facility, in iocTop/packageName, create the current softlink and point it to the release that is currently being used in production.
    drwxrwxr-x  9 iocegr  lcls MatlabSupport-R3-0-0-0
    drwxrwxr-x  9 softegr lcls MatlabSupport-R3-1-0-0
    drwxrwxr-x 10 softegr lcls MatlabSupport-R3-2-0-0
    lrwxrwxrwx  1 softegr lcls current -> MatlabSupport-R3-2-0-0
    [softegr@lcls-builder MatlabSupport]$
  3. For each applicable facility, in iocCommon, for each IOC belonging to the software package, create a iocSpecificRelease softlink and point it to the current softlink defined above. You must use relative paths for these softlinks.
    lrwxrwxrwx st.cmd -> startup.cmd
    -rwxrwxr-x startup.cmd
    lrwxrwxrwx iocSpecificRelease -> ../../iocTop/MatlabSupport/current
    [softegr@lcls-builder sioc-sys0-ml00]$
    • For softIOC's, remove the bin softlink to the iocTop area (many softIOC's used this approach) if you previously had one. It does not hurt to have the bin softlink but it is confusing and it makes it very hard to debug cram failures. The recommended way to refer to collateral in iocTop is to use the iocSpecificRelease link.
  4. For softIOC's, for each IOC belonging to the software package, change startup.cmd to use the iocSpecificRelease softlink. You will need to change the hashbang portion of the startup script by replacing #!bin/linux-x86 with #!iocSpecificRelease/bin/linux-x86
    [softegr@lcls-builder sioc-sys0-ml00]$ cat startup.cmd 
    #!iocSpecificRelease/bin/linux-x86/matlabSupport
    #==============================================================
    #
    #  Abs:  EPICS Soft IOC Startup Script for sioc-sys0-ml00
    in addition to location from where iocBoot/st.cmd is loaded where bin/../../iocBoot/ is replaced with iocSpecificRelease/iocBoot. Note this listing has some portions alterted to make it fit.
    # Go to IOC top directory and execute IOC startup script
    cd <iocCommon>/sioc-sys0-ml00/iocSpecificRelease/iocBoot/sioc-sys0-ml00
    < st.cmd 
    
    # Execute soft_post_st.cmd script
    < /usr/local/lcls/epics/iocCommon/All/Prod/soft_post_st.cmd
  5. For softIOC's, you'll also need to change screeniocs to use the iocSpecificRelease softlink instead of the bin link. For example, change .../sioc-sys0-ml00/bin/linux-x86/matlabSupport to .../sioc-sys0-ml00/iocSpecificRelease/bin/linux-x86/matlabSupport. Note this listing has some portions alterted to make it fit.
    [softegr@lcls-builder Prod]$ grep -e "^sioc-sys0-ml00" ${IOC_SCREEN}/screeniocs
    sioc-sys0-ml00 <iocCommon>/sioc-sys0-ml00/iocSpecificRelease/bin/linux-x86/...
    [softegr@lcls-builder Prod]$
  6. Reboot your IOC and make sure it comes up correctly with these changes.
    1. For softIOC's, after changing screeniocs, you'll need to log into the machine hosting the softIoc and restart the softIoc using the /etc/init.d scripts. You need to do this only during migration; this is because merely exiting from iocSh in iocConsole does not start a new screen session. You can also accomplish the same using iocConsole your_ioc -cleanup
  7. Make sure your IOC has started up normally after the restart.
  8. Your IOC deployment is now ready for use with cram.

Preparing your IOC software package for cram

Do these steps and commit into CVS before tagging your release.
  1. Checkout your IOC software package in your working area/home folder using eco and cd into it.
    [mshankar@lcls-dev2 test]$ eco
    Enter name of module/package to checkout: MatlabSupport
    Enter name of tag or [RETURN] to use HEAD>
    Using MAIN_TRUNK. The name of the directory will be 'MAIN TRUNK'.
    ...
    [mshankar@lcls-dev2 test]$ cd MatlabSupport/MAIN_TRUNK 
    [mshankar@lcls-dev2 MAIN_TRUNK]$ 
  2. Generate the application description needed using cram describe. This creates a .cram folder with the package information in packageinfo. This allows you to skip the name and type command line parameters when using cram.
    [mshankar@lcls-dev2 MAIN_TRUNK]$ cram describe
  3. If converting to a static build
    1. Change configure/CONFIG for static builds with:
      SHARED_LIBRARIES=NO
      STATIC_BUILD=YES
      
      
    2. If using streamdevice, add USR_SYS_LIBS += pcre to your application's Makefile. This will include libpcre.a from the system libraries.
    3. If using areadetector, add asyn to PROD_LIBS using PROD_LIBS += asyn and remove it from your <APP_libs> in your application's Makefile.
    4. If using the motor module, you may need to pay attention to the order of the includes in the <APP_libs> in your application's Makefile. For example,
      • smarActMCSMotor uses motor
      • motor uses asyn
      These modules then need to be included in dependency order. That is,
      • SelfSeedingMotion_LIBS += smarActMCSMotor
      • SelfSeedingMotion_LIBS += motor
      • SelfSeedingMotion_LIBS += asyn
      Alternativelty, you can include the motor and asyn module's as part of PROD_LIBS and remove it from your <APP_libs> in your application's Makefile.
    5. If using the calc module, you may need to include the sscan module in your configure/RELEASE
      SSCAN_MODULE_VERSION=sscan-R2-8-1_1-0
      SSCAN=$(EPICS_MODULES)/sscan/$(SSCAN_MODULE_VERSION)
      
      
      and in your app's Makefile. Note that the order of libs in the app's Makefile matters for static builds, calc before sscan before EPICS_BASE_IOC_LIBS seems to work.
      softLlrf_LIBS += sscan
      
      
    6. If you are using software from an external vendor and they do not provide static libraries, consider using USR_SYS_LIBS to add the library to the dynamic portion of the linker. For example, you can add a line like so USR_SYS_LIBS += giomm-2.4 in your app's src/Makefile. However, you will still need to make sure the shared library is available on the production system or include the library in your package using LIB_INSTALLS.
    7. On linux, ldd is a very useful tool. If you do an ldd bin/linux-x86/your_app, you should only see .so's from /usr/lib or /lib. If you see .so's from other folders, you need to include the .a's or you need to ship the .so's using LIB_INSTALLS.
  4. If using dynamic builds (where we pick various .so files from the environment), please bear in mind that it is likely that the .so's may not be physically present on the production side(s). Consider bundling these as part of your software package using the LIB_INSTALLS variables of the EPICS build system. Every dependency that needs to exist on the production side(s) is an entity that the system's team needs to maintain. Please do make sure that they are aware of it and that there is a facility agnostic way to determine the location of these assets.
  5. Make the IOC self contained
    • We are building on dev and copying over to production. Some of the variables in envPaths will be fixed by the upgrade script. However, there will not be any modules on the production side of the fence. So, any templates that are being loaded from the module folder need to be included in the application. The EPICS build system already has support for adding a wide variety of file types to your software package and also allows for custom types. In general, adding a file to your application consists of adding the file using something suitable like DB_INSTALLS. For example, to add autosave's save_restoreStatus.db to your application, add the following line to your applications <app>/Db/Makefile
      DB_INSTALLS += $(AUTOSAVE)/db/save_restoreStatus.db
      and change your st.cmd to point to dbLoadRecords("db/save_restoreStatus.db", "P=SIOC_$(AREA)-$(IOC_SUFF):") instead of dbLoadRecords("$(AUTOSAVE)/db/save_restoreStatus.db", "P=SIOC_$(AREA)-$(IOC_SUFF):")
    • If using areaDetector, remember to change areaDetector.cfg if you have one. Some areaDetector.cfg have a cd $(AREA_DETECTOR)/ADApp/Db as the first line. You may want to comment this out as we are not loading templates from the db area.
    • cram has some ability to detect external resources using cram lint. You typically run cram lint in a cycle with make clean and make. An IOC make generates the st.cmd and envPaths for all the ioc's in the software package. cram lint parses the st.cmd for all the ioc's in the software package and points out
      1. All includes from external sources like < $(AREA_DETECTOR)/include.cmd
      2. All cd's to external folders like cd $(AREA_DETECTOR)/ADApp/Db
      3. External EPICS databases loaded using dbLoadRecords; for example, dbLoadRecords("$(IOCADMIN)/db/iocAdminSoft.db", "IOC=SIOC:$(AREA):$(IOC_SUFF)")
      4. Missing EPICS databases loaded using dbLoadRecords; these could be elements that were not added to the app's Makefile using DB_INSTALLS
      To use cram lint, run make clean followed by make and then run cram lint. You should see some thing like this
      
      [mshankar@lcls-dev2 ProfileMonitorAD-R1-0-2]$ cram lint
      Linting IOC iocBoot/sioc-b34-pm02
      Linting IOC iocBoot/sioc-li20-pm20
      iocBoot/sioc-b34-pm02/st.cmd:125 Missing file db/iocAdminSoft.db in ${TOP}; perhaps you...
      iocBoot/sioc-b34-pm02/st.cmd:147 Add DB_INSTALLS += $(AREA_DETECTOR)/ADApp/Db/ADBase.te...
      
      
      Fix these errors as appropriate and then run make clean followed by make followed by cram lint.
  6. Make sure we pick msi from EPICS base and not anywhere else. That is, change references to msi with IOC Makefiles and point it to ${EPICS_BASE_RELEASE}/bin/${EPICS_HOST_ARCH}/msi.
  7. Run make and confirm that you are able to build your IOC app with these changes.
  8. Commit the changes and tag this as your next release.
    [mshankar@lcls-dev2 MAIN_TRUNK]$ cvs add .cram
    [mshankar@lcls-dev2 MAIN_TRUNK]$ cvs add .cram/packageinfo
    [mshankar@lcls-dev2 MAIN_TRUNK]$ cvs commit
  9. You are now ready to use cram to deploy your IOC package.

Deploying your IOC software package with cram

  1. In dev, check out a properly tagged release of your software package in your working area/home using eco and cd into it.
    [mshankar@lcls-dev2 test]$ eco
    Enter name of module/package to checkout: MatlabSupport
    Enter name of tag or [RETURN] to use HEAD>MatlabSupport-R3-2-0-0
    Using MatlabSupport-R3-2-0-0. The name of the directory will be ...
    ...
    [mshankar@lcls-dev2 test]$ cd MatlabSupport/MatlabSupport-R3-2-0-0/
    [mshankar@lcls-dev2 MatlabSupport-R3-2-0-0]$
    
    1. At least during the initial migration, avoid using MAIN_TRUNK.
  2. If your current IOC deployment (with the current and the iocSpecificRelease softlinks) has been set up correctly, you should be able to run cram ls from within your tagged release folder. This should get a list of the current releases for your software package.
    [mshankar@lcls-dev2 MatlabSupport-R3-2-0-0]$ cram ls
    Current versions on facility: LCLS 
    Current master release => MatlabSupport-R3-2-0-0
    IOC: sioc-sys0-ml00 => MatlabSupport-R3-2-0-0
    ...
    
    
  3. Run make. If your IOC software package has been currently set up, the make should complete successfully.
  4. Submit/push the release for this software package using cram push to all facilities.
    [mshankar@lcls-dev2 MatlabSupport-R3-2-0-0]$ cram push
    Pushed release MatlabSupport-R3-2-0-0 to LCLS
    ...
    
    
    If you are doing this for the first time, make sure that the new version of the software package made it to the iocTop across all facilities.
  5. Upgrade one or more IOC's using cram upgrade. You will need to specify the facility and the IOC and the release.
    
    ...MatlabSupport-R3-2-0-0]$ cram upgrade -f TestFac -i sioc-sys6-ml00 MatlabSupport-R3-2-0-0
    ...MatlabSupport-R3-2-0-0] 
    
    
  6. Confirm that the upgrade happened correctly by using cram ls again.
  7. Restart the IOC and make sure that the IOC restarted correctly.
  8. You can upgrade more than one IOC in a facility by using a comma separated list of IOC names to the -i argument. For example, cram upgrade -f TestFac -i sioc-sys6-ml00,sioc-sys6-ml01 MatlabSupport-R3-2-0-0
  9. You can upgrade all the IOC's in a facility by using -i ALL. For example, cram upgrade -f TestFac -i ALL MatlabSupport-R3-2-0-0
  10. You can upgrade all the IOC's in all facilities by using -f ALL. For example, cram upgrade -f ALL -i ALL MatlabSupport-R3-2-0-0

Preparing your current HLA deployment for cram

cram uses the same approach used by HLA applications. Therefore, the preparation for HLA applications is straightforward.
  1. Choose a name for your software package. We should use the same name in all the build and deployment tools
    1. This should be the module name in CVS.
    2. This should be the name used with eco.
    3. This should be the name of the softlink in ${PHYSICS_TOP} that points to the release within the release folder.
      [mshankar@lcls-dev2 ~]$ cd $PHYSICS_TOP 
      [mshankar@lcls-dev2 physics]$ ls -ltrd archiveviewer 
      lrwxr-xr-x 1 mshankar root 43 Nov  4 10:41 archiveviewer -> release/ArchiveViewer-R1-2-11/
      
  2. In ${PHYSICS_TOP}/release of each facility, create a packageName folder (if it does not exist) and move the existing releases for the software package to the packageName folder.
    [mshankar@lcls-dev2 ~]$ cd ${PHYSICS_TOP}/release
    [mshankar@lcls-dev2 release]$ mkdir archiveviewer
    [mshankar@lcls-dev2 release]$ mv ArchiveViewer* archiveviewer
    
    
  3. Change the application soft link in ${PHYSICS_TOP} to point to the current production release within the new release/packageName folder. Note this link will be broken because of the mv in the previous step.
    [mshankar@lcls-dev2 ~]$ cd $PHYSICS_TOP
    [mshankar@lcls-dev2 physics]$ rm archiveviewer 
    rm: remove symbolic link `archiveviewer'? y   
    [mshankar@lcls-dev2 physics]$ ln -s release/archiveviewer/ArchiveViewer-R1-2-11 archiveviewer
    [mshankar@lcls-dev2 physics]$ ls -ltra
  4. Confirm that the HLA application is working correctly.
  5. Your HLA deployment is now ready for use with cram.

Preparing your current HLA software package for cram

cram uses the same approach used by HLA applications. As long as you can build your application using ant, you should be good to go. However, there are a couple of steps. Do these steps and commit into CVS before tagging your release.
  1. Checkout your HLA software package in your working area/home folder using eco and cd into it.
    [mshankar@lcls-dev2 test]$ eco
    Enter name of module/package to checkout: archiveviewer
    Enter name of tag or [RETURN] to use HEAD>
    Using MAIN_TRUNK. The name of the directory will be 'MAIN TRUNK'.
    ...
    [mshankar@lcls-dev2 test]$ cd archiveviewer/MAIN_TRUNK/
    [mshankar@lcls-dev2 MAIN_TRUNK]$ 
  2. Generate the application description needed using cram describe. This creates a .cram folder with the package information in packageinfo. This allows you to skip the name and type command line parameters when using cram.
    [mshankar@lcls-dev2 MAIN_TRUNK]$ cram describe
  3. Run ant and make sure the application builds correctly.
  4. Commit the changes and tag this as your next release.
    [mshankar@lcls-dev2 MAIN_TRUNK]$ cvs add .cram
    [mshankar@lcls-dev2 MAIN_TRUNK]$ cvs add .cram/packageinfo
    [mshankar@lcls-dev2 MAIN_TRUNK]$ cvs commit
  5. Make the HLA app as self contained as possible. Remove references to jars that are outside the package and include those jars within the package. This is not strictly necessary but will minimize migration headaches.

Deploying your HLA software package with cram

  1. In dev, check out a properly tagged release of your software package in your working area/home using eco and cd into it.
    [mshankar@lcls-dev2 test]$ eco
    Enter name of module/package to checkout: archiveviewer
    Enter name of tag or [RETURN] to use HEAD>ArchiveViewer-R1-2-11
    Using ArchiveViewer-R1-2-11. The name of the directory will be ...
    ...
    [mshankar@lcls-dev2 test]$ cd archiveviewer/ArchiveViewer-R1-2-11/
    [mshankar@lcls-dev2 ArchiveViewer-R1-2-11]$
    
    1. At least during the initial migration, avoid using MAIN_TRUNK.
  2. If your current HLA deployment has been set up correctly, you should be able to run cram ls from within your tagged release folder. This should get a list of the current releases for your software package.
    [mshankar@lcls-dev2 ArchiveViewer-R1-2-11]$ cram ls
    Current version on facility: LCLS  => ArchiveViewer-R1-2-11
    Current version on facility: FACET  => ArchiveViewer-R1-2-11
    Current version on facility: TestFac  => ArchiveViewer-R1-2-11
    Current version on facility: Dev  => ArchiveViewer-R1-2-11
    [mshankar@lcls-dev2 ArchiveViewer-R1-2-11]$
    
    
  3. Run ant. If your HLA software package has been currently set up, the ant should complete successfully.
  4. Submit/push the release for this software package using cram push to all facilities.
    [mshankar@lcls-dev2 ArchiveViewer-R1-2-11]$ cram push
    Pushed release ArchiveViewer-R1-2-11 to LCLS
    ...
    
    
    If you are doing this for the first time, make sure that the new version of the software package made it to the ${PHYSICS_TOP}/release across all facilities.
  5. Upgrade one or more facilities using cram upgrade. You will need to specify the facility and the IOC and the release.
    
    ...ArchiveViewer-R1-2-11]$ cram upgrade -f LCLS ArchiveViewer-R1-2-11
    ...ArchiveViewer-R1-2-11] 
    
    
  6. Confirm that the upgrade happened correctly by using cram ls again.
  7. Confirm that the HLA application is working correctly.
  8. You can upgrade more than one facility by using a comma separated list of facilities to the -f argument. For example, cram upgrade -f TestFac,LCLS ArchiveViewer-R1-2-11
  9. You can upgrade all facilities by using -f ALL. For example, cram upgrade -f ALL ArchiveViewer-R1-2-11