Python Management


This document describes how to install and manage Python and required modules for accelerator controls.  Let's take Python-2.7.4 for example.

Main steps –


Install Python-2.7.4

on one of our machines that has an access to Internet (e.g., lcls-dev3)
cd /scratch/jingchen/python; mkdir python2.7.4

cd python2.7.4; mkdir build;mkdir src
(Building in separate build directories brings about many advantages over building in-source. For example, by separating src from build,  one can have multiple builds, each with different configuration)

cd src
tar -xjf Python-2.7.4.tar.bz2
look README in it.
cd ../build

../src/Python-2.7.4/configure --prefix=/scratch/jingchen/python/python2.7.4
make install
export PATH=/scratch/jingchen/python/current/bin:$PATH

After the installation, the basic directory structure should look like the following;

bin build include lib share src

Install setuptools and pip within a Python package (e.g., Python-2.7.4)

setuptools and pip help with module installation. This is prerequisite.

cd /scratch/jingchen/python/python2.7.4/src/Python-2.7.4-files/download
scp jingchen@mcclogin:/u/cd/divekar/setuptools-0.6c11-py2.7.egg .
sh setuptools-0.6c11-py2.7.egg
scp jingchen@mcclogin:/u/cd/divekar/pip-1.3.1.tar.gz .
tar xvfz pip-1.3.1.tar.gz
cd pip-1.3.1
python install


        Install Modules

        Modules come from three different sources and we create directories for those three types –

        1) From Python official site PyPi

        2) Other external packages

        3) Site Specific packages. These are mostly SLAC specific packages developed here.


First prepare the list of required modules from the official site. Please describe how. Here is my guess.

pip list | cut -f1 -d' ' > package-list




Install Python and required modules on Dev and Prod

Create required directory structure for installation on Dev and Prod.  


For example on Prod these are repository directories for Python2.7.4–


For official PyPi/Python packages 



For external packages



For SLAC packages



Copy modules repository created in above steps on Dev and Prod.

Install Python and required modules for Dev and Prod environment.


Include Site-specific packages

Create a file lcls-tools.pth under /usr/local/lcls/package/python/current/lib/python2.7/site-packages director and add the directory paths which contain slac packages and external packages.

[divekar@lcls-builder site-packages]$ cat /usr/local/lcls/package/python/current/lib/python2.7/site-packages/lcls-tools.pth

Check the included packages with sys.path in Python

Procedure for installing additional modules

For installing additional PyPi (Official Python Modules) do following -

Create directory in python2.7.4/src/Python-2.7.4-files/download-ongoing

Download the modules from PyPi website and copy those under download-ongoing directory.

Create a temporary /tmp/list file containing module names.

Install modules using following script -

cat /tmp/list | while read a; do pip install --find-links file:///afs/slac/g/lcls/package/python/current/src/Python-2.7.4-files/download/download-ongoing --index-url=file:///dev/null $a; done
Check that all modules are installed by issuing "pip list" command.

Tar and copy the modues to download dir in DEV and PROD, install the modules.

After installation is complete remove all contents from download-ongoing directory on lcls-dev3

In most cases, we install one module at a time. Take one module, pyqtgraph, for example:

1) download it in /scratch area on lcls-dev3

pip install --download-cache="/scratch/jingchen/python/python2.7.4/src/Python-2.7.4-files/cache" --download="/scratch/jingchen/python/python2.7.4/src/Python-2.7.4-files/download-ongoing" pyqtgraph

2) install it on DEV on lcls-dev2

pip install --find-links file:///afs/slac/g/lcls/package/python/current/src/Python-2.7.4-files/download-ongoing --index-url=file:///dev/null pyqtgraph

3) install it on PROD on lcls-builder

pip install --find-links file:///usr/local/lcls/package/python/current/src/Python-2.7.4-files/download-ongoing --index-url=file:///dev/null pyqtgraph




          1   Normally pip also downloads the dependencies. But that’s not the case for all the modules. For example “chaco” module requires “numpy” module.

               But pip won’t download “numpy” automatically. For such exceptions you need to install and download the dependencies before.

          2  “Pip install <module>” command downloads the current stable module from PyPi website. Some of the modules won’t get installed because they have dependencies of older version. 

               For example Crypto and rsa need distribute-0.6.10. While the stable version on PyPi website is distribute-0.7.3.

               For installing Crypto and rsa modules, you need to uninstall default distribute module and install distribute-0.6.10.

          3  distribute-0.6.10 wipes out (or corrupts setuptools package)

              To fix the error manually  install distribute –

              $ cd distribute-0.6.10

              $ python install

         4    On Dev, we get error while importing _sqlite3.

               This is workaround solution – Add following line in /afs/slac/g/lcls/package/python/current/lib/python2.7/site-packages/lcls-tools.pth file


         5 ideally, all modules which are neither part of Python distribution nor SLAC developed, should be installed in external-packages

           ( whether via pip or other means), to make the modules soft binded with a Python version. To do!



Created by Shashi Divekar.

Modified by Jingchen Zhou, on 08/21/13 .