Python extension module tutorial

The hippo Python extension module is designed to be used interactively or via Python scripts.

Thus the interface is somewhat different from the C++ interface to the HippoDraw library.

Using HippoDraw interactively can be as simple as two lines of Python code. Below is an example session.

> python
Python 2.4 (#2, Apr 15 2005, 17:09:59)
[GCC 3.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import hippo
>>> app = hippo.HDApp()
>>>

Obviously, even typing these two lines for every session can get boring. Instead one can put the commands in a file and use that as initialization step for the Python session. For example, the file, canvas.py, in the testsuite directory contains

import hippo
app = hippo.HDApp()
canvas = app.canvas()

where we also show how to get a handle on the current canvas window. One can run this script from a UNIX shell or Windows command prompt like this

> python -i canvas.py
>>>

This launches the complete HippoDraw application in a separate thread with the ability to interact with it from the Python shell.

Getting help and documentation

Python has interactive help system. The get all help for the hippo module, just do the following...

> python
Python 2.4 (#2, Apr 15 2005, 17:09:59)
[GCC 3.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import hippo
>>> help ( hippo )

This gives you all the built-in documentation in a pager like the UNIX more or less command which is not always convienent. However, one can get the documentation on one class by ...

> python
Python 2.4 (#2, Apr 15 2005, 17:09:59)
[GCC 3.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import hippo
>>> help ( hippo.HDApp )

Or even one member function like this ...

> python
Python 2.4 (#2, Apr 15 2005, 17:09:59)
[GCC 3.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import hippo
>>> help ( hippo.HDApp.canvas )

Another way to access the same information is to use the pydoc program that came with your Python installation (except under Windows).

> pydoc hippo

This also gives you all the built-in help in a pager. Probably the most convienent method is to generate html version of the documentation. You do this by typing ...

> pydoc -w hippo

and a hippo.html file is created in your working directory. Here is what it looks like. Not very pretty, but it is the standard ouput from pydoc. You can use this link as your documentation for HippoDraw. However, it will get updated for each release and you may be using an older version.

The following sections shows and explains some example script. See also Examples of HippoDraw use with Python for more examples

Creating the NTuple in Python

One might generate some data in Python that you want to display with HippoDraw. For example, you could generate a Python list with random Gaussian distribution like this

>>> import random
>>> x = []
>>> for i in range ( 10000 ) :
...     x.append ( random.gauss ( 45, 10 ) )
...
>>>

To display the data as a histogram, one can then type

>>> from hippo import Display
>>> hist = Display ( 'Histogram', ( x, ), ('Gaussian', ) )
>>> canvas.addDisplay ( hist )
>>>

The first argument to the Display function specifies the type of display to create. The second is a Python tuple of Python list objects that will be used by the display. The third argument is a Python tuple of string labels for the lists.

You can now modify the plot, for example, changing the width of the bins in two ways. From the Python shell, one can invoke a member function of the histogram object like this...

>>> hist.setBinWidth ( 'x', 2 )
>>>

But it is much easier to use Axis inspector and change it with the slider or text field.

The function created a DataSource called a ListTuple. It holds references to the Python list objects as columns. The list is not copied, just referenced. It also holds the labels of each column. Displays don't redraw themselves unless they know there's been a change, like changing the bin width. But should the contents of your Python list change, the Display wouldn't know about it. But you can force the display to redraw like this...

>>> hist.update()
>>>

The Python tuple of strings provide the data source labels, but they also giving the bindings of the displays to the data source. Some displays have binding that are optional. For the example, an "XY Plot" display had binding for the X and Y axis, and optionally, for an error on X or Y. To say which optional bindings not to use the "nil" column label is used. The we can do the following

"""
   Demonstrates making simple XY plot.  

   author Paul F. Kunz <Paul_Kunz@slac.stanford.edu>
"""
#
# load the HippoDraw module
#
from load_hippo import app, canvas

from hippo import Display

# Create list of data
energy = [90.74, 91.06, 91.43, 91.50, 92.16, 92.22, 92.96, 89.24, 89.98, 90.35]
sigma  = [ 29.0, 30.0,  28.40, 28.80, 21.95, 22.90, 13.50,  4.50, 10.80, 24.20]
errors = [  5.9,  3.15,  3.0,   5.8,  7.9,   3.1,   4.6,    3.5,   4.6,   3.6]

# make a plot to test it.
xy = Display ( "XY Plot", [ energy, sigma, errors ],
               ['Energy', 'Sigma', 'nil', 'error' ] )

canvas.addDisplay ( xy )

xy.setTitle ( 'Mark II Z0 scan' )

print "An XY plot is now displayed.   You can use the Inspector dialog"
print "to modify the appearance or fit a function to it."

The "nil" string can also be use by the Data inspector as well. Note in this example, we used list of lists instead of tuple of lists. Either can be used.

Speaking of the Data Inspector, sometimes it is more convenient to give HippoDraw all the data you might want to use for displays, and use the Data Inspector to create them. To do this, one creates a DataSource manually. There are three kinds supported: ListTuple, NTuple, and NumArrayTuple. They share a common interface and differ on how they store the column data. As we've seen, the ListTuple stores references to Python list objects. The NTuple makes copies of Python list objects and stores it internally as a C++ vector of doubles. The NumArrayTuple stores references to rank 1 numarray objects. The NTuple has the feature that you can add and replace rows or columns.

Creating displays with the DataInspector doesn't preclude one from also creating them with Python. The interface is similar to what we've already seen. For example

>>> energy = [90.74, 91.06, 91.43, 91.5, 92.16, 92.22, 92.96, 89.24, 89.98 ]
>>> sigma  = [ 29.0, 30.0,  28.40, 28.8, 21.95, 22.9,  13.5,   4.5,  10.8 ]
>>> errors = [  5.9,  3.15,  3.0,   5.8,  7.9,   3.1,   4.6,   3.5,   4.6,]
>>> ntuple = NTuple () # an empty NTuple
>>> ntc = NTupleController.instance ()
>>> ntc.registerNTuple ( ntuple )
>>> ntuple.addColumn ( 'Energy', energy )
>>> ntuple.addColumn ( 'Sigma', sigma )
>>> ntuple.addColumn ( 'error', errors )

>>> xy = Display ( "XY Plot", ntuple,  ('Energy', 'Sigma', 'nil', 'error' ) )
>>> canvas.addDisplay ( xy )
>>>

Registering the ntuple with the NTupleController is necessary in order for the Data Inspector to know of their existence.

Getting data from a file

An NTuple data source can also be created by reading a plain text file. See ASCII file for the details. The example file, histogram.py, in the testsuite directory shows how to read a file and create displays from Python. It contains ...

""" -*- mode: python -*-

This script tests the creation and modification of Histogram along
with some test of exception handling.

Copyright (C) 2001, 2003   The Board of Trustees of The Leland Stanford
Junior University.  All Rights Reserved.

@author Paul F. Kunz <Paul_Kunz@slac.stanford.edu>

$Id: python_interact.html,v 1.40 2008/03/24 19:27:26 pfkeb Exp $

"""
import sys
from hippo import HDApp

app = HDApp()
canvas = app.canvas()

from hippo import NTupleController
ntc = NTupleController.instance()

nt1 = ntc.createNTuple ( '../../hippodraw/examples/aptuple.tnt' )

from hippo import Display

hist = Display ("Histogram", nt1, ('Cost',) )
canvas.addDisplay( hist )

hist.setRange ( 'x', 0., 30000. )

hist2 = Display ( "Histogram", nt1, ('Service',) )
canvas.addDisplay ( hist2 )

hist2.setBinWidth ( 'x', 1.0 )
hist2.setOffset ( 'x', 0.5 )

After reading a HippoDraw compatible data source file, this Python script creates two displays. It sets the range on the first and the bin width on the second. The results of running this script are shown below.

Result of using histogram.py

The Display class is actually a small wrapper around the internal HippoDraw C++ library class. It is needed because Qt is running in a separate thread from Python. Since creating a display and perhaps modifying it requires interaction with Qt's event loop, the application must be locked before calling a member function of the actual HippoDraw class and then unlocked when returning.

Using the hippoplotter interface

Making use of Python's default parameter value feature in calling functions, Jim Chiang has extended the HippoDraw interface with his The hippoplotter.py module.

The file, pl_exp_test.py, in the testsuite directory shows an example of using this module.

""" -*- mode: python -*-

   Testing the PowerLaw and Exponential classes and
   exercising the hippoplotter.py wrapper.

   author: James Chiang <jchiang@slac.stanford.edu>
   
"""
#
# $Id: python_interact.html,v 1.40 2008/03/24 19:27:26 pfkeb Exp $
#
# Author: J. Chiang <jchiang@slac.stanford.edu>
#

from setPath import *

import random, math

import hippoplotter as plot

#
# Generate power-law and exponential distributions
#
nt1 = plot.newNTuple( ([], ), ("x", ) )
pl_display = plot.Histogram(nt1, "x", xlog=1, ylog=1,
                               title="power-law dist.")

nt2 = plot.newNTuple( ([], ), ("x", ) )
exp_display = plot.Histogram(nt2, "x", ylog=1, title="exponential dist.")

both = plot.newNTuple( ([], ), ("x",) )
combo_display =plot.Histogram(both, "x", ylog=1,
                                 title="power-law & exponential dists.")

shoot = random.uniform

#
# The parameters describing the distributions
#
x0 = 5.            # Characteristic scale for exponential

xmin = 1.          # Bounds for the power-law distribution
xmax = 100.
gamma = 2.1        # The power-law index

xpmax = math.pow(xmax, 1.-gamma)
xpmin = math.pow(xmin, 1.-gamma)

nsamp = 10000

print "Filling NTuple with data."

for i in range(nsamp):
    xi = shoot(0, 1)
    xx1 = math.pow( xi*(xpmax - xpmin) + xpmin, 1./(1.-gamma) )
    nt1.addRow( (xx1, ) )
    both.addRow( (xx1,) )
    
    xi = shoot(0, 1)
    xx2 = -x0*math.log(1. - xi)
    nt2.addRow( (xx2, ) )
    both.addRow( (xx2, ) )

#
# Fit these distributions
#
Function = plot.hippo.Function
powerlaw = Function( "PowerLaw", pl_display.getDataRep() )
powerlaw.addTo( pl_display )
powerlaw.fit()

exponential = Function( "Exponential", exp_display.getDataRep() )
exponential.addTo( exp_display )
exponential.fit()

#
# Do a fit to sum of functions.
#
pl = Function ( "PowerLaw", combo_display.getDataRep() )
pl.addTo( combo_display )

exp2 = Function ( "Exponential", combo_display.getDataRep() )
exp2.addTo( combo_display )

# Fit to linear sum
exp2.fit()

print "Demonstrated power law, exponential, and linear sum fitting"
print ""

The above script leads to the canvas shown below

Results of pl_exp_test.py script

Extracting data from a display

The interaction with HippoDraw from Python is not just one direction. Once can extract data from the displays and use them in Python. The file function_ntuple.py illustrates this...

""" -*- python -*-

   This script adding functions and fitting.  It also demonstrates
   retreiving an ntuple from the histogram to do something with its
   contents.

Author: Paul_Kunz@slac.stanford.edu

$Id: python_interact.html,v 1.40 2008/03/24 19:27:26 pfkeb Exp $nt

"""
import sys


from load_hippo import app, canvas

#
# Create NTuple with its controller so Inspector can see it.
#
from hippo import NTupleController
ntc = NTupleController.instance()
#
# Create full path to example file in case this script is not run from
# this directory.
#
full_path = sys.path[0] + '/' + 'aptuple.tnt'

nt1 = ntc.createNTuple ( full_path )

canvas.setPlotMatrix ( 2, 3 )
from hippo import Display

hist = Display ( "Histogram", nt1, ("Cost", ) )
canvas.addDisplay( hist )

# Get the data representation so we can add function to it.
datarep1 = hist.getDataRep()
from hippo import Function
gauss = Function ( "Gaussian", datarep1 )
gauss.addTo ( hist )

# Get the function parameters and display them.
print "Before fitting"
parmnames = gauss.parmNames ( )
print parmnames

parms = gauss.parameters ( )
print parms

# Now do the fitting.
gauss.fit ( )

print "After fitting"
parms = gauss.parameters ( )
print parms

# Add another function.
gauss1 = Function ( "Gaussian", datarep1 )
gauss1.addTo ( hist )

# Do another fit, should fit to linear sum
gauss1.fit ()


# Add Chi-squared per d.f. display
canvas.addTextRep ( hist, 'Chi-squared' )

# Create an NTuple from the histogram.
# Calculate the residuals

result = hist.createNTuple ()
ntc.registerNTuple ( result )

coords = result.getColumn ( 'Cost' )
values = result.getColumn ( 'Density' )
res = []
for i in range ( result.rows ) :
    x = coords[i]
    diff = values[i] - gauss1.valueAt ( x )
    res.append ( diff )

# Add a column and display it.
result.addColumn ( 'residuals', res )
resplot=Display ( "XY Plot", result, ( 'Cost', 'residuals', 'nil', 'Error' ) )
canvas.addDisplay ( resplot )

print "The histogram was fitted to the sum of two gaussians."
print "Then histogram bins were retrieved to calculate "
print "the residuals.  These were then plotted as an XY Plot."
print 'One could have used the "Create residuals display" button on the'
print "Inspector, but that wouldn't have demonstrated anything."

Like the previous script, it fits two functions to a histogram. It also shows how to extract the function parameter names and their values. Near the end of the script, one extracts the contents of the histogram bins in the form of an NTuple. In the for loop at the end, one uses the NTuple to calculate the residuals between the function and the bin contents and put them in a Python list. The the list is added as a column to the NTuple. Finally, one creates an XYPlot to display them and adds it to the canvas. The result looks like this...

Results of function_ntuple.py

However, one didn't have to write this script to plot the residuals, as the is a control in the Function inspector that does it for you.

Using a FITS file

A FITS file can be used as input to HippoDraw. Here's how one can it to view an image of the EGRET All-Sky survey from such a file

""" -*- mode: python -*-

   Displaying the EGRET All Sky survey

   author: James Chiang <jchiang@slac.stanford.edu>
   
"""
#
# $Id: python_interact.html,v 1.40 2008/03/24 19:27:26 pfkeb Exp $
#

import sys
sys.path.reverse()
sys.path.append('../../hippodraw/python')
sys.path.append('..//python')
sys.path.reverse()

from setPath import *

import hippoplotter as plot

file = "../../hippodraw/examples/EGRET_AllSky.fits"
plot.fitsImage(file, zlog=1, aspect=2)

The resulting canvas is shown below

The EGRET All-Sky survey.

The FITS data format is a standard astronomical data and mandated by NASA for some projects. It supports images as well as binary or ASCII tables. A FITS table is essentially a NTuple with added information in the form of keyword-value pairs. James Chiang also wrote the following Python function to convert a FITS table to a HippoDraw NTuple.

"""
Read in a series of FITS table files and make them accessible as
numarrays, optionally creating a HippoDraw NTuple.

@author J. Chiang <jchiang@slac.stanford.edu>
"""
#
# $Id: python_interact.html,v 1.40 2008/03/24 19:27:26 pfkeb Exp $
#
import sys, pyfits
try:
    import numarray as num
except ImportError:
    import numpy as num

class FitsNTuple:
    def __init__(self, fitsfiles, extension=1):
        cat = num.concatenate
        #
        # If fitsfile is not a list or tuple of file names, assume
        # it's a single file name and put it into a single element
        # tuple.
        #
        if type(fitsfiles) != type([]) and type(fitsfiles) != type(()):
            fitsfiles = (fitsfiles, )
        #
        # Process each file named in the list or tuple.
        #
        columnData = {}
        for i, file in zip(xrange(sys.maxint), fitsfiles):
            #print "adding", file
            table = pyfits.open(file.strip(" "))
            if i == 0: 
                self.names = table[extension].columns.names
            for name in self.names:
                if i == 0:
                    columnData[name] = table[extension].data.field(name)
                else:
                    columnData[name] = cat((columnData[name],
                                            table[extension].data.field(name)))
        #
        # Add these columns to the internal dictionary.
        #
        self.__dict__.update(columnData)
        
    def makeNTuple(self, name=None, useNumArray=1):
        import hippo
        if useNumArray:
            nt = hippo.NumArrayTuple()
        else:
            nt = hippo.NTuple()
        if name != None:
            nt.setTitle(name)
        ntc = hippo.NTupleController.instance()
        ntc.registerNTuple(nt)
        for name in self.names:
            if len(self.__dict__[name].shape) > 1: # have multicolumn variable
                columns = self.__dict__[name]
                columns.transpose()
                for i, col in enumerate(columns):
                    colname = "%s%i" % (name, i)
                    nt.addColumn(colname, col)
            else:
                try:
                    nt.addColumn(name, self.__dict__[name])
                except TypeError:
                    pass
        return nt

Using ROOT files

Another example is reading a ROOT file that has the form of an ntuple as define in RootNTuple class. The Python code might look like this...

""" -*- mode:python -*-

Demo of reading ROOT file with function, cuts, and calculation

author: Paul F. Kunz <Paul_Kunz@slac.stanford.edu>

"""

#import setPath
from load_hippo import  app, canvas

from hippo import RootController, Display
rc = RootController.instance()

# filename = "/nfs/farm/g/glast/u33/InstrumentAnalysis/MC/EngineeringModel-v6r070329p28/Surface_muons/surface_muons_4M_merit.root"

filename =  "/nfs/farm/g/glast/u41/Integration/rootData/077014364/v7r0913p11/svacTuple/v3r0p3/svacTuple-v3r9p3_077014364_svac_svac.root"
ntuple_names = rc.getNTupleNames ( filename  )
print "In this file, tree names are ", ntuple_names

ntuple = rc.createDataArray ( filename, 'Output' )
print "Number of columns = ", ntuple.columns

labels = ntuple.getLabels()
print "column labels are ... ", labels

print "Number of rows = ", ntuple.rows

xi = ntuple['AcdTkrIntSecGlobalX']
print xi.shape

This script not only uses ROOT, but it also uses numarray. It converts a ROOT brach into a numarray array so it can do vector calculations. The ROOT C++ macro to do the equivalent of the above Python script would be considerable more complex.

Limitations.

With this release, not all of HippoDraw's C++ library is exposed to Python. Although this could be done, it is thought to be not necessary. Rather, selected higher level components are exposed in one of two ways. Some classes are exposed directly with a one to one relationship between the C++ member functions and Python member functions. An example is the NTuple class.

One can view the reference documentation for the hippo extension module with Python's online help command, One can also use the pydoc program to view it or generated HTML file with the command "pydoc -w hippo".

In order to be able to have an interactive Python session that interacts with the HippoDraw canvas items and at the same time have interaction with the same items from the Inspector, it was necessary to run the HippoDraw application object in a separate thread. Threading conflicts could then occur. Thus some of HippoDraw's C++ classes are exposed to Python via a thin wrapper class which locks the Qt application object before invoking an action and unlocks it when done.

One good thing about Python is that what ever you do, Python never crashes. Thus, what ever you do with the HippoDraw extension module should not crash Python. An interactive user, however, can easily mis-type an argument to a function. For example, he could try to create a display with "ContourPlot" instead of "Contour Plot". For such errors, the C++ library throws a C++ exception. The HippoDraw extension module catches them and translates them to a Python exception. Thus, when the Python user makes an error, he will receive a message instead of crashing his session.

Another reason the wrapper classes exist is to try to present a more Python "interactive friendly" interface to the user than the raw C++ interface which was designed for the application writer. With this release, it is not clear what a more "friendly" interface should look like. Maybe the Python extension module should be closer to the C++ interface and provide Python classes to wrap them in a more friendly way like James Chiang has done. Feed back on this topic would be very welcome.

---