Explore LAT Data

This tutorial demonstrates various ways of examining and manipulating the LAT data, but it is a simple approach useful for quickly exploring the data.

You will be using the following FITS File Viewers:

• ds9 (image viewer); download and install from Harvard's: SAOImage DS9: Astronomical Data Visualization Application page.

• fv (view images and tables; can also make plots and histograms. Download and install from NASA's HEASARC: Software --> Fv: The Interactive FITS File Editor page.

Assumptions

It is assumed that: you are working either in a Unix style terminal window, or in a Windows Command shell window; and that the following are installed on your system and are in your path:

• LAT ScienceTools
  
• fv
• ds9

Note: You may wish to perform these exercises on SLAC Central Linux. [See rednavbar --> SAS Software --> Running on SLAC Central Linux (SCL).]

See: Example: Setup (when running Science Tools on SLAC Central Linux).

Tip: If you are running on SLAC Central Linux and experience problems with either ds9 or fv when using the astrotools environment variable (/afs/slac.stanford.edu/g/glast/applications/astroTools) try changing to a bash shell and sourcing kipac's profile.xray script (source /afs/slac/g/ki/software/etc/profile.xray).

Steps:

1. Obtain the Data
2. Exploring the Data
   
• ds9 QuickLook
  
• Exploring with fv
• Viewing an Extension Header
• Making a Counts Maps
• Making 1D Histograms
• Looking at the Raw Data
  
3. Binning the Data
• Images
• Raw Count Lightcurves
• Energy Spectra
4. Looking at the Exposure
• Calculate the Livetime
• Generate an Exposure Map
• Look at the Exposure Maps

Click on the links below to download the files used in this tutorial

• 3C279_region_filtered_gti_fits (9 MB) - a 20 degree region around the blazar 3C279 with appropriate selection cuts applied.

• spacecraft.fits (62 MB) - the spacecraft data file for 3C279.

• grb_events.fits (130 kB) - a 40 degree region around GRB080916C (trigger time = 243216766 MET).

• grb_spacecraft.fits (52 kB) - the spacecraft data file for GRB 080916C.

Note: The grb_events.fits and grb_spacecraft.fits files were selected from the FSSC's LAT data server using the following criteria:

• Search Center (RA,Dec) = (121.8,-61.3)
• Radius = 40 degrees
• Start Time (MET) = 243216266 seconds (2008-09-16T00:04:26)
• Stop Time (MET) = 243218266 seconds (2008-09-16T00:37:46)
• Minimum Energy = 20 MeV
• Maximum Energy = 300000 MeV

This generates a data file from 500s before to 1500 seconds after the trigger time of the burst. There is no need to select event classes for bursts, as you will need all three event classes to gain statistics.

In this section we look at several different ways to explore the data and make quick and dirty plots and graphs of the data.  First, we'll look at making quick counts maps with ds9; then we'll perform a more in depth exploration of the data using fv.

ds9 Quick look

To see the data, use ds9 to create a quick counts maps of the events in the file. 

For example, to look at the 3C279 data file, enter the following command:

A ds9 window will open up and an image similar to the one shown below will be displayed. 

Breaking the command line into its parts, we find that: ds9 - Invokes the command. -bin factor 0.1 0.1 - Tells ds9 that the x and y bin sizes are to be 0.1 units in each direction.  Since we will be binning on the coordinates (RA, DEC), this means we will have 0.1 degree bins. Note: The default factor is 1, so if you leave this off the command line you will get 1 degree bins

-cmap b - Tells ds9 to use the "b" color map to display the image.  This is completely optional and the choice of map "b" represents the personal preference of the author.  If left off, the default color map is "gray" (a grayscale color map). -scale sqrt - Tells ds9 to scale the colormap using the square root of the counts in the pixels.  This particular scale helps to accentuate faint maxima, where there is a bright source in the field as is the case here.  Again this is the author's personal preference for this option.  If left off, the default scale is linear. 3C279_region_filtered_gti.fits[bin=RA,DEC] - The first part is just the filename containing the events you want to bin up.  The part in brackets [ ] tells ds9 the names of the columns to use for the x and y coordinates of the data to bin.  In this case we've chosen to make a map in Right Ascention and Declination, but you could use galactic coordinates ([bin=L,B]) or any other pair of columns as well. Note:  If you do this a lot and do not want to have to type "[bin=RA,DEC]" at the end of each file name, it is possible to make this default action for ds9.  To do this, set the environment variable DS9_BINKEY equal to [bin=RA,DEC].  If this environment variable is set, you only need to type the file name as part of the ds9 command line. Tip: Normally, ds9 is an interactive utility with a GUI. However, it can be run non-interactively from a script, and even on a batch machine. The following examples use ds9 v5.2. To run non-interactively, build up a command line with options to do what you wish. For example, to create a counts map image: $ ds9 -file cmap.fits -zoom to fit -cmap b -grid skyformat degrees -grid yes -regions EMS-names.reg -saveimage png mytest.png -exit This command creates a counts map file in png format called mytest.png with various options set. A full list of options appears in the ds9 reference manual. Running in batch requires another component as ds9 relies upon the X11 rendering engine. For example, a script like this works: #!/bin/sh export DISPLAY=:1 Xvfb :1 -screen 0 1024x768x16 & ds9 -file cmap.fits -zoom to fit -cmap b -grid skyformat degrees -grid yes -regions ../EMS-names.reg -saveimage png mytest.png -exit kill %1 exit The 'Xvfb' is an X11 virtual frame buffer – basically an X11 server without a physical display. Note that for batch jobs, the "kill %1" kills the Xvfb process, allowing the batch job to terminate normally (else it will hang until the job times out).

Exploring with fv

fv gives you much more interactive control of how you explore the data.  It can make plots and 1D and 2D histograms; allow you look at the data directly; and enable you to view the FITS file headers to look at some of the important keywords.  Starting it up is easy, just type fv and the filename:

This will bring up two windows: the general fv menu window (at right); and another window (shown below), which contains summary information about the file you are looking .
Summary Window: For the purposes of the tutorial, we will only be using the summary window, but feel free to explore the options in the main menu window as well. Looking at the summary window, notice that: There are three FITS extensions (Primary, EVENTs & GTI).  This is what should be there.  If you don't see three, there is something wrong with your file. There are 89804 events in the file (the number of rows in the EVENTS extension), and 22 data points (the number of columns) for each event. There are 3680 GTI entries. From this window, data can be viewed in different ways:
For each extension, the FITS header can be examined for keywords and their values. Histograms and plots can be made of the data in the EVENTS and GTI extensions. Data in the EVENTS or GTI extensions can also be viewed directly. Let's look at each of these in turn.

Viewing an Extension Header

Click on the button for the EVENTS extension; a new window listing all the header keywords and their values for this extension will be displayed. Notice that the same information is presented that was shown in the summary window; namely that: the data is a binary table (XTENSION='BINTABLE'); there are 89804 entries (NAXIS2=89804); and there are 22 data values for each event (TFIELDS=22). 

In addition, there is information about the size (in bytes) of each row and the descriptions of each of the data fields contained in the table.

As you scroll down, you will find some other useful infomation as shown in the screen shot below:

There are some fairly important keywords on this screen, including:

• DATE - The date the file was created.

• DATE-OBS - The starting time, in UTC, of the data in the file.

• DATE-END - The ending time, in UTC, of the data in the file.

• TSTART - The equivilant of DATE-OBS but in Mission Elapased Time (MET).  Note:  MET is the time system used in the event file to time tag all events.

• TSTOP - The equivilant of DATE-END in MET.

• MJDREF - The Modified Julian Date (MJD) of the zero point for MET.  This corresponds to midnight, Jan. 1st, 2001 for Fermi.
  

Finally, as you scroll down to the bottom of the header, you will see:

This part of the header contains information about the data cuts that were made to extract the data.  These are contained in the various DSS keywords.  For a full description of the meaning of these values see the DSS keyword page. In this file the DSVAL1 keyword tells us that the data was extracted in a circular region, 20 degrees in radius, centered on RA=193.98 and DEC=-5.82. The DSVAL2 keyword shows us the that the valid time range is defined by the GTIs. The DSVAL3 keywords shows the selected energy range in MeV, DSVAL4 shows the event class selection, and DSVAL5 indicates that a zenith angle cut has been defined.

Making a Counts Map fv can also be used to make a quick counts map to see what the region you extracted looks like:
In the Summary Window for the FITS file, click on the EVENTS --> Hist button.  A Histogram window (shown at right) will open.  Note: We use the histogram option since the "Plot" button is used to make scatter plots. From the X column's drop down menu in the column name field, select RA.
fv will automatically fill in the TLMin, TLMax, Data Min and Data Max fields based on the header keywords and data values for that column.  It will also make guesses for the values of the Min, Max and Bin Size fields.
From the Y column's drop down menu in the column name field, select DEC from the list of columns.
Select the limits on each of the coordinates in the Min and Max boxes.  Set the limits to be just larger than the values in the Data Min and Data Max field for each column: Min=173, -26 Max=215, 15 Set the bin size for each column to 0.1o.
Note: Always set the bin size i n the units of the respective column; in this case, the bin size is in degrees, and we've selected 0.1 degree bins for this map.  You can also select a data column from the FITS file to use as a weight, e.g., ENERGY.

Click on the “Make” button to generate the map.  This will create the plot in a new window and keep the histogram window open in case you want to make changes and create a different image.  The "Make/Close" button will create the image and close the histogram window. Note: fv also allows you to adjust the color and scale, just as you can in ds9.  However, it has a different selection of color maps. As in ds9, the default is gray scale.  The image at right was displayed with the color3 color map, selected by clicking on: Colors --> Continuous --> color3

Making 1D Histograms

Making a 1D histogram is similar to making the counts map, except that you only select one column to plot instead of two.  In this example, we will take a quick look at the light curve of the gamma-ray burst data we extracted. 

Start by opening the burst.fits file with fv:

This brings up a File summary window that looks like:

Note: There were a total of 1102 events in the 750 seconds time window we extracted.

To make a lightcurve for this data:

Click on the "Hist" button to bring up the histogram window. In the X column, click on the drop down box and select TIME. Set the Min and Max fields for the start and stop time you want the plot to cover; for example: Min=243216274 Max=243218274 To look at the entire time range, we selected Min and Max values, which were just larger than the time range of the file.
Set the Bin Size of the time bins you want to use; in this case, we selected 10 seconds.

To create the image, click on the Make button. This generates the image shown below on the left.  The burst stands out quite nicely in this image as a huge spike above the typical background rate of ~5 photons.
	Note: fv also allows you to zoom in by simply clicking on the window and dragging the cursor to enclose the region you wish to zoom in on.		For example: By zooming in on the narrow region close to the burst we get the image below which shows that all the events basically fall into one of our 10 second time bins.

Looking at the Raw Data

3. Binning the Data

In this section we will use the 3C279_region_filtered_gti.fits and grb_events.fits files to make images, lightcurves, and energy spectra using gtbin and then look at them using fv and ds9.

While ds9 and fv can be used to make quick plots when exploring the data, they don't automatically do all the things you would like when making data files for analysis; for this, use the Fermi-specific gtbin tool to manipulate the data. (See gtbin parameters and gtbin Help.)

You can use gtbin to bin photon data into the following representations:

• Images (maps).
• Light curves.
• Energy spectra (PHA files).

This has the advantage of creating the files in exactly the format needed by the LAT ScienceTools (as well as by tools such as XSPEC). gtbin also adds the correct WCS keywords to the images, so that the coordinate systems are properly displayed when using image viewers (such as ds9 and fv) that can correctly interperet the WCS keywords. For example, when using gtbin, the proper header keywords are added, so that the coordinate system is properly displayed as you move around an image. 

Images

In this section, we'll make the same image of the 3C279 region that we made with fv and ds9, but this time we'll use gtbin to make the image. (See gtbin parameters and gtbin Help.)

Raw Count Lightcurves

gtbin can also be used to generate light curves of the data you extracted.

Note: This procedure produces raw counts lightcurves which have not been corrected for exposure variations or the presence of background events.

To generate a light curve of the extracted burst data:

prompt>gtbin
This is gtbin version ScienceTools-09-20-00
Type of output file (CCUBE|CMAP|LC|PHA1|PHA2) [CMAP] LC
Event data file name[] grb_events.fits
Output file name[] grb_lc.fits
Spacecraft data file name[] NONE
Algorithm for defining time bins (FILE|LIN|SNR) [] LIN
Start value for first time bin in MET[0] 243216266
Stop value for last time bin in MET[0] 243218266
Width of linearly uniform time bins in seconds[0] 10
prompt>

Notes:

• In this case, we chose the lightcurve (LC) option when prompted for the type of output file, then entered the name of the output file; a spacecraft file is not needed. 

• When asked for the algorithm for defining the time bins, choices were bins defined in a file (FILE), linear bins (LIN), or bins with equal signal-to-noise ratios (SNR). 

In this case, the linear option was chosen.

For details of how the other options work, see gtbin parameters and gtbin Help.

• Next are prompts for the start and stop time of the bins to be generated. 

• Since linear was selected, gtbin prompts for the width of the bins in seconds; here we selected 10 second bins.

• The resulting lightcurve is simply the total number of counts per time bin contained in the input file. No exposure effective correction is applied. (See gtexposure.)

Use fv instead of ds9 to look at the output file since the output is not an image, but rather a table of values; enter (at the prompt): fv grb_lc.fits

fv summary window for the grb_lc.fits file:

Note that the main data extension, RATE, has four columns and, 200 rows.

Click on the 'Plot' button in the summary window in the RATE extension line. The window shown below will be displayed.
To plot the counts verses time, first click on the TIME column, then the X axis button. Click on the COUNTS field in the left pane and the Y button in the right pane. Click on the Go button.
A plot of the light curve will be displayed and, as in the example on making 1D histograms with fv, you can zoom in on any particular part of the image by drawing a selection around that part of the image with the mouse:

Energy Spectra

The final use of gtbin is to make energy spectra.  There are actually two options:

• PHA1 -- A spectrum over a single time bin.
• PHA2 -- A set of spectra over several time bins. 

This example will focus on PHA1 as we construct an energy spectrum for the entire 3C279 region extracted earlier. 

Note: While there is arguably no scientific value in this as there are a dozen sources in the field, it does demonstrate how to perform the analysis and, after we have talked about making data subselections, this can be used to make spectra of individual objects. 

To begin, run gtbin on the anticenter data set:

prompt>gtbin
This is gtbin version ScienceTools-09-20-00
Type of output file (CCUBE|CMAP|LC|PHA1|PHA2) [] PHA1
Event data file name[] 3C279_region_filtered_gti.fits
Output file name[] 3C279_spectrum.fits
Spacecraft data file name[NONE] 
Algorithm for defining energy bins (FILE|LIN|LOG) [] LOG
Start value for first energy bin in MeV[30] 100
Stop value for last energy bin in MeV[200000] 100000
Number of logarithmically uniform energy bins[0] 100
prompt>

Notes:

• Choices for how you want to define the energy bins are FILE (the bins defined in an input file), LIN (equally spaced in energy), or LOG (equally spaced in log(E)). For this example, LOG was chosen. 

• Like the light curve file, the output file is a binary table -- not an image; using fv to look at it, gives the following summary window:

In addition to the standard primary and GTI exensions, This file consists of two data exensions; the:  SPECTRUM extension -- Contains the actual spectrum and is shown in the image at the near right. It contains two columns, the CHANNEL number and the COUNTS in that channel.

EBOUNDS extension -- Contains the energy limits of each of the channels specified in the SPECTRUM extension.  For each CHANNEL, the minimum and maximum energy is given in the E_MIN and E_MAX columns, respectively.

To look at the energy spectrum:

1. On the SPECTRUM extension, click on the 'Plot' button.

Note: This is the same as we did on the RATE extension when looking at the light curve. 

2. Select the CHANNEL column for the X axis.

3. Select the COUNTS column for the Y axis.

4. Press the 'Go' button. 

The image on the left will be displayed.

Note: The channels are in log energy, so this is a log-linear plot of the energy spectrum.			Click on the 'Edit' menu item, then select: 'Axes Transforms' --> 'Linear-log'.

4. Looking at the Exposure

In order to determine the exposure for your source, it is necessary to understand how much time the LAT has observed any given position on the sky at any given inclination angle. gtltcube calculates this 'exposure cube' for the entire sky for the time range covered by the spacecraft file.

To look at the exposure:

• Create a livetime exposure cube from the spacecraft data file using gtltcube. (See gtltcube parameters and gtltcube Help.)

Note: Merge, as necessary, multiple exposure cubes, which cover different time ranges. (See tip, below.)

• Create an exposure map.

• Examine the map using ds9.

Calculate the Livetime

prompt> gtltcube
  Event data file [] 3C279_region_filtered_gti.fits
  Spacecraft data file [] spacecraft.fits
  Output file [] 3C279_exposure_cube.fits
  Step size in cos(theta) (0.:1.) [] 0.025
  Pixel size (degrees) [] 1
  Working on file spacecraft.fits
  .....................!
prompt>

Notes:

• Some values, such as "0.1", are known to give unexpected results for "Step size in cos(theta)". Use 0.099 instead.

• The event file is needed to get the time range over which to generate the exposure cube.

• Size of the bins to use in accumulating the livetime (defaults are typically sufficent).

• Generating the exposure cube is probably the most time consuming portion of your analysis, as it is calculating exposure for all angles over the entire sky.

It processes much more slowly than making the exposure map, which is the next step.

The details recorded in the exposure cube file are multi-dimensional and difficult to visualize. For that, we will need an exposure map: enter (at the prompt):

fv 3C279_exposure_cube.fits         The following GUI will be displayed:

Tip: Use gtltsum to combine multiple livetime exposure cubes. In some cases, you will have multiple exposure cubes covering different periods of time that you wish to combine in order to examine the exposure over the entire time range; e.g., a researcher who generates weekly flux datapoints for lightcurves and needs to analyze the source significance over a larger time period. In such a case, it is much less CPU-intensive to combine previously generated exposure cubes before calculating the exposure map than it is to start the exposure cube generation from scratch. To combine multiple exposure cubes into a single cube, use gtltsum. (See gtltsum parameters and gtltsum Help.) Note: gtltsum is quick, but it does have a few limitations: It will only add two cubes at a time. If you have more than one cube to add, you must do them one at a time. It does not allow you append to or to overwrite an existing exposure cube file. For example, if you want to add four cubes (c1, c2, c3 and c4), you cannot add c1 and c2 to get cube_a, then add cube_a and c3 and save the result as cube_a; you must give it a different name. The calculation parameters that were used to generate the exposure cubes (step size and pixel size) must be identical between the exposure cubes. Optional example, if you wish, you can download the first and second halves of the six months of 3C279 data (where the midpoint was 247477908 MET): 3C279_region_first_expcube.fits 3C279_region_second_expcube.fits then combine them using gtltsum: prompt> gtltsum Livetime cube 1 or list of files[] 3C279_region_first_expcube.fits Livetime cube 2[none] 3C279_region_second_expcube.fits Output file [] : 3C279_region_summed_expcube.fits prompt>

Generate an Exposure Map

Once you have an exposure cube for the entire dataset, run gtexpmap to generate the exposure map for your dataset. (See gtexpmap parameters and gtexpmap Help.)

To generate an all sky Aitoff map of the exposure for the 3C 279 data file, use gtexpmap.

Note: Use the exposure maps generated by gtexpmap for unbinned likelihood analyses only; do not use them for a binned analysis.

prompt> gtexpmap
The exposure maps generated by this tool are meant
to be used for *unbinned* likelihood analysis only.
Do not use them for binned analyses.
Event data file[] 3C279_region_filtered_gti.fits
Spacecraft data file[] spacecraft.fits
Exposure hypercube file[] 3C279_exposure_cube.fits
output file name[] 3C279_exposure_map.fits
Response functions[] P6_V3_DIFFUSE
Radius of the source region (in degrees)[] 25
Number of longitude points (2:1000) [] 500
Number of latitude points (2:1000) [] 500
Number of energies (2:100) [20] 20
The radius of the source region, 25, should be significantly larger
(say by 10 deg) than the ROI radius of 20
Computing the ExposureMap using 3C279_exposure_cube.fits
....................!
prompt>

You are prompted for the:

• Name of an events file to determine the energy range to use.
• Name of the exposure cube file to use.
• Name of the output file and the instrument response function to use.

  For more discussion on the proper instrument response function (IRF) to use in your data analysis, see the:

  • Livetime and Exposure
  • LAT Data Selection Recommendations
  • Caveats on the Use of LAT Data

• The next set of parameters specify the size, scale, and position of the map to generate.
• The radius of the 'source region'.

  The source region is different than the region of interest (ROI). This is the region that you will model when fitting your data. As every region of the sky that contains sources will also have adjacent regions containing sources, it is advisable to model an area larger than that covered by your dataset. Here we have increased the source region by an additional 5 degrees. Be aware of what sources may be near your region, and model them if appropriate (especially if they are very bright in gamma rays).

• Number of longitude and latitude points.

• Number of energy bins that will have maps created.

  This number can be small (∼5) for sources with flat spectra in the LAT regime. However, for sources like pulsars that vary in flux significantly over the LAT energy range, a larger number of energies (10-20) is recommended.

Look at the Exposure Maps

These maps are mono-energetic, and represent the exposure at the midpoint of the energy band, not integrated over the band's energy range.

To look at the maps, enter (at the prompt): ds9 3C279_exposure_map.fits

The 1st map (i.e., layer) will be displayed, and a "Data Cube" window will appear, allowing you to select between the various maps generated. In this example, a total of 20 maps were generated.
Note: The following images were obtained using DS9 v. 6, obtained on SLAC Central Linux by opening a bash shell, then entering: source /afs/slac/g/ki/software/etc/profile.xray ds9 3C279_exposure_map.fits

First Layer:	Fourth Layer:
Seventh Layer:	Tenth Layer: