areaDetector Plugin NDPluginFile

May 19, 2010

Mark Rivers

University of Chicago

Contents

Overview

NDPluginFile is a base class from which actual file plugins are derived. There are currently file plugins for JPEG, TIFF, netCDF, and Nexus (HDF) file formats.

NDPluginFile inherits from NDPluginDriver. The NDPluginFile class documentation describes this class in detail. This class is designed to simplify the task of supporting a new file format. A derived class to support a new file format will typically need to implement only the pure virtual functions openFile(), readFile(), writeFile(), and closeFile(). Note that none of the current file plugins actually support the readFile() function yet, but this is planned for future releases.

The NDArray callback data can be written to disk in 1 of 3 modes:

  1. Single mode. In this mode each NDArray callback results in a separate disk file.
  2. Capture mode. In this mode a memory buffer is allocated before saving begins. Callback arrays are placed into this buffer, and when capture stops the file is written to disk. This mode limits the number of frames that can be saved, because they all must fit in a memory buffer. It is the fastest mode, with the least probability of dropping arrays, because no disk I/O is required while capture is in progress.
  3. Stream mode. In this mode the data are written to a single disk file for those file formats that support multiple arrays per file (currently only netCDF; NeXus/HDF support for multiple arrays per file is planned for a future release). Each frame is appended to the file without closing it. It is intermediate in speed between single mode and capture mode, but unlike capture mode it is not limited by available memory in the number of arrays that can be saved. For file formats that do not support multiple arrays per file (e.g. JPEG, TIFF and currently NeXus/HDF) this mode is really the same as Single mode, except that one can specify a total number of files to save before stopping.

At least one array with the same datatype, array size, and attributes must have been collected by the driver (and/or plugins) from which the file saving plugin is getting its data before capture or stream mode file saving is started. This is required so that the openFile() function can know the dimensions and datatype of the arrays.

NDPluginFile supports all of the file saving parameters defined in asynNDArrayDriver, e.g. NDFilePath, NDFileName, etc. Thus, the same interface that is used for saving files directly in a driver is used for this plugin.

JPEG file plugin

NDFileJPEG inherits from NDPluginFile. This plugin saves data in the JPEG file format, which is a compressed file format for storing images. There is JPEG support for almost all languages and programs such as IDL and Matlab.

The JPEG plugin is limited to 8-bit arrays. It supports all color modes (Mono, RGB1, RGB2, and RGB3). It is limited to a single array per file, but capture and stream mode are supported by writing multiple JPEG files.

The JPEG plugin supports the Int32 parameter NDFileJPEGQuality to control the amount of compression in the file. This parameter varies from 0 (maximum compression, lowest quality) to 100 (least compression, best quality). NDFileJPEG.template defines 2 records to support this: $(P)$(R)JPEGQuality (longout) and $(P)$(R)JPEGQuality_RBV (longin).

The NDFileJPEG class documentation describes this class in detail.

The NDFileJPEG plugin is created with the NDFileJPEGConfigure command, either from C/C++ or from the EPICS IOC shell.

NDFileJPEGConfigure (const char *portName, int queueSize, int blockingCallbacks, 
                     const char *NDArrayPort, int NDArrayAddr, size_t maxMemory, 
                     int priority, int stackSize)
  

For details on the meaning of the parameters to this function refer to the detailed documentation on the NDFileJPEGConfigure function in the NDFileJPEG.cpp documentation and in the documentation for the constructor for the NDFileJPEG class.

TIFF file plugin

NDFileTIFF inherits from NDPluginFile. This plugin saves data in the TIFF file format, which is a popular file format for storing images. There is TIFF support for almost all languages and programs such as IDL and Matlab.

The TIFF plugin is limited to 8, 16 and 32-bit integer arrays. It supports all color modes (Mono, RGB1, RGB2, and RGB3). Note that some TIFF readers do not support 16 or 32 bit TIFF files, and many do not support 16 or 32 bit color files. NDFileTIFF is limited to a single array per file, but capture and stream mode are supported by writing multiple TIFF files.

The NDFileNetTIFF class documentation describes this class in detail.

The NDFileTIFF plugin is created with the NDFileTIFFConfigure command, either from C/C++ or from the EPICS IOC shell.

NDFileTIFFConfigure (const char *portName, int queueSize, int blockingCallbacks, 
                     const char *NDArrayPort, int NDArrayAddr, size_t maxMemory, 
                     int priority, int stackSize)
  

For details on the meaning of the parameters to this function refer to the detailed documentation on the NDFileTIFFConfigure function in the NDFileTIFF.cpp documentation and in the documentation for the constructor for the NDFileTIFF class.

netCDF file plugin

NDFileNetCDF inherits from NDPluginFile. This plugin saves data in the netCDF file format, which is a portable self-describing binary file format supported by UniData at UCAR (University Corporation for Atmospheric Research). There are netCDF libraries for C, C++, Fortran, and Java. Other languages, including Matlab and IDL have built-in support for netCDF. There are also add-on interfaces available for Python, Ruby and other languages.

The netCDF plugin supports all NDArray data types and any number of array dimensions. It also has full support for NDArray attributes. It will write all attributes associated with the NDArray to the file. If multiple arrays are written to a single netCDF file (stream or capture mode) then each attribute will be an array, with the attribute value for each NDArray in the file being stored. Note that the number and data types of attributes must not be changed while file capture or file streaming are in progress because that would change the structure of the attribute array. Also the colorMode attribute must not be changed while capture or streaming is in progress, because that would change the structure of the NDArray data.

The NDFileNetCDF class documentation describes this class in detail.

The NDFileNetCDF plugin is created with the NDFileNetCDFConfigure command, either from C/C++ or from the EPICS IOC shell.

NDFileNetCDFConfigure (const char *portName, int queueSize, int blockingCallbacks, 
                       const char *NDArrayPort, int NDArrayAddr, size_t maxMemory, 
                       int priority, int stackSize)
  

For details on the meaning of the parameters to this function refer to the detailed documentation on the NDFileNetCDFConfigure function in the NDFileNetCDF.cpp documentation and in the documentation for the constructor for the NDFileNetCDF class.

The following is the header contents of a netCDF file produced by this plugin. This information was produced with the following command:

ncdump -h test_netCDF_68.nc

netcdf test_netCDF_68 {
dimensions:
	numArrays = UNLIMITED ; // (10 currently)
	dim0 = 240 ;
	dim1 = 320 ;
	dim2 = 1 ;
	attrStringSize = 256 ;
variables:
	int uniqueId(numArrays) ;
	double timeStamp(numArrays) ;
	float array_data(numArrays, dim0, dim1, dim2) ;
	int Attr_colorMode(numArrays) ;
	double Attr_AcquireTime(numArrays) ;
	double Attr_RingCurrent(numArrays) ;
	int Attr_ImageCounter(numArrays) ;
	char Attr_CameraModel(numArrays, attrStringSize) ;
	int Attr_BinX(numArrays) ;
	int Attr_BinY(numArrays) ;
	double Attr_AttrTimeStamp(numArrays) ;
	double Attr_ROI0Mean(numArrays) ;
	double Attr_ROI1Mean(numArrays) ;
	char Attr_FilePath(numArrays, attrStringSize) ;
	char Attr_FileName(numArrays, attrStringSize) ;

// global attributes:
		:dataType = 6 ;
		:NDNetCDFFileVersion = 3. ;
		:numArrayDims = 3 ;
		:dimSize = 1, 320, 240 ;
		:dimOffset = 0, 0, 0 ;
		:dimBinning = 1, 2, 2 ;
		:dimReverse = 0, 0, 0 ;
		:Attr_colorMode_DataType = "Int32" ;
		:Attr_colorMode_Description = "Color mode" ;
		:Attr_colorMode_Source =  ;
		:Attr_colorMode_SourceType = "Driver" ;
		:Attr_AcquireTime_DataType = "Float64" ;
		:Attr_AcquireTime_Description = "Camera acquire time" ;
		:Attr_AcquireTime_Source = "13SIM1:cam1:AcquireTime" ;
		:Attr_AcquireTime_SourceType = "EPICS_PV" ;
		:Attr_RingCurrent_DataType = "Float64" ;
		:Attr_RingCurrent_Description = "Storage ring current" ;
		:Attr_RingCurrent_Source = "S:SRcurrentAI" ;
		:Attr_RingCurrent_SourceType = "EPICS_PV" ;
		:Attr_ImageCounter_DataType = "Int32" ;
		:Attr_ImageCounter_Description = "Image counter" ;
		:Attr_ImageCounter_Source = "ARRAY_COUNTER" ;
		:Attr_ImageCounter_SourceType = "Param" ;
		:Attr_CameraModel_DataType = "String" ;
		:Attr_CameraModel_Description = "Camera model" ;
		:Attr_CameraModel_Source = "MODEL" ;
		:Attr_CameraModel_SourceType = "Param" ;
		:Attr_BinX_DataType = "Int32" ;
		:Attr_BinX_Description = "X binning" ;
		:Attr_BinX_Source = "13SIM1:ROI1:0:BinX_RBV" ;
		:Attr_BinX_SourceType = "EPICS_PV" ;
		:Attr_BinY_DataType = "Int32" ;
		:Attr_BinY_Description = "Y binning" ;
		:Attr_BinY_Source = "13SIM1:ROI1:0:BinY_RBV" ;
		:Attr_BinY_SourceType = "EPICS_PV" ;
		:Attr_AttrTimeStamp_DataType = "Float64" ;
		:Attr_AttrTimeStamp_Description = "Time stamp" ;
		:Attr_AttrTimeStamp_Source = "TIME_STAMP" ;
		:Attr_AttrTimeStamp_SourceType = "Param" ;
		:Attr_ROI0Mean_DataType = "Float64" ;
		:Attr_ROI0Mean_Description = "Mean value ROI 0" ;
		:Attr_ROI0Mean_Source = "MEAN_VALUE" ;
		:Attr_ROI0Mean_SourceType = "Param" ;
		:Attr_ROI1Mean_DataType = "Float64" ;
		:Attr_ROI1Mean_Description = "Mean value ROI 0" ;
		:Attr_ROI1Mean_Source = "MEAN_VALUE" ;
		:Attr_ROI1Mean_SourceType = "Param" ;
		:Attr_FilePath_DataType = "String" ;
		:Attr_FilePath_Description = "File path" ;
		:Attr_FilePath_Source = "13SIM1:netCDF1:FilePath_RBV" ;
		:Attr_FilePath_SourceType = "EPICS_PV" ;
		:Attr_FileName_DataType = "String" ;
		:Attr_FileName_Description = "File name" ;
		:Attr_FileName_Source = "13SIM1:netCDF1:FileName_RBV" ;
		:Attr_FileName_SourceType = "EPICS_PV" ;
}  

ncdump is one of a number of very useful command line utilities that come with the netCDF package. The -h option to ncdump means to dump only the header information, not the variable data. This is an explanation of this output:

There is an IDL function, read_nd_netcdf that can be used to read the netCDF files created by this plugin. This routine is contained in the CARS IDL detector package. This function is also contained in the areaDetector distribution in the Viewers/IDL directory.

There is a plugin for the popular ImageJ program that can be used to read the netCDF files created by this plugin. This ImageJ plugin can be downloaded here. This plugin is also contained in the areaDetector distribution in the Viewers/ImageJ/EPICS_areaDetector directory.

NeXus (HDF) file plugin

A plugin to write NeXus files was written by John Hammonds from the APS. NeXus is a standard format for x-ray and neutron data based on HDF. This is a very general file format, capable of storing any type of array data and meta-data.

The NDFileNexus class documentation describes this class in detail.

The NDFileNexus plugin is created with the NDFileNexusConfigure command, either from C/C++ or from the EPICS IOC shell.

NDFileNexusConfigure (const char *portName, int queueSize, int blockingCallbacks, 
                      const char *NDArrayPort, int NDArrayAddr, size_t maxMemory, 
                      int priority, int stackSize)
  

For details on the meaning of the parameters to this function refer to the detailed documentation on the NDFileNexusConfigure function in the NDFileNexus.cpp documentation and in the documentation for the constructor for the NDFileNexus class.

NDFileNeXus uses 2 additional parameters to define the location of an XML file that is read to determine the contents of the NeXus files written by this plugin. These are described in the following table.

Parameter Definitions in NDFileNexus.h and EPICS Record Definitions in NDFileNexus.template
Parameter index variable asyn interface Access Description drvInfo string EPICS record name EPICS record type
Location of XML file to configure NeXus file contents
NDFileNexusTemplatePath asynOctet r/w Path to XML template file TEMPLATE_FILE_PATH $(P)$(R)TemplateFilePath
$(P)$(R)TemplateFilePath_RBV
waveform
waveform
NDFileNexusTemplateFile asynOctet r/w Name of XML template file TEMPLATE_FILE_NAME $(P)$(R)TemplateFileName
$(P)$(R)TemplateFileName_RBV
waveform
waveform

There is currently no documentation on the contents of the XML template file. However, there are example XML template files in the iocSimDetector and iocPerkinElmer directories. Documentation on the XML file contents will be written ASAP.

The prebuilt Linux libraries libhdf5. and libNeXus.a are built with HDF5 1.6.9. When they are built with the latest version, 1.8.2, they require GLIBC version 2.7 or higher, i.e. /lib/libc-2.7.so or higher. Since users may want to install areaDetector on older Linux systems (which predate Fedora Core 8 for example), it was decided to use this older version of HDF5. Future releases of areaDetector may use HDF5 1.8.2 or later, and hence not work with older Linux systems.

Screen shots

The following is the MEDM screen that provides access to the parameters in NDPluginDriver.h and NDPluginFile.h through records in NDPluginBase.template and NDFileNetCDF.template. This is the MEDM screen that is used to control the saving of images to disk in netCDF format.

NDPluginFileNetCDF.adl

NDFileNetCDF.png