areaDetector Plugin NDFileHDF5

NDFileHDF5 inherits from NDPluginFile. This plugin uses the HDF5 libraries to store data. HDF5 file format is a self-describing binary format supported by the hdfgroup.

The plugin supports all NDArray datatypes and any number of NDArray dimensions (tested up to 3). It supports storing multiple NDArrays in a single file (in stream or capture modes) where each NDArray gets appended to an extra dimension.

NDArray attributes are stored in the HDF5 file. In case of multi-frame files the attributes are stored in 1D datasets (arrays).

The NDFileHDF5 plugin is created with the NDFileHDF5Configure command, either from C/C++ or from the EPICS IOC shell.

NDFileHDF5Configure (const char *portName, int queueSize, int blockingCallbacks, 
                     const char *NDArrayPort, int NDArrayAddr, size_t maxMemory, 
                     int priority, int stackSize)
  

For details on the meaning of the parameters to this function refer to the detailed documentation on the NDFileHDF5Configure function in the NDFileHDF5.cpp documentation and in the documentation for the constructor for the NDFileHDF5 class.

File Structure Layout

The HDF5 files comprises a hierachial data structure, similar to a file system structure with directories (groups) and files (datasets) [ref]

XML Defined File Structure Layout

It is possible to define the layout of the data structures in an XML definition file. The XML file allow defining the location of HDF5 Groups, Datasets and Attributes based on detector data (NDArrays), metadata (NDAttributes) and constants (for instance NeXus tags).

The XML definition contains the following 4 main elements: group, dataset, attribute, and hardlink. The terms refer to the HDF5 elements of the same names.

• attribute: represent a HDF5 attribute (key, value metadata) and is assigned to groups and datasets
• dataset: represent a HDF5 dataset (N-dimensional array). Can contain attributes and is assigned to a group
• group: represent a HDF5 group. Groups can contain datasets, attributes and other groups (recursively)
• hardlink: represent an HDF5 hard link. Hardlinks can point to other elements in the file, such as datasets. They are analogous to hard links in the Linux file system.
• global: Rather than representing a HDF5 object; this element represents a global functionality definition

An example XML layout file is provided in "ADExample/iocs/simDetectorIOC/iocBoot/iocSimDetector/hdf5_layout_demo.xml".

An XML schema is provided in "ADCore/iocBoot/hdf5_xml_layout_schema.xsd". The schema defines the syntax that is allowed in the user's XML definition. It can also be used with the 'xmllint' command to validate a user's XML definition:

xmllint --noout --schema ADCore/iocBoot/hdf5_xml_layout_schema.xsd /path/to/users/layout.xml

Default File Structure Layout

If no XML Layout Definition file is loaded, the plugin will revert to using its default file structure layout. The default layout is compatible with the plugin's original, hard-code layout. This layout is actually defined by an XML layout string defined in the source code file NDFileHDF5LayoutXML.cpp.

This default layout is compatible with the Nexus file format. This is achieved by defining a specific hierachial structure of groups and datasets and by tagging elements in the hierachy with certain "NX_class=nnn" attributes. Although Nexus libraries are not used to write the data to disk, this file structure allow Nexus-aware readers to open and read the content of these HDF5 files. This has been tested with the Nexus reader in the GDA application. Hardlinks in the HDF5 file can be used to make the same dataset appear in more than one location. This can be useful for defining a layout that is Nexus compatible, as well as conforming to some other desired layout.

NDArray attributes

The attributes from NDArrays are stored in the HDF5 files. The list of attributes is loaded when a file is opened so XML attributes files should not be reloaded while writing a file in stream mode.

If the dataset is defined in the XML layout file then the user-specified name is used for the dataset. If the dataset is not defined in the XML layout file then the dataset name will be the the NDArray attribute name. The NDArray attribute datasets automatically have 4 HDF "attributes") to indicate their source type and origin. These are:

• NDAttrName: The name of the NDArray attribute.
• NDAttrDescrption: The description of the NDArray attribute.
• NDAttrSourceType: The source type of the NDArray attribute: NDAttrSourceDriver, NDAttrSourceParam, NDAttrSourceEPICSPV, or NDAttrSourceFunct.
• NDAttrSource: The source of the NDArray attribute data, i.e. the name of the EPICS PV, the drvInfo string for the parameter, or the name of the attribute function.

NDArray attributes will be stored as 1D datasets. If the location of an NDArray attribute dataset is not defined in the XML layout file then the dataset will appear in the group that has the property ndattr_default="true". If there is no group with that property then the dataset will appear in the root group. If the root group has the property auto_ndattr_default="false" then datasets that are not explicitly defined in the XML layout file will not appear in the HDF5 file at all.

There are 4 "virtual" attributes that are automatically created. These are:

• NDArrayUniqueId: The NDArray.uniqueId value.
• NDArrayTimeStamp: The NDArray.timeStamp value.
• NDArrayEpicsTSSec: The NDArray.epicsTS.secPastEpoch value.
• NDArrayEpicsTSnSec: The NDArray.epicsTS.nsec value.

These properties are added to the property list and will be written to the HDF5 file following the same rules as the actual NDArray ndAttributes described above.

It is possible to validate the syntax of an NDArray attributes XML file. For example, the command (starting from the iocBoot directory) to validate the syntax of the iocBoot/iocSimDetector/simDetectorAttributes.xml file is:

xmllint --noout --schema ./attributes.xsd iocSimDetector/simDetectorAttributes.xml

Default tree structure

The group/dataset structure of the HDF5 files, generated by this plugin:

entry                   <-- NX_class=NXentry
|
+--instrument           <-- NX_class=NXinstrument
   |
   +--detector          <-- NX_class=NXdetector
   |  |
   |  +--data           <-- NX_class=SDS, signal=1
   |  |
   |  +--NDAttributes
   |     |
   |     +--ColorMode
   |
   +--NDAttributes      <-- NX_class=NXCollection, ndattr_default="true"
   |  |
   |  +---              <-- Any number of NDAttributes from the NDArrays as individual 1D datasets
   |
   +--performance       <-- Performance of the file writing
      |
      +--timestamp      <-- A 2D dataset of different timing measurements taking during file writing
+--data                 <-- NX_class=NDdata
   |  |
   |  +--data           <-- Hardlink to /entry/instrument/detector/data

HDF5 File Viewers

Note that with the current (1.8) release series of the HDF5 libraries it is not possible to do a live "monitoring" of a file which is being written by another process. The file writers mentioned in this section can only be used to open, browse and read the HDF5 files after the NDFileHDF5 plugin has completed writing and closed the file. This limitation will be lifted in the 1.10 release of the HDF5 library which will include the Single Writer Multiple Reader (SWMR) feature.

HDFView is a simple GUI tool for viewing and browsing HDF files. It has some limited support for viewing images, plotting graphs and displaying data tables.

The HDF5 libraries also ships with a number of command-line tools for browsing and dumping data.

The screenshot below shows the hdfview application with a datafile open. The datafile is generated by the plugin and a number of elements are visible:

• The NDArray NDAttributes appear as 1D datasets in the group "/entry/instrument/NDAttributes/"
• The image data is in the dataset "/entry/instrument/detector/data". The metadata (in HDF known as "attributes") for that dataset indicate 8bit unsigned char data, 10 frames of 60x40 pixels
• Image and table view of the first frame data is open

Multiple Dimensions

Both areaDetector and the HDF5 file format supports multidimensional datasets. The dimensions from the NDArray are preserved when writing to the HDF5 file. In multi-frame files (plugin in Stream or Capture mode) an additional dimension is added to the dataset to hold the array of frames.

In addition to the dimensions of the NDArray it is also possible to specify up to 2 extra "virtual" dimensions to store datasets in the file. This is to support applications where a sample is scanned in up to two dimensions, say X and Y. For each scan point a dataset comprising of multiple frames can be stored. The length of (i.e. number of points in) each of the two virtual dimensions have to be specified before the plugin opens the file for writing. This feature is only supported in the Stream and Capture modes.

This feature allow for creating very large sets of scan data which matches the dimensions of the performed scan in one datafile. Depending on the application this can be a benefit in post processing.

The figure below illustrate the use of the two extra "virtual" dimensions in a 2D (X,Y) raster scan with N frames per point:

Prior to starting a scan like this the user will need to configure the number of virtual dimensions to use (none, 1 or 2); the number of frames per point; and the length of each of the virtual dimensions (4 x 2 in the example figure). It is not possible to change the number or size of dimensions while the file is open.

For 2D image (greyscale) formats the dimensions in the multiframe HDF5 file are organised as follows:

• For a multiframe file with no use of "virtual" dimension the order is: {Nth frame, width, height}
• For a multiframe file using 1 "virtual" dimension (X) the order is: {X, Nth frame, width, height}
• For a multiframe file using 2 "virtual" dimension (X,Y) the order is: {Y, X, Nth frame, width, height}

Chunking

This plugin uses HDF5 chunking to store the raw image data. The chunk size (the size of each I/O block) can be configured for the frame X and Y dimensions as well as the N'th image (which essentially implies memory caching before writing to disk). Configuring chunking correctly for a given application is a complex matter where both the write performance and the read performance for a given post processing application will have to be evaluated. As a basic starting point, setting the nColChunks and nRowChunks parameters to the X and Y frame size respectively, should give a decent result. In fact if these parameters are configured by the user to the special value 0, they will default to the dimensions of the incoming frames. Further explanations and documentation of the HDF5 chunking feature is available in the HDF5 documentation:

• HDF5 documentation advanced topics: Chunking in HDF5
• HDF5 User guide: 14.3 Data Chunking
• hdfgroup presentation: HDF5 Advanced Topics - Chunking in HDF5

Compression

The HDF5 library supports a number of compression algorithms. When using HDF5 libraries to write and read files the compression is seemless: it only need to be switched on when writing and HDF5 enabled applications can read the files without any additional configuration. Only one compression filter can be applied at the time.

The following compression filters are supported in the NDFileHDF5 plugin:

• Lossless SZIP compression is using a separate library from the hdfgroup. NOTE: The szip library contains the following in its COPYING license agreement file:
  Revocable (in the event of breach by the user or if required by law), royalty-free, nonexclusive sublicense to use SZIP compression software routines and underlying patents for non-commercial, scientific use only is hereby granted by ICs, LLC, to users of and in conjunction with HDF data storage and retrieval file format and software library products.
  This means that the szip compression should not be used by commercial users without first obtaining a license.
• External libz -also lossless
• N-bit compression is a bit-packing scheme to be used when a detector provide fewer databits than standard 8,16,32 bit words. Data width and offset in the word is user configurable

Parameters and Records