Applications that query into very large multidimensional datasets are becoming more common. Many self-describing scientific data file formats have also emerged, which have structural metadata to help navigate the multidimensional arrays that are stored in the files. The files may also contain application specific semantic metadata. Our Generic Multidimensional Scientific Indexing Library (GMIL) enables us to perform efficient searches for subsets of multidimensional datasets, using semantic information to build multidimensional indexes, and grouping items into properly sized chunks to maximize disk I/O bandwidth. More information about data chunking can be found in the paper Improving Access to Multidimensional Self-describing Scientific Datasets.

The most common type of retrieval pattern into multidimensional datasets is a spatial range query, which reads a contiguous subset of one or more multi-dimensional arrays within the given query range. Using spatial indexing techniques, such as R*-trees or SH-trees, our indexing library allows for direct access to subsets of a dataset.

Figure 1. HDF-EOS AVHRR Level 1B Dataset Representation

Currently GMIL supports a few self-describing scientific data formats such as netCDF, HDF4 and HDF5, which contain structural metadata. Additional scientific data formats such as SILO or Chombo mesh data formats will be supported in the future releases. GMIL has an index creation module, an index search module, a resolution interpolation module, and a filtering module to facilitate performing range queries. Any new self-describing scientific data format can utilize GMIL by adding small amount of codes on top of it, as long as the indexing semantic is supported by GMIL. At the moment, GMIL supports AVHRR Level1B style indexing semantics such as HDF-EOS. In the futre releases, we plan to add as many indexing semantics as we can.

GMIT Version 1.0

The Generic Multidimensional Indexing Tool (GMIT) is a GUI tool for GMIL.