Work with ARM Data

Resources and information are provided to help ARM users work with data.
 

ARM Data Formats

Justin Monroe, from the ARM Data Quality Office at the University of Oklahoma, led a tutorial on ARM NetCDF files and how to navigate the ARM Python notebook during the 2014 Joint User Principal Investigator Meeting.

Most ARM data are made available as time-series data in netCDF (network Common Data Form), which is a set of libraries and self-describing, platform-independent data formats that support creating, accessing, and sharing array-oriented scientific data. It is relatively compact, appendable, and capable of storing descriptive “metadata” along with measurement data.

NetCDF includes data access libraries for a wide variety of programming languages. Example software programs for reading and writing netCDF files using C, C++, Fortran, IDL®, JavaTMMATLAB®, PythonTM, and Perl, as well as more information about netCDF, including downloads, documentation, and frequently asked questions can be found on the Unidata website.

Some data files also contain measurements distributed over a region from a single time (e.g., satellite images), which are stored in HDF (Hierarchical Data Format) files. Raw model data files may also be stored in GRIB (GRIdded Binary) and GRIB2 file formats. For more information, see Reading netCDF, HDF, and GRIB Files.

Tutorials

To help new users get started, ARM has developed a short Introduction to Reading and Visualizing ARM Data tutorial that steps through setting up some basic tools and illustrates how to read a simple ARM data file (using the Python programming language as an example).

An introductory, video tutorial on how to submit, discover, and acknowledge ARM data is also available.

Preview Plots of ARM Data

Quick-look plots provide users the ability to assess the content of ARM data files without specialized software or downloading. The plots are available from the ARM Data Center through Data Discovery after selecting a set of data to order.

After data have been ordered, ARM’s NCVWeb tool provides the means to perform a variety of data extraction, data conversion, and data visualization tasks.

Open Source Programming

Scott Collis, Argonne National Lab, led the first-ever PyART short-course during the 2014 Joint User Principal Investigator Meeting. This course covers an intro to radar data and what it looks like in PyART, plotting RHIs, and basic analysis, and is an at-home exercise.

Scientists working with ARM Facility data are encouraged to share experiences and codes and are invited to engage in community software development and make use of the ARM GitHUB account—especially for routine processing activities—to accelerate the collective application of these data.

Coding guidelines have been developed to help streamline the process of integrating PI-developed algorithms with ARM processes and libraries. The ARM development team has also created an open-source tool called the ARM Data Integrator (ADI) to integrate and transform multiple diverse datastreams and facilitate the creation of datastreams that meets ARM standards. ADI is available to use on OS-X and can be downloaded from the ADI folder in the ARM GitHUB account.

The Python ARM Radar Toolkit (Py-ART) is another open-source toolkit that combines a variety of utilities for processing ARM radar data. Py-ART is the first ARM open-source project.

Computing Resources

The ARM Data Center offers a public software development space that provides a mechanism to work with large volumes of data without having to download them. The Data Processing and Visualization Cluster provides programming ability in ADI, Python, MATLAB, and IDL, as well as specialized software such as Py-ART.

Data Questions

For questions regarding ARM data, please use the Data Questions form, or call 1-888-ARM-DATA. If your question is instrument-specific, please use the ARM contact form and select Instruments from the topics. If your question is related to Value Added Products (VAP) please contact the translator associated with that VAP.