History Tool Records Changes to Datastreams

 
Published: 31 May 2005

As the name implies, a “datastream” is a continuous record of a specific type of data that is collected and saved in a time series. Occasionally, modifications are needed to update or correct a datastream. Knowing the history of changes to a datastream is a critical issue for ARM data users. Recognizing this need, the ARM Data Management Facility (DMF) created a tool to record the history of every datastream for every instrument from every ARM site since the beginning of the ARM Program.

The History of Datastream tool looks for changes to the data object design (DOD) files weekly and creates a history link for DODs that have changed. By selecting "history," users will have access to the old DOD file and a file outlining the differences.

Each day, the DMF processes approximately 1875 datastreams from 287 instrument systems and delivers them to the Data Archive. These datastreams contain netCDF (network Common Data Form) files, which is the standard file format for ARM measurement collection. Each datastream has a specific netCDF structure that is defined and managed by a data object design (DOD) file. Historically, the metadata within the DODs were revised and overwritten whenever a change occurred, with no record of the change saved for reference. Not anymore!

In March, the History of Datastream tool began providing information on changes to the metadata, and when the changes occurred—a historical record—for all the datastreams available at the DMF. Once a week, the tool scans the current DOD files, identifies any changes, and adds a link to “history” (see figure) for each DOD impacted. This information is available from the ARM netCDF File Header Description webpage by selecting the instrument or value-added product’s DOD. Soon, ARM’s Data Archive and External Data Center (XDC) will begin using the History of Datastream tool to help build a historical record of datastream changes since the sites first began collecting data, as well as to maintain a history of datastream changes in the future.