External Data Center Gets a Big Boost

 
Published: 31 March 2004

In addition to providing database support to the ARM Program, the External Data Center (XDC) identifies and acquires “external” sources of data to augment the data being generated within the program. These external data are transformed from their original received form to data sets of greater value to ARM researchers. These transformations may involve subsetting, merging distinct sets, changing format, averaging, or applying algorithms to produce new geophysical parameters. Recently, a locally attached storage system was added to the XDC production server, removing a significant bottleneck in the data transfer process and improving throughput by more than a factor of two.

The storage area on the XDC server is used to collect intermediate files generated during data ingest processing, most prominently in the Rapid Update Cycle (a short-range weather forecast numerical model) and GOES-9 (satellite) ingests. By optimizing existing local storage resources, the XDC was able to process the existing backlog of GOES-9 data months ahead of schedule. Following completion of backlog data processing, current data started to be ingested on February 17, 2004.

This concludes an intensive effort that was initiated in June 2003, and significantly enhances ARM’s ability to access timely external atmospheric data. This optimization also provides sufficient extra processing capacity for all the external data streams until a major hardware upgrade planned for later this fiscal year.