Smart Filter Clears the Way for Speedy Data Transfer

 
Published: 30 April 2009
These data plots illustrate the results of the smart filter in reducing the volume of MMCR data. The left column shows the full reflectivity data for individual radar data collection modes: cirrus, precipitation, general, and boundary layer. The right column shows the data retained after applying the clear-sky filter.
These data plots illustrate the results of the smart filter in reducing the volume of MMCR data. The left column shows the full reflectivity data for individual radar data collection modes: cirrus, precipitation, general, and boundary layer. The right column shows the data retained after applying the clear-sky filter.

As reported in mid-February, data transfer from the ARM Tropical Western Pacific sites was greatly improved through an upgraded internet connection, which eliminated the need to ship hard disks of data from the millimeter wavelength cloud radar (MMCR) around the world. But this was only half of the story. The other, and more complicated half, involved the development of a sophisticated “smart” filter, which successfully reduced the volume of MMCR data by up to 15 times. By filtering out unnecessary information from the final output files, data transfer and processing is much faster while not impacting the value of the data for scientific discoveries.

Historically, the MMCR data files included “clear sky” information that wasn’t necessary for the needed measurements of cloud boundaries, reflectivities, and vertical velocities. Recognizing that the clear sky information contributed substantially to the already large data files produced by the MMCR, researchers began developing an algorithm to remove that clear sky information and reduce the data volume. Using a copy of existing MMCR data, the team tested various algorithms on a research computer. They eventually settled on a series of algorithms that act as a mask to filter out the clear sky information.

Making sure to save a copy of the uncompressed MMCR data on a backup for the Data Archive, the algorithm was implemented first at the ARM Southern Great Plains (SGP) site. After several months of successful data collection using the new algorithm, the backup was deemed unnecessary and the filter was applied for the MMCR in Darwin, Australia, and for the W-band ARM cloud radar deployed with the ARM Mobile Facility in China. Most recently, in March 2009, the Microphysical Active Remote Sensing of Clouds evaluation product, which uses the filtered data as input, was released for evaluation by the scientific community. This initial release included a year of data (May 2007 through May 2008) from the SGP site, with additional years to be added in the future. This evaluation product, combined with the substantially smaller filtered data sets, will greatly speed data processing and use of the information-rich MMCR spectral data.