All the data collected through the ARM Climate Research Facility have to go somewhere, but it’s not easy to pack them all in—or deliver them from—one spot. Every month, hundreds of users access the ARM Data Archive to download portions of ARM data collections at rates from 20 terabytes (TB) to 25 TB per month.
Responding to an increase in the volume of data requests, the ARM Data Center, through the ARM Data Discovery Tool, offers users three new ways to acquire data: Globus, Dropbox, and Unidata’s Thematic Real-time Environmental Distributed Data Services (THREDDS). From the Data Discovery Tool cart, users can select one or more of these data delivery options.
“Over the years, our users have wanted a better way to download large quantities of ARM data,” says Giri Prakash, ARM Data Services and Operations Manager at Oak Ridge National Laboratory.
Each method has its advantages, particularly over conventional FTP delivery.
Globus provides higher bulk data transfer speeds and automatically resumes downloads after any network interruption.
“Globus will make sure all the files you order are securely transferred to your system,” Prakash says.
Users preferring a commercial file-hosting service can use the Dropbox option. Users can link their personal Dropbox accounts—a 2-gigabyte account is free—to ARM’s institutional-level account and not drain their space. Meanwhile, if you’re looking for specific data in a large file, THREDDS allows users to extract the measurements they need using a variety of remote data access protocols.
# # #
The ARM Climate Research Facility is a national scientific user facility funded through the U.S. Department of Energy’s Office of Science. The ARM Facility is operated by nine Department of Energy national laboratories, including Oak Ridge National Laboratory.