20 years of ARM information is settled safely in its new platform
There’s a reason that moving always makes the “top ten” list of life’s biggest stressors. The effort requires organization and meticulous attention to detail to avoid losing anything in the process. Moving was a particularly challenging feat when the ARM Climate Research Facility migrated 20 years worth of metadata, archived at the Oak Ridge National Laboratory (ORNL) in Tennessee, to an open-source database platform earlier this year.
“The archive database is the ‘brains’ of ARM data systems, so you just can’t shut it down,” said Giri Palanisamy, the ARM Data Services and Operations Manager who led the transfer effort. With advance planning that included sequestering a troubleshooting team in a “war room,” the informatics experts at ORNL made the move smooth.
ARM collects information from routine operations and field experiments using more than 500 instruments—ranging from rain gauges to Raman lidar—located at stationary and mobile sites around the world. All that information is then made available to the public for free. At ORNL, the ARM Data Archive contains “metadata,” the critical details about these measurements and the instruments.
However, the amount of incoming data—and the accompanying metadata—is continually increasing as ARM deploys more instruments (such as the radar systems recently shipped to Alaska, Antarctica, and the Azores). From 2002 to 2011, ARM collected 300 terabytes of data. During the next 4 years, Palanisamy noted that number climbed to 750 terabytes. Looking ahead, he expects the ARM Data Archive to store metadata that describes about 1.5 petabytes of new ARM data each year.
“We need to be forward thinking,” said Harold Shanafield, an environmental informatics developer at ORNL. “The datastreams we get on a daily basis range in size from two megabytes to tens of gigabytes generated by some radars. The goal is to run more of these larger scale operations, so we need a platform robust enough to handle that growth, making sure nothing gets lost or corrupted in the process.”
The new platform is called PostgreSQL (Postgres for short). This free, community-based software gives users the flexibility to expand and, in more recent iterations, replicate the system as needed. Other large government agencies, such as National Aeronautics and Space Administration and the U.S. Geological Survey, already use this platform.
Packing Up
The ARM Data Archive has used a third-party database server since they started collecting metadata about ARM’s atmospheric, cloud, and climate measurements in 1992. Over the years, using one server spawned “legacy” applications, all of which needed to migrate along with the database. Although these applications made the move more complex, they also provided an opportunity to update data structures created back in the 90s—to clean the cobwebs out of the proverbial attic.
“We got a chance to go back and look at the way we were doing things, see if there was a better way to do it, then go ahead and implement changes before we made the transfer,” said Shanafield. That work included streamlining database access permissions and developing more efficient ways to track incoming and outgoing data.
However, most of the applications depended on other applications, so they couldn’t be pushed out in piecemeal fashion to the new database to see if they worked, explained Palanisamy. So, the team set up a test database to migrate the contents, duplicated all the applications in the test environment, and then tested them; a process that lasted until February 2015.
Although the testing “went okay,” their task wasn’t done until the entire archive of metadata and applications resided on the new server. Yet the team wasn’t confident enough to make the big move. So, Palanisamy created a war room scenario: one big room, 14 team members, good food, and 2 days during which ARM Data Archive would “go dark.” It was “an all or nothing push,” he said.
Before the 2-day sequester, Palanisamy led a strategy session and set out the plan of attack: First they’d migrate the latest metadata content and then the old database, and finally bringing the applications on board, one at a time. Still, the team worried about executing the plan.
By late afternoon on the first day, most of the critical functionality was transferred; the ARM Data Archive could receive information and users could request data from the archive using the Postgres database. Those incremental successes gave everyone confidence—including Palanisamy. He might have otherwise reconsidered the timeline since he intended to finish the massive move on April Fools’ Day.
The second day, a wall chart marked the progression as applications were migrated, tested, and then up and running. As the day ended, Palanisamy marked off almost 90% of the chart. Nothing was left that couldn’t be finished or tweaked in the following weeks.
Moving On
With confidence and technical experience to share, Palanisamy and Shanafield are now helping their colleagues at the Brookhaven National Laboratory in New York move ARM’s External Data Center to the Postgres platform.
“Once we’re all on the same platform, we’ll be able to more seamlessly transfer information,” said Shanafield. Ultimately, both ORNL and Brookhaven will have reciprocal back-up of each other’s databases.
Beyond the technical aspects of the big move, Palanisamy credits the war room experience with instilling a new level of collaboration among his team members.
“Before this, we were solving just our own problems with this migration,” he said. “But in that room, we also had a chance to seek out others and share our expertise.”
Another bonus to the community-based platform move is that software can be changed to meet the needs of the ARM Facility, noted Palanisamy. “If we think those modifications might be useful for others, then we can share those and contribute back to other communities,” he said.
# # #
The ARM Climate Research Facility is a national scientific user facility funded through the U.S. Department of Energy’s Office of Science. The ARM Facility is operated by nine Department of Energy national laboratories, including Oakridge National Laboratory, which manages the ARM Data Archive.