Getting top data from hundreds of instruments requires tip-top technical oversight
For a few days this July, University of Wisconsin-Madison research scientist Jonathan Gero was on Graciosa Island in the Azores to install an Atmospheric Emitted Radiance Interferometer (AERI), a device that detects infrared radiance propagating downward from the atmosphere. He was there on behalf of the Atmospheric Radiation Measurement (ARM) Climate Research Facility, which operates the Eastern North Atlantic (ENA) atmospheric observatory on Graciosa.
Among ARM’s myriad staff members are something like 40 “instrument mentors,” 20 assistants, and assorted developers that watch over about 400 instruments supporting the Facility’s mission of climate observation.
Gero, who like most mentors is part-time, nonetheless keeps track of nine of the AERI devices worldwide. (ARM has three fixed and three mobile observation facilities and an aerial facility.) The job is a big one: Just an outline of roles and responsibilities of instrument mentors fills five closely spaced pages.
“I feel like a frog in a blender sometimes,” said ARM Aerosol Observing System (AOS) mentor Stephen Springston, an atmospheric chemist at Brookhaven National Laboratory and one of just a few full-time mentors. Watching over his suite of 16 instruments, some of them designed just for airborne campaigns, he added, “is my first love.”
Argonne National Laboratory’s Jenni Kyrouac oversees all of the surface meteorological instruments, some of them the size of a soup bowl, at all of the ARM observatories. How many instruments? She answers with a laugh. “I would have to look up the number.”
Remote World Windows
Working largely at their office laptops, mentors monitor how an instrument is performing in the field. They assess and troubleshoot problems. They write and update operational manuals, some more than 400 pages long. And they work on data quality algorithms.
Instrument Team Coordinator Doug Sisterson, from his office at Argonne, works with the ARM Data Quality Office to reduce the amount of time between the discovery of an instrument problem, and the reporting of questionable data to end users. “Mentors have the final word in data quality,” he said. “They are our final set of eyes.”
Technologist Victor Morris agreed that mentors take the last data quality step. (He watches over a suite of cloud property instruments from Pacific Northwest National Laboratory.) But mentors also “take one of the first steps,” said Morris. When data are still raw, the right expert can spot problems that escape operators based at a fixed or mobile site.
To be an instrument mentor, “you have to be a specialist,” agreed Argonne’s Michael Ritsche, a mentor from 2000 to 2014 and now assistant site manager at ARM’s 55,000-square-mile Southern Great Plains atmospheric observatory. A lot of datastreams in every day, and mentors have to see flaws in the data as it flies past—then reach out to get it fixed.
That can be even years later, said Springston, long after data have been archived. The responsibility for good data starts at the beginning, with pulses of information from instruments, and never ends. He called the goal of good data “keeping the lights blinking.”
Looking for Trouble
All the while the data lights are blinking, things can go wrong in a system that produces data on an industrial scale, and that has so many instruments that have to run on their own in sometimes extreme conditions. “We have problems the (lab) researchers have never seen,” said Springston. A site might have to run at 50 hertz instead of the 60 hertz standard in the Americas. Spare parts might be six weeks away. A site might be extremely hot or cold.
Troubleshooting instruments hundreds or thousands of miles away is “a third to half of our jobs,” said Springston. He calls himself “a MacGyver” for the resourcefulness that fixing things from a distance sometimes requires, as when an instrument breaks, or fails as it ages, or for those times when nature interferes. (Every instrument mentor learns to cope with bee nests, invading rodents, pecking birds, or cows that use field instruments as rubbing posts.)
Kyrouac, an atmospheric instrument specialist and a full-time mentor, sums up her typical day at the keyboard: “The problems come to me before I can go looking for them.”
Mentors watch for emails from the field, alerting them to breakdowns or anomalies in the data. They monitor the streams of raw data, oversee site documentation, upgrade instrument systems, juggle multiple time zones, document instrument problems, and write reports, which takes up “most of my time,” she said.
All that oversight requires eyes on the ground for mentors, who rely on site technicians for daily checks on the sophisticated instrumentation needed to pull in climate data. At the remote North Slope of Alaska (NSA) atmospheric observatory, one mile south of the Arctic Ocean, facility manager Walter Brower is a 17-year veteran of keeping watch on gear housed in weather-hardened instrument shelters.
“We’ve taken over a lot of the troubleshooting,” as well as calibration, said the native Iñupiaq Eskimo, who checks email traffic and juggles time zone challenges to keep in touch with mentors. He fills out a seven-day weekly schedule with a father-son team from the same area. “We stay home only one or two days a year” because of severe weather, Brower added, when the wind chill might dip to 80 degrees below zero.
Checking on instruments at NSA includes trips to Very Point, 6 miles from the main research site. A tower there holds an eddy correlation (ECOR) flux measurement system that is regularly “inspected” by local polar bears, said Brower. They knock down cables, which Arctic foxes then snack on. “In all, the days are diverse,” he said, from email instrument troubleshooting to track rig snow removal. “It makes the days fun.”
In the Lower 48, at a snow-free university or national laboratory desk, being an instrument mentor can be a grind, but there are satisfactions. Kyrouac listed some that echo other mentors: problem-solving and the joy of finding solutions; working with people at all the sites, “even though I haven’t met a lot of them,” she said. “Technology today makes the world so small” and then there is just “being able to contribute” good data.
Springston also likes getting data “with a known reliability to them,” free to anyone—from the scientist publishing important papers to “some amateur in an internet café in Bangalore,” he said. “ARM is more transparent than any organization I can think of. Everything is available. That’s a lofty goal.”
And he expanded on the idea of being able to contribute to climate science. “It’s one of the most important issues in the world,” said Springston.
Red Sky, White Bears
During Gero’s July visit to the Azores, within three days he got his latest AERI installed, calibrated, and taking in operational data. But time in the field is the exception. Leaving the computer keyboard is only for very practical reasons, such as training on-site personnel and overseeing an instrument’s initial installation and calibration. Travel is infrequent. “We don’t want them spending day-to-day down in the weeds,” said Sisterson.
When travel happens, a stay might be a few days or many weeks (in the case of a field campaign). Morris recalls living for a month in Niger, where the sky was red with dust from the nearby Sahara. Visiting an ARM site at Oliktok Point, Alaska, he marveled at the stark and extreme landscape bordering the Arctic Ocean. Before going outdoors, he said, “you had to make sure there were no polar bears around the corner.”
Kyrouac watched a whale being beached by native hunters in Barrow. Ritsche has done instrument work in Alaska, Australia, China, the Azores, on board two ships traversing the Pacific, and in Finland, where deep boreal forests yielded data on biogenic aerosols, but at the cost of being there in January. “It was tough,” he said, but whether at the keyboard or at sea or in the cold, mentors “have to roll with the punches.”
Though travel is not a big part of an instrument mentor’s routine duties, it is nevertheless essential to understanding ground conditions and to siting instruments accordingly, said Springston, a 25-year field researcher. “The mentors bring something (to the site) the operators can’t provide.”
Meanwhile, site operators bring a special quality to watching over ARM instruments: being there. Carlos Sousa, lead technician at the ENA site and a six-year veteran of ARM facilities all over the world, is outside three times a day cleaning lenses and laser ports with Kim wipes and distilled water—a task that allows a close look at instruments. (On the island, he oversees all the instruments, from scanning radars to weighing bucket rain gauges.) Back inside, by monitoring email and datastreams, he said, “we can see if something is going wrong.”
Sousa and other site operators offer the gift of proximity. That includes, in his case, logging onto instrument monitors at 3 a.m. some nights. Thanks to email and the phone, he said, communication with mentors is close. “We have a good relationship with them all.”
Collaboration and Credibility
In the Azores and elsewhere, Gero is responsible for mentoring the AERI, a workhorse ARM instrument that has generated data stretching back 20 years. Observational data of that continuity and depth, the hallmark of the ARM Facility, is used by scientists to improve model simulations of how cloud and radiative processes impact climate change.
That makes ARM data critically important, along with the instrument mentors, who are the final arbiters of that data. The job they do behind the scenes, said Sisterson, “keeps our credibility high.”
Mentoring instruments also illustrates a pillar of the ARM organization going back to its founding more than 25 years ago: a sense of community and collaboration. In 1990, nine of the U.S. Department of Energy’s 17 national laboratories were approached to take part in ARM; all nine still collaborate today.
That collaborative philosophy was a first-ever break from a competitive culture in which labs competed for research funds. Sisterson wrote of the cooperating labs in an ARM history monograph this spring, “Only together did they have the required expertise.”
Today, the front lines of that collaboration include the multi-lab cohort of instrument mentors toiling to keep ARM data continuous, sharp, and transparent. “If you discover something,” Springston said, “you tell your colleague.”
# # #
The ARM Climate Research Facility is a national scientific user facility funded through the U.S. Department of Energy’s Office of Science. The ARM Facility is operated by nine Department of Energy national laboratories.