Stay in the Know: Powerful Data Quality Tools & Alerts at your Fingertips!

Authors

Sockol, Alyssa Jordan — University of Oklahoma
Peppler, Randy A. — University of Oklahoma
Kehoe, Kenneth — University of Oklahoma
Godine, Corey — University of Oklahoma
Li, Lishan — Cooperative Institute for Severe and High-Impact Weather Research and Operations (CIWRO)
Schneider, James Rice — Oak Ridge National Laboratory

Category

ARM infrastructure

The ARM Data Quality Office (DQO) is an ARM user facility program located at the Cooperative Institute for Severe and High-Impact Weather Research and Operations (CIWRO) at the University of Oklahoma. The main goal of the ARM DQO is to assess and characterize the quality of ARM data to provide the best possible data to end users. This poster showcases the tools and alerts currently offered by the DQO, which provide quick access to ARM data for analysis as well as customizable notifications to flag potential data quality issues within ARM infrastructure. These tools are available to both ARM personnel and end users, ensuring broad accessibility for monitoring and analysis. The poster also explores the future direction of the DQO, highlighting the development of advanced AI algorithms to enable continuous quality oversight of ARM data.

The DQO has developed a suite of web-based tools to help visualize instrument data and quality control (QC) information. These web-tools include DQ-Explorer, DQ-Plotbrowser, and DQ-Zoom. DQ-Explorer is designed to display instrument data and QC information through DQO-created plots and metrics tables, providing a clear overview of data quality. DQ-Plotbrowser offers an efficient way to view plots generated by the DQO’s open-source Python scripts via a user-friendly interface. Finally, DQ-Zoom enables users to dynamically explore ARM data in real-time by directly reading from netCDF data files, allowing for quick, customized visualization. Together, these tools make it possible to gather ARM data quickly and efficiently, while also providing a robust platform for identifying potential instrument problems and ensuring data quality.

In addition to the current tools, the DQO is actively working on advancements through the integration of deep learning algorithms. One key initiative involves developing an AI framework for time series pattern recognition. This framework aims to detect data anomalies that might escape the predefined DQ tests, enabling timely interventions and enhancing both data quality and availability. Meanwhile, we are exploring self-learning methods to efficiently build custom AI models that capture the unique characteristics of individual ARM datastreams and variables—without requiring any manual labeling beforehand. These efforts are expected to lead to new AI-driven data products, including a searchable ARM database, a recommender system for identifying time series patterns of interest, and generative AI techniques to upsample specific data patterns, addressing class imbalance issues. Ultimately, our primary goal is to provide users with AI-ready ARM data, ensuring seamless integration with advanced AI algorithms.

The DQO plays a leading role in maintaining high quality data for the ARM program, and uses these tools, capabilities, and processes to minimize the time required to address instrument problems. Together, these efforts ensure that ARM provides the most reliable and accessible data to end users, enabling them to make informed decisions and drive impactful scientific discoveries.

Lead PI

Peppler, Randy A. — University of Oklahoma