Optimal Estimation retrievals and their uncertainties: What every atmospheric scientist should know
Submitter:
Maahn, Maximilian — Leipzig University
Turner, David D. — NOAA Earth System Research Laboratory
Area of research:
Cloud Processes
Journal Reference:
Science
This educational paper gives an overview on developing Bayesian Optimal Estimation retrievals, which are a key method for converting indirect remote-sensing measurements into atmospheric variables. We present two sample retrievals based on microwave radiometers and radars including all data and retrieval code. Using a special website, the readers can explore and modify the retrieval by themselves interactively.
Impact
By guiding the reader through all required steps for developing a retrieval, discussing the most important sources for uncertainties, and providing extensive documentation and sample code, we enable the readers to develop their own Optimal Estimation retrievals quickly. These can be applied, e.g., to the extensive ARM remote-sensing data sets.
Summary
Remote-sensing instruments are heavily used to provide observations for both the operational and research communities, but retrieval algorithms are necessary to convert the indirect observations into the variable of interest. It is critical to be aware of the underlying assumptions made by many retrieval algorithms, including that the retrieval problem is often ill-posed and that there are various sources of uncertainty that need to be treated properly. In short, the retrieval challenge is to invert a set of noisy observations to obtain estimates of atmospheric quantities. The problem is often complicated by imperfect forward models, imperfect prior knowledge, and by the existence of non-unique solutions. Optimal Estimation (OE) is a widely used physical retrieval method that combines measurements, prior information, and the corresponding uncertainties based on Bayes’ theorem to find an optimal solution for the atmospheric state. Here, we provide a novel Python library to illustrate the use of OE for inverse problems in atmospheric sciences. We introduce two example problems: How to retrieve drop size distribution parameters from radar observations and how to retrieve the temperature profile from ground-based microwave sensors. Using these examples, we discuss common pitfalls, how the various error sources impact the retrieval, and how the quality of the retrieval results can be quantified. The examples can be explored interactively with a web browser.
Optimal estimation (OE) principle: The ellipses show prior state (left) and measurement (right) uncertainty. The iterative process starts with applying the forward operator F to the first guess of the atmospheric state x (here x_0 = x_a). Based on the difference of F(x_a) to the observations y_obs (which is close to, but not equal to an ideal measurement y_truth representing x_truth), x_1 is obtained (which requires inverting F). This is repeated until the retrieval converges to a solution x_3=x_op that is close to the true state x_truth.