Adaptive R Configuration
and Importance
Introduction
In data assimilation, the observation error covariance matrix R is a key component that determines the confidence assigned to observations. In particular, representation error depends on the continuum state of the geophysical system and therefore exhibits state- and time-dependent characteristics. Because of this dependence, representation error introduces spatial correlations into the observation-error statistics.
The observation-error covariance matrix is generally defined as
R = E[(εᵒ)(εᵒ)ᵀ]
εᵒ : observation-error vector
E[·] : expectation operator
R : observation-error covariance matrix
1. Why the Specification of R Matters
-
Problems Caused by Underestimating Representation Error
When representation error is underestimated, the analysis becomes excessively fitted to signals that the model effectively treats as noise. As a result, unresolved-scale variability is incorporated into the analysis, producing subgrid-scale structures in the analysis field that cannot be physically interpreted by the model.
-
Problems Caused by Overestimating Representation Error
When representation error is overestimated, useful observational information is unnecessarily discarded. Observations receive excessively low weights, reducing the observational influence on the analysis more than necessary.
-
Problems When Correlated Errors Are Assumed to Be Uncorrelated
In many operational systems, correlated observation errors are treated as uncorrelated for computational convenience. The resulting effects are often compensated for by inflating the observation-error variances. However, variance inflation cannot correctly represent the true correlation structure. Moreover, beyond a certain threshold, increasing observation density no longer improves analysis accuracy, because the neglected correlations limit the effective information content of the observations.
Underestimating representation error makes the analysis overfit noise and produce unrealistic small-scale structures. Overestimating it weakens the impact of useful observations. Ignoring correlated observation errors also limits analysis accuracy, even when observation density increases.
2. Covariance Structure in the Kalman Filter
In the Kalman filter, the background-error covariance matrix represents uncertainty in the model state estimate.
To account for both resolved and unresolved scales, an augmented covariance structure is introduced:
-
Bʳʳ: covariance of resolved-scale errors
-
Bᵘᵘ: covariance of unresolved-scale errors
-
Bʳᵘ, Bᵘʳ: cross-covariances between resolved and unresolved scales
The Kalman gain in the traditional filter formulation is Kʳ = (HBʳʳ)ᵀ O⁻¹
where the observation-space covariance matrix is O = H(Bʳʳ)Hᵀ + Hᵘ(Wᵘᵘ)(Hᵘ)ᵀ + E
The three terms represent:
-
H(Bʳʳ)Hᵀ: Resolved-scale background-error covariance
-
Hᵘ(Wᵘᵘ)(Hᵘ)ᵀ: Representation error associated with unresolved scales
-
E: Instrument-error covariance
3. Impact of R Configuration
As indicated by the Kalman gain equation:
-
When R decreases, the observational weight increases.
-
When R increases, greater weight is assigned to the background state.
Therefore, the specification of R directly controls the relative trust assigned to observations versus the model, the spatial structure of the analysis field, forecast stability, and the effective utilization of high-resolution observations in data assimilation systems.
In the Kalman filter, covariance structures represent uncertainties from both resolved and unresolved scales. The observation-error covariance R determines the balance between observations and the model background: smaller R increases observational influence, while larger R favors the model.
4. Benefits of Implementing Adaptive R
1) Improved Analysis Accuracy
Several studies have demonstrated that incorporating correlated observation errors leads to more accurate analyses.
-
Healy and White, Stewart, and Stewart et al. showed that accounting for correlated observation errors improves analysis quality, even when approximate covariance representations are used.
-
Seaman demonstrated that including spatial correlations improves the accuracy of gradients in the analyzed observation field.
-
Rainwater et al. showed improvements at the smallest scales resolved by numerical weather prediction (NWP) models.
2) Improved Forecast Skill Scores
Adaptive and correlated observation-error covariance formulations have also been shown to improve forecast skill.
-
Weston et al. demonstrated that incorporating inter-channel error correlations for IASI observations in the Met Office forecasting system improved forecast skill scores.
-
Bormann et al. showed that updating the IASI observation-error covariance matrix at ECMWF led to measurable forecast improvements.
Both studies found that including inter-channel correlations modifies observational weights in a situation-dependent manner:
-
when departures project strongly onto directions associated with large eigenvalues, the observational weight is relatively reduced,
-
whereas projections onto higher-order eigenvalue directions increase the observational weight.
This demonstrates that correlated R matrices dynamically redistribute observational influence depending on the structure of the observation departures.
5. Practical Issues in Implementation
Computational Cost and Convergence Issues
A major challenge in implementing fully correlated observation-error covariance matrices is the computational cost associated with storing and processing large covariance matrices.
Several practical approaches have been proposed to reduce computational expense:
-
assuming a block-diagonal structure corresponding to different observing instruments,
-
neglecting spatial correlations while retaining inter-channel correlations only
(as in the operational Met Office system; Weston, 2011; 2014), -
using approximate forms of the covariance matrix or its inverse
(Fisher, 2005; Healy & White, 2005; Stewart et al., 2008; 2013).