Technical Note: Impact of nonlinearity on changing the a priori of trace gas profiles estimates from the Tropospheric Emission Spectrometer (TES)

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés. Technical Note: Impact of nonlinearity on changing the a priori of trace gas profiles estimates from the Tropospheric Emission Spectrometer (TES) S. S. Kulawik, K. W. Bowman, M. Luo, C. D. Rodgers, L. Jourdain


Introduction
Optimal estimation is a powerful technique for performing atmospheric retrievals because of its capability to characterize errors and sensitivity (Rodgers, 2000;Bowman et al., 2006). This characterization allows data to be assimilated into chemistry and transport models (Jones et al., 2003) compared to other datasets (Rodgers and Con-used TES data linearly converted to a uniform prior include Zhang et al. (2006)  PITT) carbon monoxide results and explores the influence of the a priori. MOPITT processing currently uses a uniform prior to reduce artefacts arising from the prior and maximize the impact of the satellite data (Deeter et al., 2003).

EGU
A retrieved profile can be expressed as a first order expansion in (x − x a ) (Rodgers, 2000;Bowman et al., 2002): where x a ,x, and x are the prior, retrieved, and true profile state in log(volume mixing ratio (VMR)), A is the averaging kernel which describes the sensitivity of the retrieval to 5 the true state, and ε represents the error resulting from spectral noise, spectroscopic errors, cross-state error, and inaccuracies of non-retrieved species, as discussed in Worden et al. (2004). Adjustment to a new prior can be done using the following equation (Rodgers and Connor, 2003): where x a and x a are the original and new priors, respectively,x is the original retrieved value, andx is the retrieved value with the new prior. Equation (2) shows that when averaging kernel matrix, A, is unity then changes to the prior have no effect on the retrieved value. Conversely when the averaging kernel matrix is zero, Eq. (1) shows 15 that the retrieved state is equal to the prior. The averaging kernel is almost always somewhere in between these two extremes for atmospheric retrievals. Equation (1) assumes not worse than moderate non-linearity between the retrieved state and the true state while Eq. (2) assumes not worse than moderate non-linearity between the two retrieved states (Rodgers 2000). As a consequence, the averaging 20 kernel derived from a non-linear optimal retrieval with a priori, x a , should be sufficiently close to an averaging kernel derived from a non-linear optimal retrieval with a priori, x a . This linearity assumption is tested with a day's worth of TES data. For non-linear optimal estimates, the initial guess used in the minimization does not affect the solution as long as that solution represents the global minimum. On the other hand, if a local Introduction

EGU
To compare datasets quantitatively, histograms were made of the fractional differences defined as: Sincex represents Log(VMR), a value of 0.10 for the fractional difference indicates a 10% difference. 5 We also plot differences between (SSC-SU) versus the amount of change in the prior, which shows whether there is a breakdown in the accuracy of the results if changes to the prior are too large, and shows whether changes in the prior introduce biases in the result. Linear regression is used to calculate the slope of differences between (SSC-SU) versus the change in the prior. 10 Finally, averaging kernels at the result state are compared between the SSC and SU datasets to see if the reported degrees of freedom are consistent when the prior is swapped. This gives an indication of the relative Jacobian strengths, and whether the error analysis is cross-applicable.

15
A TES global survey (Run ID 2147) consisting of 1152 globally distributed targets from 20-21 September 2004 was run for three different configurations for the prior and initial guess, as described in the methods section. Following the non-linear retrievals, the standard retrieval dataset (SS) was converted to the fixed prior dataset (SSC) using Eq. (2). 20 Figures 1 and 2 show the initial and retrieved values at 681 hPa for ozone and carbon monoxide, respectively, for datasets SS, SU, and SSC. The TES target locations are shown with white +'s and interpolation is done between the TES targets. The TES standard prior for both figures (panel a) is taken from a climatological run of the MOZART-3 model binned by 60 degrees longitude, and 10 degrees latitude. For the ozone prior, 25 shown in Fig. 1,  EGU and an enhanced band from South America through southern Africa to Australia (the biomass burning region (discussed iiin Bowman et al., 2007)), and a minimum is seen north of Australia. The standard retrieval shown in Fig. 1b represents these same patterns with a marked enhancement in the biomass burning region. The constant prior cases (panels c and d) agree remarkably well with each other indicating that the linearly 5 converting the prior is valid throughout most of the data. The features in panels c and d can be confidently attributed to the TES data without preconceptions introduced by the prior; however large differences between panels b and c or d indicate a dependence on the prior rather than the data. The absence or presence of particular points passing quality flags can cause minor changes in the three different results. Most of ozone Figure 2 shows the same plots as in Fig. 1, for carbon monoxide. The carbon monoxide prior (Fig. 2a) indicates enhancement over South America and southern Africa (in the biomass burning region), north of 40 N, and over India and southeast Asia. The standard retrieval Fig. 2b displays marked enhancement over the prior in eastern South America and western sub-Sahara Africa, and in eastern Asia. The uniform prior results, 20 panels c and d, show good agreement with each other. The East Asia enhancement is present but muted and the pattern and values in the biomass burning region are very similar between panels b, c, and d, however the CO enhancement poleward of 40 N is markedly reduced in c and d indicating that TES retrievals have less sensitivity in those regions.
25 Figure 3 shows global maps of the VMR fractional difference (using Eq. 3) for O 3 and CO at 681 hPa for the SSC and SU datasets. The plots show that outliers occur predominately in the tropics, and to a lesser extent, Antarctica. The pattern may suggest two cloud layers, which occur frequently in the tropics (Zipser, 1969), could contribute ACPD 8,2008 Linearly exchanging the prior vector for TES estimates EGU to the retrieval variation since TES assumes one cloud layer (Kulawik et al., 2006b), however determining correlations between outliers and atmospheric conditions was not explored further in this paper.

Statistical analysis
To quantify differences, statistical analysis was done on the 681 targets which have 5 good quality flags for all three runs (SS (and by extension SSC), SU, and US). The quality flags check for retrieval convergence using thresholds for the radiance residual and mean, maximum allowed changes in the retrieved surface temperature or emissivity, the amount of signal remaining in the residual; or other known issues (Osterman et al., 2007). The quality flags are set to screen out about 80% of the bad cases, but will 10 also screen out perhaps 20% of good cases as well (Osterman et al., 2007). A histogram of the fractional difference between the SSC and SU datasets shows the overall accuracy of changing the prior using Eq. (2) vs. using a uniform prior in the non-linear retrieval. From this histogram several relevant quantities can be calculated: (1) the fraction of the targets are within 5% of each other, (2) the fractional difference 15 that encompasses 95% of the targets, and (3) the standard deviation of the fractional difference.

Results for ozone
In Fig. 4, a histogram of the VMR fractional difference, using Eq. (3), is shown comparing dataset SSC (the standard retrieval converted to a uniform prior using Eq. (2) to SU 20 (the non-linear retrieval using a uniform prior) at 681, 178, 38 hPa, and over the entire profile. Figure 4 shows that for ozone, 70-80% of the SSC and SU results are within 5% difference. It is not surprising that histogram for the 177.8 hPa pressure level has the widest spread among the 3 pressure levels chosen because ozone at that pressure level has an order of magnitude variability due to the variations in the tropopause 25 height; a globally constant value for ozone between 100-300 hPa is very challenging 1268 ACPD 8,2008 Linearly exchanging the prior vector for TES estimates EGU to the retrieval. Note that the errors introduced by changing the prior are small when compared to the TES reported total error (green dashed line in Fig. 4). In comparison, the VMR fractional difference of the prior had a 1-sigma value of 0.41, 1.08 (i.e. 108%), and 0.16 at 681, 178, and 38 hPa, respectively, indicating significantly more spread in the prior than in the resulting retrieval. The 1-sigma values for the results are shown in Table 1. The histograms in Fig. 4 all show sharp peaks centered near zero but also show more outliers than would be expected from a Gaussian distribution. To determine if the outlying points are a result of a breakdown in the linear transform in Eq. (2) that occurs when the a priori change is too large, the difference (SSC-SU) is plotted versus 10 the change in the prior, averaged over the profile, in Fig. 5. Figure 5 shows no obvious difference between small and large prior changes. In Fig. 5, panel a shows the rms of (SSC-SU), and panel b shows the mean difference, both averaged over the entire profile. For the rms difference, the slope tells whether, on average, larger differences in the prior lead to larger differences in the results. This slope was 0.10. For the mean 15 difference, the slope indicates if the changes in the prior bias the results. The slope of this was found to be −0.02. Together these results mean that less than 10% of the prior's change will end up as unbiased fluctuations in the answer. The lack of bias show that the differences are not a function of the choice of the uniform prior.
To check the whether the outliers in Fig. 4 are a result of converging to a different 20 local minimum, a run was done with a globally uniform initial guess (dataset US). The initial guess is the starting location for the retrieval, which iterates until convergence is reached. Since the initial guess is not included in the cost function, which determines the final solution, it should not affect the retrieval assuming the retrieval gets to the global minimum. However, an initial guess far from true can lead the retrieval to a non-25 global minimum, and systematic errors in the forward model or observed radiance can roughen the error landscape and introduce local minima. A more complete description of TES retrievals is discussed in Bowman et al. (2006). Theoretically, the initial guess does not influence the results (as seen also in Eq. (1) and dataset US should con-Introduction EGU verge to the same answer as the standard retrieval (dataset SS). Differences in these datasets indicate convergence to different local minima, but we do not know whether either has reached a global minimum. The histograms from this run for ozone are shown in Fig. 6. In general, histograms of SS vs. US show a sharper peak and more outliers than the histograms from Fig. 4. For O 3 at 681 hPa, for example, 17% of tar-5 gets change greater than the TES reported error compared to 2% for results shown in Fig. 4. Figure 7 has all "initial guess outliers" removed, and compares remaining targets for datasets SSC and SU. "Initial guess outliers" are set to be those where the average rms difference over the profile between SS and US were more than 5%, and represent 10 targets that show a tendency to converge to different minima. Results are shown in Fig. 7 for 681 hPa, and correlations shown for the profile standard deviation. In this case, there are significantly fewer outliers (compared to Figs. 4 and 5). The right plot in Fig. 7 shows that the spread in the prior is still about the same, but that the spread in the result is markedly less. This means that the outliers in Figs. 4 and 5 likely result 15 from retrievals converging to different local minima. Table 1 summarizes the results for Figs. 4, 5, and 7 for ozone.
As discussed following Eq. (2), when a retrieval is not sensitive, it will converge to the prior and exchanging the prior will move the retrieval to the new prior, as seen for retrievals poleward of 60 N in Fig. 1. The effects of changing the prior on the most 20 sensitive points is of interest, so statistics were calculated for only those points with a corresponding averaging kernel diagonal value of 0.04 or greater. For 681 hPa, the number of samples dropped from 648 to 290; the bias increased from 0.01 to 0.02, the 1-sigma value increased from 2.0% to 2.7%, the 3-sigma value increased from 15% to 17%, and the fraction within 5% error dropped from 78% to 65%. For 177.8 hPa and 25 38.3 hPa, the changes are smaller, for example for 38.3 hPa the fraction within 5% error dropped from 78% to 72%. However the result that the error is unbiased and smaller than the reported total error still holds true for the most sensitive points. 8,2008 Linearly exchanging the prior vector for TES estimates For TES retrievals, carbon monoxide is retrieved following the retrieval of temperature/water/ozone steps. Consequently, changes to the temperature, surface temperature, or cloud parameters resulting from the uniform ozone prior will propagate into differences in the carbon monoxide step. Swapping only the carbon monoxide, rather 5 than all the species together, may improve on the results shown in this study. Figure 8 shows the histogram of the fractional VMR change for CO at 383 and 681 hPa (note Figs. 8 and 9 do not have initial guess outliers removed). Additionally results are shown for averages over the entire profile. Carbon monoxide shows fewer outliers beyond 10% than found with ozone. Results for CO are summarized in Table 2. In 10 comparison, the VMR fractional difference of the prior had a 1-sigma value of 0.30 and 0.17 at 681 and 381 hPa, respectively, indicating significantly more spread in the prior change than in the resulting retrieval.

Results for methane
Methane is also retrieved following the temperature/water/ozone steps, and changes to 15 the temperature, surface temperature, or cloud parameters resulting from the uniform ozone prior will propagate into differences in the methane step. The results seen in this study are likely to be worse than the results from swapping only the methane. Figure 9 shows results at 287 hPa and for the whole profile, and shows that changing to a uniform prior results in less than a 1% difference in methane for 95% of the cases. 20 Results for methane are summarized in Table 3. In comparison, the VMR fractional difference of the prior had a 1-sigma value of 0.06 at 287 hPa indicating significantly more spread in the prior change than in the resulting retrieval. Introduction EGU 3.1.4 Error analysis differences when changing the prior When one changes to a different prior following the nonlinear retrieval, the error analysis available is the one calculated at the original retrieval. This section determines whether this error analysis is accurate by looking the change in the averaging kernel between runs SS and SU. We compare the total degrees of freedom for signal, and at 5 individual values in the averaging kernel diagonal, through comparisons of the mean values, and at the fractional difference (calculated for values greater than 0.001). For ozone, the mean degrees of freedom for signal (DOF) is 3.80. The mean DOF changes 0.01 between the two runs. The rms difference of the DOF is 0.04, which is about 1%. The mean value of the averaging kernel diagonal between the surface and 10 10 hPa is 0.069. The mean difference between the two runs is 8×10 −5 , and the rms fractional difference of the averaging kernel diagonals are 15%.

Conclusions References
For retrievals in Log(VMR), sensitivity is positively correlated to the VMR (Deeter et al., 2007). We find that retrievals with a 10% increase in the retrieved ozone column density also have about a 0.15 increase in the degrees of freedom, a 4% increase.

15
Since the uniform prior is set to the global mean, this does not cause a biased change between the two runs for this test.
For carbon monoxide, the mean DOF is 1.09, with a mean difference of 0.004 between the two runs. The rms difference is 0.02, or 2%. The mean value of the averaging kernel diagonal between the suface and 10 hPa is 0.039. The mean difference 20 between the two runs is 0.0006, and the rms fractional difference of the averaging kernel diagonals are 22%.
For methane, the mean DOF is 1.27, with a mean difference of 8×10 −6 between the two runs. The rms difference is 0.04, or 3%. The mean value of the averaging kernel diagonal between the suface and 10 hPa is 0.024. The mean difference between the 25 two runs is 0.00003, and the rms fractional difference of the averaging kernel diagonals are 12%.
For all three species, the total DOF varies by less than 3% when the prior is changed, EGU and the individual averaging kernel diagonal values vary by about 20%. This indicates that the error bars and sensitivities may have about a 20% unbiased change for any particular level when the prior is changed, however the total DOF remains fairly impervious to changes in the prior.

5
Linearly converting the prior following a non-linear retrieval is shown to have a minor effect on TES trace gas retrievals as compared to a non-linear retrieval using the desired prior, when compared to the reported total error. Histograms of differences between these two methods show a sharp peak centered near zero with some outliers, especially for ozone. Further analysis of the characteristics of the outliers, and comparisons 10 to retrievals with a uniform initial guess indicates that the many of the outliers result from convergence to a local minimum rather than breakdown of the linear conversion in Eq.
(2). For ozone, the 1-sigma difference is less than 4% for each of three pressure levels studied, and the mean change for all levels is 2.7%. For methane, the 1-sigma change is 0.3% at 287 hPa and 0.3% for the profile average, and for carbon monoxide 15 the 1-sigma change is about 2%. The degrees of freedom comparison between shows a 1-sigma difference of less than 3% for all the species, and shows changes of the averaging kernel diagonal are on the order of 20% for individual levels.          . 9. Statistical comparison for methane between non-linear retrievals using a uniform prior vs. conversion to a uniform prior using Eq.

EGU (a) Variable Initial Guess / prior (b) Standard processing (SS) (c) Converted to uniform prior (SSC) (d) Retrieved w/ uniform prior (SU)
(2). The black line shows the histogram of the Fractional difference using Eq. (3) of SSC-SU for 287 hPa. The red line shows the histogram of the differences in the priors, which show significantly more spread. The upper right panel shows the histogram of the average error for all pressures. The lower right panel shows the difference in the retrieval result vs. the difference in the prior for 287 hPa, and the lower right is the same for the mean difference over the whole profile.