Atmospheric Chemistry and Physics www.atmos-chem-phys.net 1680-7316 1680-7324 9 12 2009 10.5194/acp-9-4197-2009 http://www.atmos-chem-phys.net/9/4197/2009/ http://www.atmos-chem-phys.net/9/4197/2009/acp-9-4197-2009.html http://www.atmos-chem-phys.net/9/4197/2009/acp-9-4197-2009.pdf 4197 4206 2009-06-24 Gap filling and noise reduction of unevenly sampled data by means of the Lomb-Scargle periodogram K. Hocke klemens.hocke@iap.unibe.ch N. Kämpfer Institute of Applied Physics and Oeschger Centre for Climate Change Research, University of Bern, Switzerland The Lomb-Scargle periodogram is widely used for the estimation of the power spectral density of unevenly sampled data. A small extension of the algorithm of the Lomb-Scargle periodogram permits the estimation of the phases of the spectral components. The amplitude and phase information is sufficient for the construction of a complex Fourier spectrum. The inverse Fourier transform can be applied to this Fourier spectrum and provides an evenly sampled series (Scargle, 1989). We are testing the proposed reconstruction method by means of artificial time series and real observations of mesospheric ozone, having data gaps and noise. For data gap filling and noise reduction, it is necessary to modify the Fourier spectrum before the inverse Fourier transform is done. The modification can be easily performed by selection of the relevant spectral components which are above a given confidence limit or within a certain frequency range. Examples with time series of lower mesospheric ozone show that the reconstruction method can reproduce steep ozone gradients around sunrise and sunset and superposed planetary wave-like oscillations observed by a ground-based microwave radiometer at Payerne. The importance of gap filling methods for climate change studies is demonstrated by means of long-term series of temperature and water vapor pressure at the Jungfraujoch station where data gaps from another instrument have been inserted before the linear trend is calculated. The results are encouraging but the present reconstruction algorithm is far away from being reliable and robust enough for a serious application. Adorf, H.-M.: Interpolation of irregularly sampled data series – a survey, in: Astronomical data analysis software and systems IV, edited by: Shaw, R. E., Payne, H. E., and Hayes, J. J. E., ASP conference series, 77, 1–4, 1995. Altadill, D., Apostolev, E M., Jacobi, C., and Mitchell, N J.: Six-day westward propagating wave in the maximum electron density of the ionosphere, Ann. Geophys., 21, 1577–1588, 2003. Baluev, R V.: Assessing the statistical significance of periodogram peaks, Mon. Not. R. Astron. Soc., 385, 1279–1285, 2008. Bengtsson, L., Hagemann, S., and Hodges, K I.: Can climate trends be calculated from reanalysis data?, J. Geophys. Res., 109, D11111, \doi10.1029/2004JD004536, 2004. Bretthorst, G.: Generalizing the Lomb-Scargle periodogram, in: Maximum Entropy and Bayesian Methods in Science and Engineering, edited by: Mohammad-Djafari, A., AIP conference proceedings, USA, 2001a. Bretthorst, G.: Generalizing the Lomb-Scargle periodogram – The nonsinusoidal case, in: Maximum Entropy and Bayesian Methods in Science and Engineering, edited by: Mohammad-Djafari, A., AIP conference proceedings, USA, 2001b. Calisesi, Y.: Monitoring of stratospheric and mesospheric ozone with a ground-based microwave radiometer: data retrieval, analysis, and applications, Ph.D. thesis, http://www.iap.unibe.ch/publications/, Philosophisch-Naturwissenschaftliche Fakultät, Universität Bern, Bern, Switzerland, 2000. Calisesi, Y.: The Stratospheric Ozone Monitoring Radiometer SOMORA: NDSC Application Document, IAP Research Report 2003-11, http://www.iap.unibe.ch/publications/, Institut für angewandte Physik, Universität Bern, Bern, Switzerland, 2003. Calisesi, Y., Soebijanta, V T., and van Oss, R O.: Regridding of remote soundings: Formulation and application to ozone profile comparison, J. Geophys. Res., 110, D23306, \doi10.1029/2005JD006122, 2005. Dumitru, M C., Hocke, K., Kämpfer, N., and Calisesi, Y.: Comparison and validation studies related to ground-based microwave observations of ozone in the stratosphere and mesosphere, J. Atmos. Solar Terr. Phys., 68, 745–756, 2006. Ford, E. A K., Aruliah, A L., Griffin, E M., and McWhirter, I.: Thermospheric gravity waves in Fabry-Perot interferometer measurements of the 630.0 nm OI line, Ann. Geophys., 24, 555–566, 2006. Funatsu, B. M., Claud, C., Keckhut, P., and Hauchecorne, A.: Cross-validation of Advanced Microwave Sounding Unit and lidar for long-term upper-stratospheric temperature monitoring, J. Geophys. Res., 113, D23108, doi:10.1029/2008JD010743, 2008. Ghil, M., Allen, M R., Dettinger, M D., Ide, K., Kondrashov, D., Mann, M E., Robertson, A W., Saunders, A., Tian, Y., Varadi, F., and Yiou, P.: Advanced spectral methods for climatic time series, Rev. Geophys., 40, 1003, \doi10.1029/2000RG000092, 2002. Hall, C M., Nozawa, S., Meek, C E., Manson, A H., and Luo, Y.: Periodicities in energy dissipation rates in the auroral mesosphere/lower thermosphere, Ann. Geophys., 21, 787–796, 2003. Hernandez, G.: Time series, periodograms, and significance, J. Geophys. Res., 104, 10355–10368, 1999. Hocke, K.: Phase estimation with the Lomb-Scargle periodogram method, Ann. Geophys., 16, 356–358, 1998. Hocke, K. and Kämpfer, N.: Bispectral Analysis of the Long-Term Recording of Surface Pressure at Jakarta, J. Geophys. Res., 113, \doi10.1029/2007JD009356, 2008. Hocke, K., Kämpfer, N., Feist, D G., Calisesi, Y., Jiang, J H., and Chabrillat, S.: Temporal variance of lower mesospheric ozone over Switzerland during winter 2000/2001, Geophys. Res. Lett., 33, L09801, \doi10.1029/2005GL025496, 2006. Hocke, K., Kämpfer, N., Ruffieux, D., Froidevaux, L., Parrish, A., Boyd, I., von Clarmann, T., Steck, T., Timofeyev, Y., Polyakov, A., and Kyrölä, E.: Comparison and synergy of stratospheric ozone measurements by satellite limb sounders and the ground-based microwave radiometer SOMORA, Atmos. Chem. Phys., 7, 4117–4131, 2007. Horne, J H. and Baliunas, S L.: A prescription for period analysis of unevenly sampled time series, Astrophys. J., 302, 757–763, 1986. Kondrashov, D. and Ghil, M.: Spatio-temporal filling of missing points in geophysical data sets, Nonlin. Processes Geophys., 13, 151–159, 2006. Lomb, N R.: Least-squares frequency analysis of unequally spaced data, Astrophys. Space Sci., 39, 447–462, 1976. Nozawa, S., Brekke, A., Maeda, S., Aso, T., Hall, C M., Ogawa, Y., Buchert, S C., Röttger, J., Richmond, A D., Roble, R., and Fuji, R.: Mean winds, tides, and quasi-2 day wave in the polar lower thermospshere observed in European Incoherent Scatter (EISCAT) 8 day run data in November 2003, J. Geophys. Res., 110, A12309, \doi10.1029/2005JA011128, 2005. Press, W H., Teukolsky, S A., Vetterling, W T., and Flannery, B P.: Numerical recipes in Fortran, Cambridge University Press, Cambridge, USA, 2 edn., 1992. Rodgers, C D.: Retrieval of atmospheric temperature and composition from remote measurements of thermal radiation, Rev. Geophys. Space Phys., 14, 609–624, 1976. Scargle, J D.: Studies in astronomical time series analysis. II. Statistical aspects of spectral analysis of unevenly spaced data, Astrophys. J., 263, 1982. Scargle, J D.: Studies in astronomical time series analysis. III. Fourier transforms, autocorrelation and cross-correlation functions of unevenly spaced data, Astrophys. J., 343, 874–887, 1989. Schneider, T.: Analysis of incomplete climate data: Estimation of mean values and covariance matrices and interpretation of missing values, J. Climate, 14, 853–671, 2001.