Atmospheric Chemistry and Physics www.atmos-chem-phys.net 1680-7316 1680-7324 8 17 2008 10.5194/acp-8-5477-2008 http://www.atmos-chem-phys.net/8/5477/2008/ http://www.atmos-chem-phys.net/8/5477/2008/acp-8-5477-2008.html http://www.atmos-chem-phys.net/8/5477/2008/acp-8-5477-2008.pdf 5477 5487 2008-09-12 Technical Note: Review of methods for linear least-squares fitting of data and application to atmospheric chemistry problems C. A. Cantrell cantrell@ucar.edu National Center for Atmospheric Research Atmospheric Chemistry Division 1850 Table Mesa Drive Boulder, CO 80305, USA The representation of data, whether geophysical observations, numerical model output or laboratory results, by a best fit straight line is a routine practice in the geosciences and other fields. While the literature is full of detailed analyses of procedures for fitting straight lines to values with uncertainties, a surprising number of scientists blindly use the standard least-squares method, such as found on calculators and in spreadsheet programs, that assumes no uncertainties in the <i>x</i> values. Here, the available procedures for estimating the best fit straight line to data, including those applicable to situations for uncertainties present in both the <i>x</i> and <i>y</i> variables, are reviewed. Representative methods that are presented in the literature for bivariate weighted fits are compared using several sample data sets, and guidance is presented as to when the somewhat more involved iterative methods are required, or when the standard least-squares procedure would be expected to be satisfactory. A spreadsheet-based template is made available that employs one method for bivariate fitting. Barker, D. R. and Diana, L. M.: Simple method for fitting data when both variables have uncertainties, Am. J. Phys., 42, 224–227, 1974. Bevington, P. R.: Data reduction error analysis for the physical sciences, McGraw-Hill Book Company, New York, 1969. Borcherds, P. H. and Sheth, C. V.: Least squares fitting of a straight line to a set of data points, Eur. J. Phys. 16, 1, 204–210, 1995. Brauers, T. and Finlayson-Pitts, B. J.: Analysis of relative rate measurements, Int. J. Chem. Kin., 29, 665–672, 1997. Bruzzone H. and Moreno, C.: When errors in both coordinates make a difference in the fitting of straight lines by least squares, Meas. Sci. Technol., 9, 2007–2011, 1998. Cantrell, C. A., Edwards, G. D., Stephens, S., Mauldin, R. L., Zondlo, M. A., Kosciuch, E., Eisele, F. L., Shetter, R. E., Lefer, B. L., Hall, S., Flocke, F., Weinheimer, A. Fried, E. Apel, Y. Kondo, D. R. Blake, N. J. Blake, I. J. Simpson, A., Bandy, A. R., Thornton, D. C., Heikes, B. G., Singh, H. B., Brune, W. H., Harder, H., Martinez, M., Jacob, D. J., Avery, M. A., Barrick, J. D., Sachse, G. W., Olson, J. R., Crawford, J. H., and Clark, A. D.: Peroxy radical behavior during the Transport and Chemical Evolution over the Pacific (TRACE-P) campaign as measured aboard the NASA P-3B aircraft, J. Geophys. Res., 108(D20), 8797, doi:10.1029/2003JD003674, 2003. Cecchi, G. C.: Error analysis of the parameters of a least-squares determined curve when both variables have uncertainties, Meas. Sci. Technol., 2, 1127–1128, 1991. Chong, D. P.: Comment on "Linear least-squares fits with errors in both coordinates" by B. C. Reed, Am. J. Phys. 57, 642–646, 1989, Am. J. Phys., 59, 472–474, 1991. Chong, D. P.: On the use of least squares to fit data in linear form, J. Chem. Ed., 71, 489–490, 1994. Christian, S. D. and Tucker, E. E.: Analysis with the microcomputer. 5. General least-squares with variable weighting, Am. Lab., 16(2), 18–18, 1984. Christian, S. D., Tucker, E. E., and Enwall, E.: Least-squares analysis: A primer, Am. Lab., 18, 41–49, 1986. Crawford, J., Davis, D., Olson, J., Chen, G., Liu, S., Gregory, G., Barrick, J., Sachse, G., Sandholm, S., Heikes, B., Singh, H., and Blake, D.: Assessment of upper tropospheric HO_x sources over the tropical Pacific based on NASA GTE/PEM data: Net effect on HO_x and other photochemical parameters, J. Geophys. Res., 104(D13), 16 255–16 273, 1999. Dean, R. B. and Dixon, W. J.: Simplified statistics for small numbers of observations, Anal. Chem., 23, 636–638, doi:10.1021/ac60052a025, 1951. Faloona, I., Tan, D., Brune, W. H., Jaeglé, L., Jacob, D. J., Kondo, Y., Koike, M., Chatfield, R., Pueschel, R., Ferry, G., Sachse, G., Vay, S., Anderson, B., Hannon, J., and Fuelberg, H.: Observations of HO_x and its relationship with NO_x in the upper troposphere during SONEX, J. Geophys. Res., 105(D3), 3771–3783, 2000. Fried, A., Crawford, J., Olson, J., Walega, J., Potter, W., Wert, B., Jordan, C., Anderson, B., Shetter, R., Lefer, B., Blake, D., Blake, N., Meinardi, S., Heikes, B., O'Sullivan, D., Snow, J., Fuelberg, H., Kiley, C. M., Sandholm, S., Tan, D., Sachse, G., Singh, H., Faloona, I., Harward, C. N., and Carmichael, G. R.: Airborne tunable diode laser measurements of formaldehyde during TRACE-P: Distributions and box model comparisons, J. Geophys. Res., 108(D20), 8798, doi:10/1029/2003JD003451, 2003. Gonzalez, A. G., Marquez, A., and Fernandez Sanz, J.: An iterative algorithm for consistent and unbiased estimation of linear regression parameters when there are errors in both the $x$ and $y$ variables, Computers Chem., 16(1), 25–27, 1992. Irwin, J. A. and Quickenden, T. I.: Linear least squares treatement when there are errors in both $x$ and $y$, J. Chem. Ed., 60(5), 711–712, 1983. Jones, T. A.: Fitting straight lines when both variables are subject to error. I. Maximum likelihood and least-squares estimation, Math. Geo., 11(1), 1–25, 1979. Kalantar, A. H.: Weighted least squares evaluation of slope from data having errors in both axes, Trends in Anal. Chem., 9(1), 149–151, 1990. Kalantar, A. H.: Kerrich's method for \textity=$\alpha x$ data when both $y$ and $x$ are uncertain, J. Chem Ed., 68(1), 368–370, 1991. Kalantar, A. H.: Straight-line parameters' errors propagated from the errors in both coordinates, Meas. Sci. Technol., 3, 1113–1113, 1992. Kalantar, A. H., Gelb, R. I., and Alper, J. S.: Biases in summary statistics of slopes and intercepts in linear regression with errors in both variables, Talanta, 42(4), 597–603, 1995. Krane, K. S. and Schecter, L.: Regression line analysis, Am. J. Phys., 50(1), 82–84, 1982. Leduc, D. J.: A comparative analysis of the reduced major axis technique of fitting lines to bivariate data, Can. J. For. Res., 17, 654–659, 1987. Lybanon, M.: A better least-squares method when both variables have uncertainties, Am. J. Phys., 52(1), 22–26, 1984a. Lybanon, M.: Comment on "Least squares when both variables have uncertainties", Am. J. Phys., 52(3), 276–278, 1984b. Lybanon, M.: A simple generalized least-squares algorithm, Comp. Geosci., 11, 501–508, 1985. Macdonald, J. R. and Thompson, W. J.: Least-squares fitting when both variables contain errors: Pitfalls and possibilities, Am. J. Phys., 60(1), 66–73, 1992. MacTaggart, D. L. and Farwell, S. O.: Analytical use of linear regression, Part II: Statistical error in both variables, J. AOAC Intl., 75(4), 608–614, 1992. Markovsky, I. and Van Huffel, S.: Overview of total least-squares methods, Sig. Proc., 87, 2283–2302, 2007. Moreno, C., and Bruzzone, H.: Parameters' variances of a least-squares determined straight line with errors in both coordinates, Meas. Sci. Technol., 4, 635–636, 1993. Moreno, C.: A least-squares-based method for determining the ratio between two measured quantities, Meas. Sci. Technol., 7, 137–141,1996. Neri, F., G. Saitta, and S. Chiofalo: An accurate and straightforward approach to line regression analysis of error-affected experimental data, J. Phys. E: Sci. Inst., 22, 215–217, 1989. Neri, F., Patanè, S., and Saitta, G.: Error-affected experimental data analysis: application to fitting procedure, Meas. Sci. Technol., 1, 1007–1010, 1990. Olson, J. R., Crawford, J. H., Chen, G., Fried, A., Evans, M. J., Jordan, C. E., Sandholm, S. T., Davis, D. D., Anderson, B. E., Avery, M. A., Barrick, J. D., Black, D. R., Brune, W. H., Eisele, F. L., Flocke, F., Harder, H., Jacob, D. J., Kondo, Y., Lefer, B. L., Martinez, M., Mauldin, R. L., Sachse, G. W., Shetter, R. E., Singh, H. B., Talbot, R. W., and Tan, D.: Testing fast photochemical theory during TRACE-P based on measurements of OH, HO₂, and CH₂O, J. Geophys. Res., 109, D15S10, doi:10.1029/2003JD004278, 2004. Olson, J. R., Crawford, J. H., Chen, G., Brune, W. H., Faloona, I. C., Tan, D., Harder, H., and Martinez, M.: A reevaluation of airborne HO$_x$ observations from NASA field campaigns, J. Geophys. Res., 111, D10301, doi: 10.1029/2005/JD006617, 2006. Orear, J.: Least squares when both variables have uncertainties, Am. J. Phys., 50, 912–916, 1982; Erratum, Am. J. Phys., 52, 278–278, 1984. Pasachoff, J. M.: Applicability of least-squares formula, Am. J. Phys., 48(10), 800–800, 1980. Pearson, K.: On lines and planes of closet fit to systems of points in space, Philos. Mag., 2(2), 559–572, 1901. Press, W. H., Teukolsky, S. A., Vetterling, W. T., and Flannery, B. P.: Numerical Recipes in C, 2nd ed., Cambridge University Press, 1992a. Press, W. H., Teukolsky, S. A., Vetterling, W. T., and Flannery, B. P.: Numerical Recipes in Fortran, 2nd ed., Cambridge University Press, 1992b. Reed, B. C.: Linear least-squares fits with errors in both coordinates, Am. J. Phys., 57(3), 642–646, 1989; Erratum, Am J. Phys., 58(2), 189–189, 1990. Reed, B. C.: Linear least-squares fits with errors in both coordinates. II: Comments on parameter variances, Am. J. Phys., 60(1), 59–62, 1992. Riu, J. and Rius, F. X.: Univariate regression models with errors in both axes, J. Chemometrics, 9, 343–362, 1995. Sheth, C. V. A.: Ngwengwe, and P. H. Borcherds, Least squares fitting of a straight line to a set of data points: II: Parameter variances, Eur. J. Phys., 17(2), 322–326, 1996. Squire, P. T.: Comment on "Linear least-squares fits with errors in both coordinates" by B. C. Reed, Am. J. Phys. 57, 642–646, 1989, Am. J. Phys., 58(12), 1209–1209, 1990. Titterington, D. M. and Halliday, A. N.: On the fitting of parallel isochrones and the method of maximum likelihood, Chem. Geol. 26, 183–195, 1979. Williamson, J. H.: Least-squares fitting of a straight line, Can. J. Phys., 46, 1845–1847, 1968. York, D.: Least-squares fitting of a straight line, Can. J. Phys., 44, 1079–1086, 1966. York, D.: Least squares fitting of a straight line with correlated errors, Earth and Planet. Sci. Lett., 5, 320–324, 1969. York, D., Evensen, N. M., López Martinez, M., and De Basabe Delgado, J.: Unified equations for the slope, intercept, and standard errors of the best straight line, Am J. Phys., 72(3), 367–375, 2004.