Abstract
Many different methods for statistical data editing can be found in the literature but only few of them are based on robust estimates (for example such as BACON-EEM, epidemic algorithms (EA) and transformed rank correlation (TRC) methods of Béguin and Hulliger). However, we can show that outlier detection is only reasonable if robust methods are applied, because the classical estimates are themselves influenced by the outliers. Nevertheless, data editing is essential to check the multivariate data for possible data problems and it is not deterministic like the traditional micro editing where all records are extensively edited manually using certain rules/constraints. The presence of missing values is more a rule than an exception in business surveys and poses additional severe challenges to the outlier detection. First we review the available multivariate outlier detection methods which can cope with incomplete data. In a simulation study, where a subset of the Austrian Structural Business Statistics is simulated, we compare several approaches. Robust methods based on the Minimum Covariance Determinant (MCD) estimator, S-estimators and OGK-estimator as well as BACON-BEM provide the best results in finding the outliers and in providing a low false discovery rate. Many of the discussed methods are implemented in the R package \({\tt{rrcovNA}}\) which is available from the Comprehensive R Archive Network (CRAN) at http://www.CRAN.R-project.org under the GNU General Public License.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Barnett V, Lewis T (1994) Outliers in statistical data. Wiley, New York
Béguin C, Hulliger B (2004) Multivariate outlier detection in incomplete survey data: the epidemic algorithm and transformed rank correlations. J R Stat Soc Ser B (Stat Methodol) 127(2): 275–294
Béguin C, Hulliger B (2008) The BACON-EEM algorithm for multivariate outlier detection in incomplete survey data. Surv Methodol 34(1): 91–103
Billor N, Hadi AS, Vellemann PF (2000) Bacon: blocked adaptative computationally-efficient outlier nominators. Comput Stat Data Anal 34(3): 279–298
Campbell NA (1989) Bushfire maping using NOAA AVHRR data. Technical report, CSIRO
Cerioli A, Riani M, Atkinson AC (2009) Controlling the size of multivariate outlier tests with the MCD estimator of scatter. Stat Comput 19(3): 341–353
Chambers RL (1986) Outlier robust finite population estimation. J Am Stat Assoc 81: 1063–1069
Copt S, Victoria-Feser MP (2004) Fast algorithms for computing high breakdown covariance matrices with missing data. In: Hubert M, Pison G, Struyf A, Van Aelst S (eds) Theory and applications of recent robust methods, statistics for industry and technology series. Birkhauser, Basel
Croux C, Haesbroeck G (1999) Influence function and efficiency of the minimum covariance determinant scatter matrix estimator. J Multivariate Analy 71: 161–190
De Waal T (2003) Processing of erroneous and unsafe data. PhD thesis, Erasmus University, Rotterdam
De Waal T (2009) Statistical data editing. In: Peffermann D, Rao C (eds) Handbook of statistics 29A. Sample surveys: design, methods and applications. Elsevier B. V., Amsterdam, pp 187–214
Dempster AP, Laird MN, Rubin DB (1977) Maximum likelihood from incomplete data via the EM algorithm. J R Stat Soc Ser B (Stat Methodol) 39: 1–22
Dinges G, Haitzmann M (2009) Modellbasierte Ergänzung der Konjunkturstatistik im Produzierenden Bereich; Darstellung der statistischen Grundgesamtheit im Produzierenden Bereich. Statistische Nachrichten 9:1153–1166. http://www.stat.at/web_de/downloads/methodik/kjp.pdf
Donoho DL (1982) Breakdown properties of multivariate location estimators. Technical report, Harvard University, Boston. http://www-stat.stanford.edu/~donoho/Reports/Oldies/BPMLE.pdf
EUREDIT Project (2004) Towards effective statistical editing and imputation strategies—findings of the Euredit project, vols 1 and 2. EUREDIT consortium. http://www.cs.york.ac.uk/euredit/results/results.html
Eurostat (2008) NACE Rev. 2. Statistical classification of economic activites in the European community. Eurostat, methodologies and working papers, ISBN 978-92-79-04741-1
Fellegi I, Holt D (1976) A systematic approach to automatic edit and imputation. J Am Stat Assoc 71: 17–35
Filzmoser P, Garrett RG, Reimann C (2005) Multivariate outlier detection in exploration geochemistry. Comput Geosci 31: 579–587
Filzmoser P, Maronna R, Werner M (2008) Outlier identification in high dimensions. Comput Stat Data Anal 52(3): 1694–1711
Franklin S, Brodeur M (1997) A practical application of a robust multivariate outlier detection method. In: Proceedings of the survey research methods section. American Statistical Association, pp 186–191. http://www.amstat.org/sections/srms/proceedings
Franklin S, Brodeur M, Thomas S (2000) Robust multivariate outlier detection using Mahalanobis’ distance and Stahel–Donoho estimators. In: ICES II, international conference on establishment surveys II
Granquist L (1990) A review of some macro-editing methods for rationalizing the editing process. In: Proceedings of the statistics Canada symposium, Ottawa, Canada, pp 225–234
Granquist L (1997) The new view on editing. Int Stat Rev 65: 381–387
Hampel FR, Ronchetti EM, Rousseeuw PJ, Stahel WA (1986) Robust statistics, the approach based on infuence functions. Wiley, New York
Hardin J, Rocke DM (2005) The distribution of robust distances. J Comput Graph Stat 14: 910–927
Hidiroglou MA, Lavallée P (2009) Sampling and estimation in business surveys. In: Peffermann D, Rao C (eds) Handbook of statistics 29A, sample surveys: design, methods and applications. Elsevier B. V., Amsterdam, pp 441–470
Huber PJ (1981) Robust statistics. Wiley, New York
Hubert M, Rousseeuw PJ, Vanden Branden K (2005) Robpca: a new approach to robust principal component analysis. Technometrics 47: 64–79
Hubert M, Rousseeuw PJ, van Aelst S (2008) High-breakdown robust multivariate methods. Stat Sci 23: 92–119
Johnson RA, Wichern DW (2002) Applied multivariate statistical analysis. 5th edn. Prentice Hall, New Jersey
Lawrence D, McKenzie R (2000) The general application of significance editing. J Official Stat 16: 243–253
Little RJA, Rubin DB (1987) Statistical analysis with missing data. Wiley, New York
Little RJA, Smith PJ (1987) Editing and imputation for quantitative data. J Am Stat Assoc 82: 58–69
Lopuhaä HP (1999) Asymptotics of reweighted estimators of multivariate location and scatter. Ann Stat 27: 1638–1665
Lopuhaä HP, Rousseeuw PJ (1991) Breakdown points of affine equivariant estimators of multivariate location and covariance matrices. Ann Stat 19: 229–248
Luzi O, De Waal T, Hulliger B, Di Zio M, Pannekoek J, Kilchmann D, Guarnera U, Hoogland J, Manzari A, Tempelman C (2007) Recommended practices for editing and imputation in cross-sectional business surveys. Report
Maronna RA, Yohai VJ (1995) The behaviour of the Stahel-Donoho robust multivariate estimator. J Am Stat Assoc 90: 330–341
Maronna RA, Zamar RH (2002) Robust estimation of location and dispersion for high-dimensional datasets. Technometrics 44: 307–317
Maronna RA, Martin D, Yohai V (2006) Robust statistics: theory and methods. Wiley, New York
R Development Core Team (2009) R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna. http://www.R-project.org/, ISBN 3-900051-07-0
Riani M, Atkinson AC, Cerioli A (2009) Finding an unknown number of multivariate outliers. J R Stat Soc Ser B (Stat Methodol) 71(2): 447–466
Rousseeuw PJ, Leroy AM (1987) Robust Regression and outlier detection. Wiley, New York
Rousseeuw PJ, van Zomeren BC (1990) Unmasking multivariate outliers and leverage points. J Am Stat Assoc 85: 633–651
Rubin DB (1993) Discussion: statistical disclosure limitation. J Official Stat 9: 462–468
Schafer J (1997) Analysis of incomplete multivariate data. Chapman and Hall, London
Schafer JL, Graham JW (2002) Missing data: our view of the state of the art. Psychol Methods 7: 147–177
Stahel WA (1981a) Breakdown of covariance estimators. Research Report 31, ETH Zurich, Fachgruppe für Statistik
Stahel WA (1981b) Robuste schätzungen: Infinitesimale optimalität und schätzungen von kovarianzmatrizen. PhD thesis no. 6881, Swiss Federal Institute of Technology (ETH), Zürich. http://www.e-collection.ethbib.ethz.ch/view/eth:21890
Templ M, Filzmoser P (2008) Visualization of missing values using the R-package VIM. Reserach report cs-2008-1, Department of Statistics and Probability Therory, Vienna University of Technology, Vienna
Todorov V, Filzmoser P (2009) An object oriented framework for robust multivariate analysis. J Stat Softw 32(3):1–47. http://www.jstatsoft.org/v32/i03/
Vanden Branden K, Verboven S (2009) Robust data imputation. Comput Biol Chem 33(1): 7–13
Venables WN, Ripley BD (2003) Modern applied statistics with S. 4th edn. Springer, Berlin
Verboven S, Vanden Branden K, Goos P (2007) Sequential imputation for missing values. Comput Biol Chem 31(5–6): 320–327
Wegman E (1990) Hyperdimensional data analysis using parallel coordinates. J Am Stat Assoc 85: 664–675
Author information
Authors and Affiliations
Corresponding author
Additional information
The views expressed herein are those of the authors and do not necessarily reflect the views of the United Nations Industrial Development Organization.
Rights and permissions
About this article
Cite this article
Todorov, V., Templ, M. & Filzmoser, P. Detection of multivariate outliers in business survey data with incomplete information. Adv Data Anal Classif 5, 37–56 (2011). https://doi.org/10.1007/s11634-010-0075-2
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11634-010-0075-2