Premium
Statistical inference in massive data sets
Author(s) -
Li Runze,
Lin Dennis K.J.,
Li Bing
Publication year - 2012
Publication title -
applied stochastic models in business and industry
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.413
H-Index - 40
eISSN - 1526-4025
pISSN - 1524-1904
DOI - 10.1002/asmb.1927
Subject(s) - estimator , inference , statistical inference , computer science , data set , set (abstract data type) , asymptotic distribution , data mining , normality , population , algorithm , asymptotic analysis , statistics , mathematics , artificial intelligence , demography , sociology , programming language
Analysis of massive data sets is challenging owing to limitations of computer primary memory. In this paper, we propose an approach to estimate population parameters from a massive data set. The proposed approach significantly reduces the required amount of primary memory, and the resulting estimate will be as efficient if the entire data set was analyzed simultaneously. Asymptotic properties of the resulting estimate are studied, and the asymptotic normality of the resulting estimator is established. The standard error formula for the resulting estimate is proposed and empirically tested; thus, statistical inference for parameters of interest can be performed. The effectiveness of the proposed approach is illustrated using simulation studies and an Internet traffic data example. Copyright © 2012 John Wiley & Sons, Ltd.