Premium
New ways of specifying data edits
Author(s) -
Petrakos George,
Conversano Claudio,
Farmakis Gregory,
Mola Francesco,
Siciliano Roberta,
Stavropoulos Photis
Publication year - 2004
Publication title -
journal of the royal statistical society: series a (statistics in society)
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.103
H-Index - 84
eISSN - 1467-985X
pISSN - 0964-1998
DOI - 10.1046/j.1467-985x.2003.00745.x
Subject(s) - computer science , process (computing) , software , quality (philosophy) , data mining , data quality , data science , software engineering , information retrieval , programming language , metric (unit) , operations management , philosophy , epistemology , economics
Summary. Data editing is the process by which data that are collected in some way (a statistical survey for example) are examined for errors and corrected with the help of software. Edits, the logical conditions that should be satisfied by the data, are specified by subject‐matter experts with a procedure which could be tedious and could lead to mistakes with practical implications. To render the process of edit specification more efficient we provide a new step—the definition of the so‐called abstract data model of a survey—which describes the structure of the phenomenon that is studied in a survey. The existence of this model enables experts to identify all combinations of variables which should be checked by edits and to avoid the definition of conflicting edits. Furthermore, we introduce an automatic data validation strategy—TREEVAL—that consists of fast tree growing to derive automatically the functional form of edits and of a statistical criterion to clean the incoming data. The TREEVAL strategy is cast within a total quality management framework. The application of the methodologies proposed is demonstrated with the help of a real life application.