Premium
Improving corpus reproducibility through modular text transformations and connected data set
Author(s) -
Pulliza Jonathan,
Shah Chirag
Publication year - 2018
Publication title -
proceedings of the association for information science and technology
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.193
H-Index - 14
ISSN - 2373-9231
DOI - 10.1002/pra2.2018.14505501159
Subject(s) - computer science , set (abstract data type) , modular design , data collection , information retrieval , natural language processing , artificial intelligence , data set , data science , world wide web , programming language , statistics , mathematics
The Enron Email Corpus is one of the most utilized collections of documents in Natural Language Processing, Machine Learning, and Network Analysis. Different groups of researchers have transformed the corpus, changing the content and format to meet their needs. The many distinct versions can all claim to be the Enron Email Corpus, though they are as distinct from the original publicly available collection as they are from each other. Researchers then have to determine the usefulness of a particular version in comparison to the many others available, as well as ascertain what has been done to the collection and how it would affect their specific research goal. This is especially important for reproducing a particular research method onto a different corpus, as transposing a method necessitates a deep understanding of the original data in the experiment. This project models the various transformations performed on different versions of the collection to form a network of connected datasets, highlighting the most important nodes as well as the most common transformations. Traversing different paths between nodes offers the community a way to model and reproduce the necessary data work performed on one collection that can be transposed onto other collections.