Helping Novices Avoid the Hazards of Data: Leveraging Ontologies to Improve Model Generalization Automatically with Online Data Sources

The infrastructure and tools necessary for large‐scale data analytics, formerly the exclusive purview of experts, are increasingly available. Whereas a knowledgeable data miner or domain expert can rightly be expected to exercise caution when required (for example, around misleading conclusions supposedly supported by the data), the nonexpert may benefit from some judicious assistance. This article describes an end‐to‐end learning framework that allows a novice to create models from data easily by helping structure the model‐building process and capturing extended aspects of domain knowledge. By treating the whole modeling process interactively and exploiting high‐level knowledge in the form of an ontology, the framework is able to aid the user in a number of ways, including in helping to avoid pitfalls such as data dredging. Prudence must be exercised to avoid these hazards as certain conclusions may only be supported if, for example, there is extra knowledge that gives reason to trust a narrower set of hypotheses. This article adopts the solution of using higher‐level knowledge to allow this sort of domain knowledge to be used automatically, selecting relevant input attributes, and thence constraining the hypothesis space. We describe how the framework automatically exploits structured knowledge in an ontology to identify relevant concepts, and how a data extraction component can make use of online data sources to find measurements of those concepts so that their relevance can be evaluated. To validate our approach, models of four different problem domains were built using our implementation of the framework. Prediction error on unseen examples of these models show that our framework, making use of the ontology, helps to improve model generalization.