Premium
The Forest or the Trees? Tackling Simpson's Paradox with Classification Trees
Author(s) -
Shmueli Galit,
Yahav Inbal
Publication year - 2018
Publication title -
production and operations management
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 3.279
H-Index - 110
eISSN - 1937-5956
pISSN - 1059-1478
DOI - 10.1111/poms.12819
Subject(s) - causal inference , computer science , decision tree , confounding , inference , big data , tree (set theory) , corporate governance , operations research , econometrics , data science , data mining , artificial intelligence , statistics , economics , mathematics , finance , mathematical analysis
Studying causal effects is central to research in operations management in manufacturing and services, from evaluating prevention procedures, to effects of policies and new operational technologies and practices. The growing availability of micro‐level data creates challenges for researchers and decision makers in terms of choosing the right level of data aggregation for inference and decisions. Simpson's paradox describes the case where the direction of a causal effect is reversed in the aggregated data compared to the disaggregated data. Detecting whether Simpson's paradox occurs in a dataset used for decision making is therefore critical. This study introduces the use of Classification and Regression Trees for automated detection of potential Simpson's paradoxes in data with few or many potential confounding variables, and even with large samples (big data). Our approach relies on the tree structure and the location of the cause vs. the confounders in the tree. We discuss theoretical and computational aspects of the approach and illustrate it using several real applications in e‐governance and healthcare.