Experimenting with International Law

A growing body of experimental research has begun to explore the causal mechanisms through which international law impacts behaviour. International legal scholars, however, are still in the early stages of adopting experimental methods. Indeed, Yahli Shereshevsky and Tom Noah’s article is one of the first experimental studies to appear in the European Journal of International Law. Its publication thus provides an opportunity to reflect not only on this pioneering work but also on the broader ‘experimental turn’ in the study of international law. To do so, we begin by motivating the experimental turn, which we argue reflects both a methodological shift from observational studies towards the increasing use of experiments and a theoretical shift from rational choice towards cognitive psychology and behavioural economics. Second, we engage in a critical reading of Shereshevsky and Noah’s study of the impact of preparatory materials on treaty interpretation. Applying the dual criteria of internal and external validity, we assess the strengths and weaknesses of Shereshevsky and Noah’s study. We conclude that experiments promise to extend our knowledge of international law and are likely to become increasingly influential in scholarly and policy debates. Hence, all international lawyers have an urgent interest in becoming knowledgeable and critical consumers of experimental research. A rapidly growing body of experimental research, conducted primarily by political scientists, has begun to identify the varied causal mechanisms through which international law impacts behaviour. International legal scholars, however, are still in the early stages of adopting (and adapting) experimental methods. Indeed, Yahli Shereshevsky and Tom Noah’s article is one of the first experimental studies to appear in the pages of the European Journal of International Law (EJIL).1 Its publication thus * Laura H. Carnell Professor of Law, Beasley School of Law, Temple University, Philadelphia, PA, USA. Email: dunoffj@temple.edu. ** Professor of Political Science and Law, Jean Monnet Chair, Temple University, Philadelphia, PA, USA. Email: mark.pollack@temple.edu. We are grateful to Sarah Bush, Adam Chilton, David Hoffman and Katerina Linos for thoughtful comments on earlier drafts. 1 Shereshevsky and Noah, ‘Does Exposure to Preparatory Work Affect Treaty Interpretation? An Experimental Study on International Law Students and Experts’, 28 European Journal of International Law (EJIL)(2017) 1287. The first experimental study in the journal, to our knowledge, was published earlier this year: Puig and Strezhnev, ‘The David Effect and ISDS’, 28 EJIL (2017), 731. 1318 EJIL 28 (2017), 1317–1340 provides a timely opportunity to reflect not only on this pioneering work but also on the broader ‘experimental turn’ in the study of international law. Our aims in this brief article are three-fold. First, as a prelude to assessing Shereshevsky and Noah’s study, we step back and situate their research in the context of the larger turn to experiments. The rise of experimental methods in international law, we argue in Part 1, reflects both a methodological shift from observational studies towards the increasing use of experiments and a theoretical shift from neo-classical rational choice assumptions towards a behavioural analysis based on empirical findings from cognitive psychology and behavioural economics. Because both of these shifts are quite recent in the study of international law, we explore each in turn, and we provide a very brief guide to the assessment of experimental studies, emphasizing the criteria of internal and external validity. As explained more fully below, a typical internal validity question is whether variations in a dependent variable can be confidently attributed to a hypothesized independent variable within an experimental setting, whereas a typical external validity question asks whether behaviours observed in experimental settings can be generalized to other real-life settings. Second, having laid this groundwork, we turn in Part 2 to Shereshevsky and Noah’s study of the impact of preparatory materials on treaty interpretation. Applying the dual criteria of internal and external validity, we assess the many strengths of Shereshevsky and Noah’s study and note the ways in which future studies can build on it to increase our confidence in both the internal and external validity of its findings. Third and finally, our overarching goal is to spark a larger dialogue over the promise – and the perils – of ‘experimenting with international law’. Experiments, we argue, can deepen and extend our knowledge of international law’s workings and impacts and, for this reason, are likely to become increasingly influential in scholarly and policy debates. Hence, all international lawyers – even those with little interest in conducting experimental research themselves – have an urgent interest in becoming knowledgeable and critical consumers of experimental research. 1 Why Experiments? Methodological and Theoretical Shifts EJIL readers will be familiar with the decades-long rise of empirical legal studies, which involves the systematic analysis of qualitative or quantitative data, often using statistical techniques on large data sets.2 The use of experiments, a particular form of empirical investigation, in the study of international law is of a much more recent vintage, however, with the first explicitly experimental studies of international law appearing only in the past half-decade. This development invites an examination of the reasons for the adoption of experimental methodologies in both international relations and international law, the criteria used to judge the validity of experimental studies and 2 Shaffer and Ginsburg, ‘The Empirical Turn in International Legal Scholarship’, 106 American Journal of International Law (AJIL) (2012) 1. See also Hafner-Burton, Victor and Lupu, ‘Political Science Research on International Law: The State of the Field’, 106 AJIL (2012) 47. Experimenting with International Law 1319 an overall assessment of the advantages and disadvantages of experimental methods for students of international law. The experimental turn, we argue, is driven by both methodological and theoretical developments in the study of international relations and international law and, indeed, in the social sciences more broadly. In terms of methodology, the disciplines of both international relations and (more recently) international law are witnessing a shift from an emphasis on observational studies, which can be highly illuminating yet often struggle to make confident causal claims, towards an increasing use of experimental methods that seek to establish robust claims about causation through careful attention to experimental design. In this sense, the move to experiments reflects the inherent limitations of even the most technically sophisticated observational studies, and the promise of experiments is to complement (rather than replace) observational studies by testing causal claims that cannot be established through other methods. Such a purely methodological account of the experimental turn, however, is incomplete because the shift is also part and parcel of a broader theoretical development variously known as the ‘cognitive’ or ‘behavioural’ revolution. To an increasing extent, scholars across the social sciences are calling into question the admirably parsimonious, but, in some cases, demonstrably inaccurate, rational choice models that have spread from economics to neighbouring disciplines and strongly shaped both international relations theory and the law and economics approach to international law. Drawing on advances in cognitive psychology and behavioural economics, international relations and international law scholars are developing theories and advancing policy proposals that are built on more realistic understandings of human cognition and decision making. In doing so, researchers in both disciplines have used experiments to ascertain empirically how both elites and mass publics think about and respond to international law. While a thorough discussion of these parallel and intertwined methodological and theoretical developments is beyond the scope of this short article, we shall briefly consider each of them in turn, before turning to a careful analysis of Shereshevsky and Noah’s innovative experimental study. A The Methodological Shift: From Observational to Experimental Studies In the past two decades, empirical legal studies have moved from the periphery to the centre of legal scholarship. Alongside formal, doctrinal, and normative scholarship, international law journals devote increasing space to the publication of empirical studies that attempt to understand ‘the conditions under which international law is formed and has effects’.3 Whether qualitative or quantitative, these ‘observational studies’ collect empirical data about phenomena such as international treaties, judicial decisions and arbitral awards in an effort to understand both the factors that shape these legal texts as well as their effects on actors in the international system. In recent years, these methods have become increasingly sophisticated and ambitious, with the 3 Shaffer and Ginsburg, supra note 2. 1320 EJIL 28 (2017), 1317–1340 rise of ‘big data’ projects capable of capturing and coding not just a sample but, rather, all of the treaties, judicial decisions or arbitral rulings in entire issue areas of international law, permitting the identification of previously unknown patterns and trends in the data.4 This emergence of big data has in turn sparked impressive methodological innovations, such as network analysis and text as data, which have allowed empirical scholars to both describe and explain international legal phenomena. Using network methodology, for example, scholars have been able to map patterns of judicial citation and precedent, formulating