Reproducing Neural Network Research Findings via Reverse Engineering: Replication of AlphaGo Zero by Crowdsourced Leela Zero

The reproducibility of scientific findings is essential to the integrity of research. The scientific method requires hypotheses to be validated independently by different laboratories. Investigators are expected to provide sufficient information in their publications to permit an objective evaluation of their methods and an independent reproduction of their results. This is particularly true for research supported by public funds, where transparency of both methods and findings represents a return on public investment. Unfortunately, many publications fall short of this standard for various reasons, including a desire to protect intellectual property or national security. The reproducibility of findings is essential in transferring machine learning findings from research into healthcare practice. Fortunately, the internet makes it easier to overcome these limitations by permitting multiple individuals to participate in reproducibility efforts and to crowdsource the reverse engineering of novel software. We present a case study of this capability from neural network research. The success of the crowdsourced project Leela Zero to reverse engineer the findings of AlphaGo Zero exemplifies the ability to reproduce novel results despite the lack of extensive computational resources or a detailed description of the initial experimental methods. The implications of this successful reverse engineering effort for future reproducibility of neural network research are discussed.