HyperPIE: Hyperparameter Information Extraction from Scientific Publications

Automatic extraction of information from publications is key to makingscientific knowledge machine readable at a large scale. The extractedinformation can, for example, facilitate academic search, decision making, andknowledge graph construction. An important type of information not covered byexisting approaches is hyperparameters. In this paper, we formalize and tacklehyperparameter information extraction (HyperPIE) as an entity recognition andrelation extraction task. We create a labeled data set covering publicationsfrom a variety of computer science disciplines. Using this data set, we trainand evaluate BERT-based fine-tuned models as well as five large languagemodels: GPT-3.5, GALACTICA, Falcon, Vicuna, and WizardLM. For fine-tunedmodels, we develop a relation extraction approach that achieves an improvementof 29% F1 over a state-of-the-art baseline. For large language models, wedevelop an approach leveraging YAML output for structured data extraction,which achieves an average improvement of 5.5% F1 in entity recognition overusing JSON. With our best performing model we extract hyperparameterinformation from a large number of unannotated papers, and analyze patternsacross disciplines. All our data and source code is publicly available athttps://github.com/IllDepence/hyperpie