Open AccessApplication of MCAT questions as a testing tool and evaluation metric for knowledge graph–based reasoning systems
Author(s) -
Fecho Karamarie,
Balhoff James,
Bizon Chris,
Byrd William E.,
Hang Sui,
Koslicki David,
Rensi Stefano E.,
Schmitt Patrick L.,
Wawer Mathias J.,
Williams Mark,
Ahalt Stanley C.
Publication year - 2021
Publication title -
clinical and translational science
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.303
H-Index - 44
eISSN - 1752-8062
pISSN - 1752-8054
DOI - 10.1111/cts.13021
Subject(s) - computer science , inference , metric (unit) , graph , context (archaeology) , test (biology) , software engineering , information retrieval , artificial intelligence , theoretical computer science , paleontology , operations management , economics , biology
“Knowledge graphs” (KGs) have become a common approach for representing biomedical knowledge. In a KG, multiple biomedical data sets can be linked together as a graph representation, with nodes representing entities, such as “chemical substance” or “genes,” and edges representing predicates, such as “causes” or “treats.” Reasoning and inference algorithms can then be applied to the KG and used to generate new knowledge. We developed three KG‐based question‐answering systems as part of the Biomedical Data Translator program. These systems are typically tested and evaluated using traditional software engineering tools and approaches. In this study, we explored a team‐based approach to test and evaluate the prototype “Translator Reasoners” through the application of Medical College Admission Test (MCAT) questions. Specifically, we describe three “hackathons,” in which the developers of each of the three systems worked together with a moderator to determine whether the applications could be used to solve MCAT questions. The results demonstrate progressive improvement in system performance, with 0% (0/5) correct answers during the first hackathon, 75% (3/4) correct during the second hackathon, and 100% (5/5) correct during the final hackathon. We discuss the technical and sociologic lessons learned and conclude that MCAT questions can be applied successfully in the context of moderated hackathons to test and evaluate prototype KG‐based question‐answering systems, identify gaps in current capabilities, and improve performance. Finally, we highlight several published clinical and translational science applications of the Translator Reasoners.