Premium
Introducing “best single template” models as reference baseline for the Continuous Automated Model Evaluation (CAMEO)
Author(s) -
Haas Juergen,
Gumienny Rafal,
Barbato Alessandro,
Ackermann Flavio,
Tauriello Gerardo,
Bertoni Martino,
Studer Gabriel,
Smolinski Anna,
Schwede Torsten
Publication year - 2019
Publication title -
proteins: structure, function, and bioinformatics
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.699
H-Index - 191
eISSN - 1097-0134
pISSN - 0887-3585
DOI - 10.1002/prot.25815
Subject(s) - casp , benchmarking , computer science , data mining , task (project management) , set (abstract data type) , data set , machine learning , protein structure prediction , artificial intelligence , engineering , systems engineering , protein structure , physics , nuclear magnetic resonance , marketing , business , programming language
Critical blind assessment of structure prediction techniques is crucial for the scientific community to establish the state of the art, identify bottlenecks, and guide future developments. In Critical Assessment of Techniques in Structure Prediction (CASP), human experts assess the performance of participating methods in relation to the difficulty of the prediction task in a biennial experiment on approximately 100 targets. Yet, the development of automated computational modeling methods requires more frequent evaluation cycles and larger sets of data. The “Continuous Automated Model EvaluatiOn (CAMEO)” platform complements CASP by conducting fully automated blind prediction evaluations based on the weekly pre‐release of sequences of those structures, which are going to be published in the next release of the Protein Data Bank (PDB). Each week, CAMEO publishes benchmarking results for predictions corresponding to a set of about 20 targets collected during a 4‐day prediction window. CAMEO benchmarking data are generated consistently for all methods at the same point in time, enabling developers to cross‐validate their method's performance, and referring to their results in publications. Many successful participants of CASP have used CAMEO—either by directly benchmarking their methods within the system or by comparing their own performance to CAMEO reference data. CAMEO offers a variety of scores reflecting different aspects of structure modeling, for example, binding site accuracy, homo‐oligomer interface quality, or accuracy of local model confidence estimates. By introducing the "bestSingleTemplate" method based on structure superpositions as a reference for the accuracy of 3D modeling predictions, CAMEO facilitates objective comparison of techniques and fosters the development of advanced methods.