English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Peer review is the most common mechanism in place for assessing requests forresources in a large variety of scientific disciplines. One of the strongestcriticisms to this paradigm is the limited reproducibility of the process,especially at largely oversubscribed facilities. In this and in a subsequentpaper we address this specific aspect in a quantitative way, through astatistical study on proposal ranking at the European Southern Observatory. Forthis purpose we analysed a sample of about 15000 proposals, submitted by morethan 3000 Principal Investigators over 8 years. The proposals were reviewed bymore than 500 referees, who assigned over 140000 grades in about 200 panelsessions. After providing a detailed analysis of the statistical properties ofthe sample, the paper presents an heuristic model based on these findings,which is then used to provide quantitative estimates of the reproducibility ofthe pre-meeting process. On average, about one third of the proposals ranked inthe top quartile by one referee are ranked in the same quartile by any otherreferee of the panel. A similar value is observed for the bottom quartile. Inthe central quartiles, the agreement fractions are very marginally above thevalue expected for a fully aleatory process (25%). The agreement fractionbetween two panels composed by 6 referees is 55+/-5% (50% confidence level) forthe top and bottom quartiles. The corresponding fraction for the centralquartiles is 33+/-5%. The model predictions are confirmed by the resultsobtained from boot-strapping the data for sub-panels composed by 3 referees,and fully consistent with the NIPS experiment. The post-meeting phase will bepresented and discussed in a forthcoming paper.

Peer-review under review - A statistical study on proposal ranking at ESO. Part I: the pre-meeting phase