Normalized to: Bordelon, D.
[1]
oai:arXiv.org:2004.04165 [pdf] - 2077319
Distributed peer review enhanced with natural language processing and
machine learning
Submitted: 2020-04-08
While ancient scientists often had patrons to fund their work, peer review of
proposals for the allocation of resources is a foundation of modern science. A
very common method is that proposals are evaluated by a small panel of experts
(due to logistics and funding limitations) nominated by the grant-giving
institutions. The expert panel process introduces several issues - most
notably: 1) biases introduced in the selection of the panel. 2) experts have to
read a very large number of proposals. Distributed Peer Review promises to
alleviate several of the described problems by distributing the task of
reviewing among the proposers. Each proposer is given a limited number of
proposals to review and rank. We present the result of an experiment running a
machine-learning enhanced distributed peer review process for allocation of
telescope time at the European Southern Observatory. In this work, we show that
the distributed peer review is statistically the same as a `traditional' panel,
that our machine learning algorithm can predict expertise of reviewers with a
high success rate, and we find that seniority and reviewer expertise have an
influence on review quality. The general experience has been overwhelmingly
praised from the participating community (using an anonymous feedback
mechanism).
[2]
oai:arXiv.org:1806.08746 [pdf] - 1703899
ESO telbib: learning from experience, preparing for the future
Submitted: 2018-06-22
The ESO telescope bibliography (telbib) dates back to 1996. During the 20+
years of its existence, it has undergone many changes. Most importantly, the
telbib system has been enhanced to cater to new use cases and demands from its
stakeholders. Based on achievements of the past, we will show how a system like
telbib can not only stay relevant through the decades, but gain importance, and
provide an essential tool for the observatory's management and the wider user
community alike.
[3]
oai:arXiv.org:1802.03272 [pdf] - 1644795
The ESO Survey of Non-Publishing Programmes
Submitted: 2018-02-09, last modified: 2018-03-06
One of the classic ways to measure the success of a scientific facility is
the publication return, which is defined as the number of refereed papers
produced per unit of allocated resources (for example, telescope time or
proposals). The recent studies by Sterzik et al. (2015, 2016) have shown that
30-50 % of the programmes allocated time at ESO do not produce a refereed
publication. While this may be inherent to the scientific process, this finding
prompted further investigation. For this purpose, ESO conducted a Survey of
Non-Publishing Programmes (SNPP) within the activities of the Time Allocation
Working Group, similar to the monitoring campaign that was recently implemented
at ALMA (Stoehr et al. 2016). The SNPP targeted 1278 programmes scheduled
between ESO Periods 78 and 90 (October 2006 to March 2013) that had not
published a refereed paper as of April 2016. The poll was launched on 6 May
2016, remained open for four weeks, and returned 965 valid responses. This
article summarises and discusses the results of this survey, the first of its
kind at ESO.
[4]
oai:arXiv.org:1801.03366 [pdf] - 1616200
On the Availability of ESO Data Papers on arXiv/astro-ph
Submitted: 2018-01-10
Using the ESO Telescope Bibliography database telbib, we have investigated
the percentage of ESO data papers that were submitted to the arXiv/astro-ph
e-print server and that are therefore free to read. Our study revealed an
availability of up to 96% of telbib papers on arXiv over the years 2010 to
2017. We also compared the citation counts of arXiv vs. non-arXiv papers and
found that on average, papers submitted to arXiv are cited 2.8 times more often
than those not on arXiv. While simulations suggest that these findings are
statistically significant, we cannot yet draw firm conclusions as to the main
cause of these differences.