Normalized to: Heuritsch, J.
[1]
oai:arXiv.org:1801.08033 [pdf] - 1623935
Insights into the effects of indicators on knowledge production in
Astronomy
Submitted: 2018-01-22
Nine interviews were conducted with astronomers from Leiden University, and a
document analysis was performed on relevant institutional (self-) evaluation
documents, annual reports, and CVs of the interviewees. The aim was to perform
a qualitative study about the relationship between the research behaviour of
astronomers and how their science is being evaluated. This study encompassed
the funding and publication system as well as the indicators used to measure
the scientific output, its quality and the research performance. This report
sheds light on how astronomers define high-quality research and how they think
that creating knowledge of value is encouraged or hampered by the evaluation
processes. We found that astronomers are realists who define scientific quality
on the basis of "truth" and are driven by curiosity. These two factors make up
their intrinsic values and motivation to perform Astronomy. Publication
pressure, arising from the requirements of "the system", creates an extrinsic
motivation to perform. This results in premature publications, low readability
and replicability, risk aversion and a focus on quantity rather than quality.
Hence, indicators do not merely represent quality, but also co-constitute what
counts as good research. While we observe such constitutive effects of
indicator use on research behaviour and content, we do not see that the
astronomer's intrinsic values are co-constituted. This gives rise to a
discrepancy between what is being measured by indicators and what astronomers
define as scientific quality; the so-called "evaluation gap". Findings on
constitutive effects and the evaluation gap in Astronomy lays out the
conceptual groundwork for further empirical research and for policy advice on
alternative evaluation practices and innovative indicators with the aim of
bridging the "evaluation gap".