Theoretical physicists do a good job when peer reviewing scientific papers, but tend to be more impressed by the significance of new research rather than the rigour with which it was carried out. That is the provocative suggestion of a new study carried out by researchers in the UK and Poland that examines the reliability of peer review in theoretical physics.
Peer review involves sending research papers to independent, external referees to decide if an article is scientifically credible and appropriate for the journal to which it has been submitted. Despite being a central part of science for hundreds of years, few academics are trained in peer review, and anecdotal evidence suggests that disagreements between individual reviewers about a paper are common.
“In an ideal world, all reviewers would evaluate each submission with perfect accuracy and render a perfect decision,” says Mike Thelwall, a data scientist from the University of Sheffield, who carried out the study. But given that no work can be perfect in all of the metrics used to judge a paper’s quality, reviewers make what Thelwall dubs “a judgement call” about the extent to which an article achieves an acceptable level.
The three core components of quality used to judge a paper are originality, rigour and significance. However, Thelwall says that no-one had previously checked if reviewers judge these metrics effectively, even though they are, for example, part of the guidelines for assessors in the UK’s Research Excellence Framework.
Decision makers
Thelwall and physicist Janusz Hołyst of the Warsaw University of Technology therefore analysed reviewer scores in 505 theory papers published in SciPost Physics — a journal that practises open peer review. The duo chose to focus on theoretical physics because they say it is best-case-scenario for peer review, in that reviewers should be more likely to understand all aspects of a given submission than in other areas of science.
Each paper in the study had at least two scoring reviewers and the analysis revealed a “moderate” degree of agreement between reviewers on all three key metrics. “The results should cause no worries at all for SciPost Physics — their reviewers are doing an excellent job,” Thelwall says.
However, he worries that journals publishing research in fields where it is harder for individual reviewers to make a sound judgement might have much lower agreement rates for reviewers. “By the law of averages, [they may] tend to make bad decisions moderately often due to an unlucky choice of reviewers,” he says. The authors suggest such issue might be minimized via editorial oversight, better guidelines and improved reviewer training.
Peer review under the spotlight
The study also looked at which of the quality measures tend to be ranked the same by individual reviewers. The results suggest that — in theoretical physics at least — referees tend to rank rigour lower. This means it could be harder to publish what Thelwall calls a “highly rigorous original study”, but easier to get a “significant original study” accepted.
In fact, theoretical physicists striving to create significant work might benefit from focusing on original ideas rather than generating exhaustive evidence to support their work, the authors suggest. “Of course, we need all types of research and if everyone gives up on rigorous ideas it will be a disaster for science,” cautions Thelwall.
Stefan Thurner, a physicist from the Medical University of Vienna who was not involved in the study but has carried out studies into peer review, says the study is interesting. “The raw data indicate that the reviewers are remarkably consistent in practically all categories,” he says. “In more than 400 out of 505 cases, they differ not at all or by one score unit. I am sure that many scientists have the subjective feeling that referee reports differ more often than that.”
Thurner adds, however, that future research could compare the results with, say, reference models in which papers were randomly assigned to referees.