Empowering Peer reviewers: How reviewer insights drive innovation at IOP Publishing
Abstract
Laura Feetham-Walker, Reviewer Engagement Manager at IOP Publishing, recently won the Academic Publishing in Europe (APE) Innovation award for invigorating the peer review process with the launch of the Peer Review Excellence programme, a comprehensive training and recognition programme, and introducing co-review and reviewer feedback. In this paper, we discuss how IOP Publishing has implemented these innovations and how they can future-proof a process that has served the academic community for centuries. We also discuss how some of these innovations have already proven effective in counteracting the peer review crisis.
1.Introduction
Peer review plays a critical role in maintaining the quality and integrity of scientific research. It involves subjecting research papers to scrutiny by experts in the field to help inform editorial decisions. However, in recent years, there has been growing concern about a “peer review crisis” [1] which is fuelled by several underlying issues.
One concern is fatigue among peer reviewers, which has led to some journals needing to invite more reviewers for each manuscript, increasing the time taken to peer review manuscripts [2]. The causes of this fatigue are manifold, but include increased global scientific output, an increase in competing demands on academics’ time (such as teaching, pressure to publish more research, and administrative tasks), and an over-reliance on a small group of people to provide peer review reports. Historically, peer reviewers have been overwhelmingly senior researchers, based in high-income countries, with a skew towards male gender. The reviewer pool has therefore not been representative of the research community as a whole, and as scientific output has increased, the burden on this small group of people has led to a reduction in acceptance rates to peer review.
In recent years, publishers have been trying to diversify and enrich their reviewer pools to include more early-career researchers, people from low- and middle-income countries, and women [3]. However, our recent State of Peer Review Report 2024 [4] indicated that there is a high proportion of researchers who are qualified to peer review and feel that they have more time available to review compared with the number of invitations that they receive. This suggests that there are still groups of people who are not being as included in the peer review process as they should be.
Amidst these pressures, the peer review process has also been criticised for failing to adequately detect errors, flaws, or fraudulent behaviour in submitted research. With the rise of papermill-generated articles, a lack of robust quality control can undermine the reliability and credibility of published research, leading to the dissemination of inaccurate or misleading findings and ultimately reducing trust in science.
The peer review process can also be time-consuming, with reviewers often taking weeks to review papers. When the peer review process takes longer than necessary, the dissemination of peer reviewed research findings gets delayed, hindering scientific progress.
The Peer Review Excellence programme [5] was founded in 2020 with the aim of addressing these issues. Based on intensive research with diverse members of the physical sciences community, the goal of the programme was to acquire a more diverse, skilled, and empowered group of peer reviewers to undertake reviews, and to improve engagement within our reviewer pool.
2.Peer review innovations
Following a 2020 survey of reviewers in the physical sciences [6], IOP Publishing (IOPP) sought to address the lack of reviewer recognition and the gap in peer review training by launching the Peer Review Excellence programme, which was later followed by the options to review with a colleague (co-review) and receive feedback on their reviewer reports (reviewer feedback).
2.1.Recognising reviewers
The first element of the Peer Review Excellence programme was the launch of IOP Trusted Reviewer status, which was introduced during Peer Review Week 2020 to better recognise IOPP’s best reviewers. IOP Trusted Reviewer certification is designed to recognise the very best peer reviewers in the physical sciences, giving reviewers recognition for the important work they do. Only the top 15% (approximately) of reviewers achieve IOP Trusted Reviewer status.
From the start of the programme up until May 2024, we certified 14,000 people as IOP Trusted Reviewers for their timely and thorough reviews which add valuable insight when informing the Editor’s decision. Feedback on the programme from reviewers has been overwhelmingly positive:
“Learning that my review has been rated as ‘Outstanding’ and has earned me the IOP Trusted Reviewer status is truly gratifying. This recognition not only highlights the importance of thorough and constructive peer review in the scientific process but also reinforces my commitment to maintaining high standards in my evaluations.”
Experienced researcher, University of Isfahan, Iran.
“Being an IOP Trusted Reviewer is an important responsibility that I take seriously. It affirms that my reviews are accurate, impartial, timely and contribute value to the scientific community. I appreciate the feedback and training provided through the Peer Review Excellence program, as it has helped me develop robust peer review skills (…) It is a privilege to support open science as an IOP Trusted Reviewer.”
Experienced researcher, University of Baghdad, Iraq.
“Thank you for choosing me as a 2023 Trusted Reviewer. This award will motivate me to continue reviewing activities to make a strong science foundation globally.”
Experienced researcher, Kansas State University, United States.
“I am honoured to be awarded the Trusted Reviewer status. To review manuscripts is a moral duty for an academic and I am proud that my contribution is recognised in this way.”
Experienced researcher, National Graduate School of Engineering & Research Center, France.
The programme has proven to be an invaluable resource for editors who can easily identify excellent reviewers and speed up peer review times.
2.2.Training reviewers
The second element of the Peer Review Excellence programme was high-quality training for reviewers in the physical sciences that can be accessed and completed by anyone globally via our online training portal. Historically, researchers were expected to figure out how to review on their own or hope that they had a hands-on supervisor who would be willing to share their peer review experiences and learnings.
Since the launch of the Peer Review Excellence training programme in 2021, over 7000 people have signed up for IOPP’s free and open online peer review course. As well as the online course, we run over 20 workshops annually. The workshops are highly interactive, co-chaired by leading academics, and are aimed at under-represented groups including people in geographies that have traditionally been excluded from the peer review process, early-career researchers, and new and emerging topics. Participants who successfully complete the course, either online or via a workshop, receive certification and are fast-tracked to IOP Trusted Reviewer status.
“I would like to say that I found the training to be very informative and helpful. The package has been put together very nicely. I have reviewed for many years and think that I am reasonably accomplished. For sure, though, I learned a number of things and obtained a much better appreciation for the breadth and complexity of the vetting process for a paper.”
Experienced researcher, General Atomics, United States.
“The course content was extremely informative and useful, and the materials were well-organized and simple to understand. I enjoyed the tests at the end of each session to assess the knowledge gained. Thank you for the Peer Review Excellence training course.”
Early-career researcher, National Institute of Technology Calicut, India.
“The presentation was very informative and structured well. The practical exercise and discussion helped to put the knowledge we had learned into practice (…) The workshop has given me confidence to accept papers that I usually get asked to review. I feel like now I’ll be able to know what to do, what to look for when reviewing manuscripts and how to give good feedback to the editor and authors. I don’t feel as clueless and anxious as I did before about being a reviewer.”
Anonymous feedback from a workshop attendee, 20th June 2023.
“The presentation was well organized. Content totally relevant. Friendly approach of contents. Time well distributed. I not only enjoyed it, but learnt a lot. I had no idea it would be a meeting so well curated and targeted, but I’m happy I attended. I do not have a very big set of colleagues, but from now on, I will tell them that at IOP, there is an approach to reviewers as real people and not only as instruments.”
Anonymous feedback from a workshop attendee, 19th April 2024.
2.3.Co-review
Addressing feedback from our reviewer community also led IOPP to introduce two innovations in the peer review process in 2023. Recognising the underrepresentation of early-career researchers (ECRs) in peer review, IOPP introduced co-review across all IOPP’s proprietary journals. Co-review allows reviewers to formally invite colleagues to collaborate on reviews, aiming to rectify the lack of acknowledgment for ECR contributions, offering mentorship and due credit. Early-career researchers often support more experienced academics by contributing ideas or comments to their peer review reports. Yet, according to a survey, “70% of co-reviewers report the experience of making significant contributions to a peer review report without knowingly receiving credit”. Lack of recognition for co-written reports was one of the issues that stood out from extensive interviews with physical science researchers.
2.3.1.Legitimising co-reviewing
When designing and implementing our co-review functionality, our goal was to legitimise the co-review process, and ensure that both parties get recognition for their work. When receiving an invitation to review a manuscript, researchers are offered the option to “co-review with a colleague”. If they choose this option and provide the details of their chosen colleague, a formal invitation is sent to write and submit the review. Both reviewers will then be able to receive formal recognition for their work through the Web of Science Reviewer Recognition Service and our reviewer APC discount. Importantly, the Editor is fully aware of who is reviewing the manuscript, alleviating any issues around confidentiality and transparency. Furthermore, both reviewers will be eligible for IOP Trusted Reviewer Status if they submit an outstanding report.
2.3.2.Learning on the job
Legitimising reviewer contributions through co-review enables early-career researchers to gain hands-on peer review experience and receive credit for their contributions. At the same time, experienced reviewers can support the next generation of scientists by mentoring these aspiring researchers while sharing their reviewing responsibilities. This collaborative approach not only benefits the individual reviewers, but also promotes a sense of community.
2.3.3.Alleviating peer review pressure
With peer review pressure on experienced researchers mounting due to the growing volume of manuscripts, co-review can also help to address the shortage of reviewers in the scientific community. Giving early-career researchers recognition for their work and encouraging collaboration with experienced reviewers makes it easier for early-career researchers to become involved in the peer review process. Bringing together a broader range of perspectives from researchers at various stages of their careers will ultimately strengthen the quality and integrity of peer review, leading to more robust scientific publications.
With almost one year’s worth of data on co-review, we have found that most co-review requests come from senior researchers wanting to share their review with a more junior colleague. However, a small number of requests are two individuals at similar career stages who have complimentary expertise (this is more common from cross-disciplinary research papers), and a minority of requests come from early-career researchers who wish to ‘delegate upwards’ to a more senior colleague as they want support in writing their review.
Our early data also suggest that the standard of co-review reports is higher than non-co-review reports on average. At IOPP, our internal editorial teams evaluate all the reports we receive on a scale of 1–5. Our statistics show that the mean evaluation for co-review reports from May 2023 to April 2024 was 3.5, while the average for all reports over the same period was 3.1.
Co-review has also enabled us to grow and diversify our reviewer pool by bringing in more early-career researchers. Because the final review report is (in almost all cases) submitted via the more junior co-reviewer’s account, the functionality has brought an influx of highly capable early-career researchers who can be invited to review again in future.
2.4.Reviewer feedback
Another initiative that was launched in response to IOPP’s 2020 Reviewer Insights Survey was the reviewer feedback programme. In 2023, IOPP became the first major publisher to offer reviewers feedback on their reports’ structure and usefulness direct from the editorial team. In this system, reviewers can opt in or out of receiving feedback.
The 2020 Reviewer Insights Survey showed us that feedback on reviewer reports would improve the peer review experience, whether that is being notified about the final decision on the paper or receiving information about the quality of the report. In particular, the survey results showed that feedback on the quality of reports was most valued by early-career researchers, as this feedback can be invaluable in developing their research skills.
Since implementing the programme, a high proportion of reviewers (60%) have chosen to receive feedback on their reviews, and comments from the community have been overwhelmingly positive:
“I’ve never gotten any feedback about my reviewer reports–not from any journal, not from any paper I’ve ever reviewed. It was incredibly helpful to see what the editors are looking for in a review, and I think it will help me to be a better reviewer in the future.”
Anonymous feedback from a reviewer, 25th September 2023.
“I have never before received feedback on my journal reviews. I strive to provide fair, polite and detailed reviews within my field of expertise (…) This feedback confirms that my efforts are worthwhile.”
Experienced researcher, Natural Resources Canada, Canada.
“I wish every journal would provide this feedback, thank you. It is very useful, and motivating.”
Anonymous feedback from a reviewer, 10th April 2024.
“I’m a junior researcher and my supervisor does not have time and resources to check my review. This feedback is very helpful for me.”
Early-career researcher, Forschungszentrum Julich GmbH, Germany.
IOPP have rolled out the new programme on an opt in basis across all its proprietary journals. Where reviewers opt in for feedback on their report, IOPP will share a numerical evaluation of how useful the in-house editor found the review on a scale of 1 to 5, with 5 being outstanding and 1 indicating that the report is not helpful to inform a decision. Reviewers are also sent an evaluation matrix which adds context to the scores.
The launch followed a successful trial of the programme across three IOP Publishing journals: Engineering Research Express, Environmental Research Letters, and Plasma Physics and Controlled Fusion. Over 85% of reviewers involved in the trial indicated that receiving feedback on their report was useful or very useful.
3.Conclusion
IOPP’s Peer Review Excellence programme represents a significant step towards addressing the pressing issues within the peer review process. By introducing a comprehensive training and recognition program, alongside peer review innovations such as co-review and reviewer feedback, IOPP has not only diversified and enriched the reviewer pool, but also empowered early-career researchers and underrepresented groups. These innovations are counteracting the peer review crisis and future-proof a process critical to maintaining the integrity and quality of scientific research.
References
[1] | https://link.springer.com/article/10.1057/s41270-022-00176-5. |
[2] | M. Breuning, J. Backstrom, J. Brannon, B.I. Gross and M. Widmeier, Reviewer fatigue? Why scholars decline to review their peers’ work, PS Political Science Politics 48: : ((2015) ), 595–600. |
[3] | JAT da Silva - Journal of Science and Medicine in Sport, 2023 – Elsevier, 10.1016/j.jsams.2023.06.006. |
[4] | State of Peer Review Report 2024. https://ioppublishing.org/state-of-peer-review-2024/. |
[5] | https://ioppublishing.org/researchers/peer-review-excellence-training-and-certification/. |
[6] | Peer Review Survey Insights 2020. https://ioppublishing.org/peer-review-survey-insights/. |
