Is the tail wagging the dog? Understanding the value of incentives in peer review

1.Background

The peer review process necessitates the survival and maintenance of a complex relationship between authors, editorial boards, ‘reviewers’ (active researchers working in specific disciplines) and ‘publishers’ (either for books or academic journals). The process is in part altruistic even though researcher-reviewers often feel ‘taken advantage of’ or even ‘exploited’ as they do a lot of work assessing articles without monetary compensation and recognition while publishers are the ones making financial profits. This nature of the relationships means that whether, or not, to reward peer reviewers for their work remains highly controversial. Is paying researchers for the service of reviewing in a process that generates profits for businesses ethical and legitimate or not? In other words: Does the payment influence a researcher’s independent judgment on the quality of research results? And if so, how can product categories like books and journal articles be handled completely differently? Data show that incentives, especially financial rewards, are much more common in book publishing than they are for academic journal articles [2]. Peer reviewers turn down requests for a number of reasons, as our data show (Fig. 1).

Peer review entails experts within a specific field assessing and providing feedback on research papers prior to their publication in journals or presentation at conferences. This process is the cornerstone of academic publishing: Peer review ensures that content can be trusted, re-used, and cited. However, despite its significance, numerous peer reviewers do not receive monetary compensation and rarely receive recognition for their valuable time and expertise. Among the frequent inquiries we receive from researchers are questions such as, ‘What constitutes payment for peer review?’ and ‘Why is compensation for peer review uncommon?’

Thus, in order to assess the significance and utility of incentives in peer review, we undertook a Reviewer Credits survey at the end of 2023. We leveraged our membership base and used social media channels as a control group. Results and key takeaways are presented and discussed in this article, alongside a series of recommendations both for the future of peer review incentives and their use and utility in academic publishing. It is worth noting at the outset that although peer reviewers feel hard done by, and often exploited by journals and publishers, actually about 60% of all published content (books and journals) in 2023 will involve some kind of recompense: article processing charges (APC) tokens, defrayed publishing fees, or credits from Reviewer Credits.

Fig. 1.

The shape of peer review: Why do some peer reviewers turn down requests from journals and editors?

2.Methods and results: The reviewer credits peer review incentives survey

We leveraged our social media and networks within academia and the publishing industry to reach out to a representative sample of peer reviewers and editorial board members (Fig. 2). We asked these cohorts a series of nine questions which we report on in this article. Our survey was conducted in November and December 2023, entirely online.

Fig. 2.

The randomized sample used in the Reviewer Credits survey.

We started off by asking how often, over the last two years, they had been offered some kind of reward by a journal or publisher for the work they had done assessing a paper. Our results show that the clear majority of researchers, more than 75%, had not been offered any kind of reward for peer review, or that this happened infrequently (in less than 25% of cases). This result is at odds with perceptions in the publishing industry: Journals tend to feel that they do provide rewards and incentives for peer review. It is also striking that a small proportion of reviewers (23%) have actually been rewarded (in more than 50% of cases) for their work for journals.

We were then interested whether reviewers saw positive effects, if publishers offered incentives. The intention was to address publishers’ frequent concerns about articles in need of peer review growing faster than the number of researchers actually reviewing content. Our results show that in 44% of answers, reviewers claim that incentives actually do have a positive effect on their willingness to review (Fig. 3).

Fig. 3.

Summary of responses from peer reviewers.

In terms of editors, our survey showed that more than 80% of journal editors do not reward peer reviewers. This is surprising given the results reported earlier. Indeed, more than 80% of respondents felt that appropriate rewarding would have a positive effect on peer review behavior. Indeed, it does seem clear that attractive rewards appear to be less important overall than simply giving reviewers something in return for their efforts, so they feel at least appreciated.

The most common kind of reward provided by journals and publishers to reviewers tends to be the use of APC tokens. These are wanted by 52% of reviewers and are actually given in 42% of cases. However, these rewards ‘hook’ peer reviewers into particular journal or publisher ecosystems. Certificates, viewed by journals and publishers as an effective rewarding system and hence being used more broadly, are actually only really appreciated in 12% of cases - these are given out as ‘rewards’ in 43% of cases. Similarly, a low number of reviewer respondents (27%) suggested that money would be appropriate as a reward for efforts working on papers: Payments are made in far fewer cases (4%), at least in journal peer review (Fig. 4).

Fig. 4.

Differences in expected and realized incentives for peer review.

3.Discussion and recommendations

Peer reviewing is a time-consuming and intellectually demanding task, and it is seen as the core value creation of the academic publishing process. Reviewers are asked to read and evaluate manuscripts, provide constructive feedback, and ensure that research meets the standards of the journal or conference. This process can take hours, and sometimes even days, depending on the complexity of the work being reviewed. So, why are so many peer reviewers not compensated for their efforts despite the fact that our survey results clearly show that the use of incentives, especially cash and APC tokens, can both speed up the process and enhance its effectiveness?

Informal discussions within STM publishing and with researchers and academics across the Reviewer Credits network reveal that several factors contribute to this interesting phenomenon. These include tradition: The practice of unpaid peer review has a long history in academia. It is seen as a professional duty and a way for researchers to contribute to their academic communities. Some ethics researchers also feel that adding clear incentives to peer review will somehow ‘pollute’ the process. At the same time, it is also true that many academic journals and conferences operate on limited budgets, especially those that are not-for-profit or have a narrow readership. Allocating funds for reviewer compensation may strain their resources.

Peer review is often considered a voluntary service provided by researchers to maintain the integrity of their field. It is viewed as part of one’s professional responsibilities. In sum, this means that paying peer reviewers may raise concerns about conflicts of interest, as reviewers may have financial incentives to favor certain authors or research outcomes. It is therefore often considered more appropriate to provide reviewers with non-monetary rewards, such as recognition, career advancement, and the opportunity to stay informed about the latest research in their field.

Nevertheless, some journals and conferences offer token honoraria or small payments to peer reviewers as a gesture of appreciation for their time and expertise. While these payments may not fully compensate for the effort involved, they acknowledge the value of reviewers’ contributions. We are starting to implement this kind of rewarding structure at Reviewer Credits.

It is important to mention that peer review is not free of inequalities, even without payments or non-financial benefits. For researchers, it makes a big difference which journal or book program they review for - not only reputationally, but also with respect to why many researchers review regularly, namely to stay up to date on the progress of research in their domain. It is a question for further research, whether economic benefits in peer review can have the effect of balancing existing inequalities in the peer review system.

3.2.What are alternative models in digital academic publishing?

Digitization has opened new opportunities to create transparency and equity in peer review. Reviewer Credits was created by researchers to make peer review visible on academic records and to reward time spent on this task. We also aim to rectify negative perceptions about peer review amongst both authors and editors in our vision for Reviewer Credits: We provide solutions for many of the challenges authors face during the peer review process (and therefore publishers) (Fig. 5).

The model is based on reviewer verification, another key issue for publishers globally. As discussed, from the perspective of peer reviewers’ and understanding their time constraints, publishers need to create a competitive incentive to other activities – first and foremost doing research or writing a paper – in favor of peer review. In a research climate increasingly characterized by a demand to hit different measurable performance indicators, how can researchers be expected to invest a substantial portion of their time on an ‘invisible’, unrewarded task? One peer review easily takes them four to six hours of work (or more if English language is an issue) – and research indicates that an aggregate of 15,000 years of peer review were conducted in 2020 alone [3].

Fig. 5.

The Reviewer Credits portfolio.