An introduction to peer review and some advice for reviewers (2021 Peer Review Series, part 1 of 3)
Welcome to Peer Review Week 2021!
PRW is an annual event that highlights the importance of peer review in the scholarly ecosystem. The theme for PRW 2021 is ‘identity in peer review,’ encouraging us to reflect on the role of identity and diversity in the peer review process.
This is our first post in a three-part series on peer review. Across this series we will feature the perspectives of four senior editors of academic journals in different fields:
Prof Simine Vazire, Professor of Psychology Ethics and Wellbeing at the Melbourne School of Psychological Sciences, UoM (Editor-in-Chief of Collabra: Psychology)
Prof William Locke, Director of the Melbourne Centre for the Study of Higher Education, UoM (Founding Joint Editor of Policy Reviews in Higher Education)
Prof Norie Neumark, Honorary Principal Fellow at the Victorian College of the Arts, UoM (Founding Editor of Unlikely: Journal for the Creative Arts)
Dr Esther Levy, Wiley (Editor-in-Chief, Advanced Materials Technologies)
In this first post, we will look at the importance of peer review, reflect on this year’s PRW theme, and offer some advice for reviewers. Next week, in part two, we will consider the different kinds and qualities of peer review and bring together some advice for authors facing peer review of their work. Finally, in part three, we will look at the changing peer review landscape and what might be the future of the peer review process.
The importance of peer review
Peer review has long been at the heart of scholarly publishing. In a typical peer-review process, the editors of a journal (or publishers of a book) will send promising manuscripts they receive to one or more experts for their review. These experts, usually leading academics in their field, will write out their reflections and critiques of the work and advise either: (a) accepting the manuscript for publication as it is; (b) accepting the manuscript pending revisions, which could be minor or major; or (c) rejecting the manuscript entirely. This is most commonly a ‘blinded’ process, where the identities of the authors and/or reviewers are hidden during review. We asked our contributors why this peer review process is important to scholarship.
Dr Levy: Peer review is the core value of scientific publications. It examines the validity, quality, and (often) originality of the work. When conducted properly, it helps maintain the integrity of science, by identifying weaknesses in the work, encouraging authors to meet the accepted standards of their discipline, and ensuring flawed work is not published. Because published research has been scrutinised by peers it gives readers some assurance that the work is valid and authentic.
Prof Locke: [Peer review] is the main way of ensuring that the research published is rigorous, coherent, builds on existing knowledge, and adds to what we already know. Asking peers – scholars with equivalent expertise in the broad area of the research – to review the work without knowing the identity of the authors ensures that their evaluation is based on the quality of the research rather than the quality or reputation of the researchers. Inviting reviews from more than one reviewer, in addition to the expertise of the editor(s), helps to achieve a consensus around the strengths and weaknesses of the research being reviewed.
Prof Neumark: To me, peer review is not really about ‘enforcing standards,’ but, rather, it is about a community of scholars, engaging in vital and ongoing conversations with each other, so that the field itself and the individual researchers can benefit. In this way, rather than having some fixed ‘external’ and bureaucratic monitoring, peer review provides researchers with access to the advice of their peers, as knowledge in a field grows and changes.
Identity in peer review
The theme of Peer Review Week 2021 is ‘Identity in Peer Review,’ with organisers encouraging us to reflect on ‘the role of personal and social identity in peer review and ways the scholarly community can foster more diverse, equitable, and inclusive peer review practices.’ Prof Neumark and Prof Locke offered some reflections on this topic and noted the significant role that diversity plays in quality peer review.
Prof Neumark: Diversity in peer review is just as important as diversity in research and teaching. This keeps a field open and accessible in a way that widens and deepens the knowledge and practices in the field.
Prof Locke: It is important that a wide range of scholars, representing all parts of the community, have the opportunity to act as peer reviewers. Otherwise, we run the risk of a very narrow – probably older, white males – dominating the definition and boundaries of a discipline or field of studies, the objects of study and the methods of investigation.
Why peer review?
What motivates someone to become a peer reviewer? Prof Neumark and Prof Locke both see contributing to peer review processes as an important contribution to scholarship and the academic community.
Prof Neumark: I [have] responded to [peer review] requests because I understood that, as a fully employed academic, it was part of my contribution to my field and assistance to fellow researchers. I was keen to review early-career researchers in a way that I could give full and useful feedback. In peer reviewing, as in teaching, I am keen to support positive feedback rather than gatekeeping. In a way, I see peer reviewing as a form of teaching and an opportunity to share my own knowledge and experience.
Prof Locke: I review as a form of academic service to the community, but also to educate myself and sharpen my skills as an author and editor.
Advice for peer reviewers
Becoming a peer reviewer can present new challenges to researchers, as they have to determine the best way to provide constructive feedback and contribute to the scholarly community. Prof Vazire and Prof Locke offered some advice for academics who have been invited to peer review.
Prof Vazire: When you review, you become part of the incentive structure – what you focus on shapes what authors are incentivized to get right. So when reviewers give positive evaluations to papers that make extravagant claims, e.g., about how novel and groundbreaking their work is, or about how their results conclusively settle an age-old debate, or can be used to solve a pressing societal problem, that contributes to a culture that incentivizes exaggeration over accuracy.
In my opinion, the focus of peer review should be on evaluating whether the research is rigorous, whether it’s reported transparently, and whether the authors’ conclusions are well-founded. Peer review should be a mechanism for discouraging exaggeration and hype in research papers – that should count against a paper. Too often, I feel that reviewers and editors implicitly encourage authors to make bolder claims than their research warrants, because papers get rejected for not being innovative enough, or being too incremental, and so on.
The flip side of that is that even if the research has flaws, it can be appropriate for it to be published in a peer-reviewed journal as long as the authors’ conclusions are calibrated to the strength of the research and the evidence. The goal is not to reject every flawed paper – the goal is to make sure authors are transparent and calibrated in their claims. So our primary focus as reviewers should be: Did the authors provide enough information for readers to critically evaluate their claims? Are the authors’ claims reasonable given the design and methods of their study, and the results/evidence presented?
Prof Locke: It should be a much more constructive process. Reviewers should ask themselves, what could the authors do to improve this piece of writing, rather than just finding excuses to reject it.
Join us again next week for part two of this series, when we will explore different kinds of peer review and bring together some advice for authors from our panel of experts.
Leave a Reply