I tried to title this blog post “7 things to keep in mind during peer review (#5 will shock you)” but just couldn’t bring myself to do it.
It’s paper review season; I’ve been working on reviewing papers for a conference for which a former IBM colleague invited me to be on the program committee. Then this morning a long-time reader of this blog submitted a question:
First time on an academic CFP review team. Have the papers. Any tips for what to do?
Do I ever. Here’s my advice:
- Don’t wait until the last minute to do your reviews. I try to do at least 1/day so that I make progress and am able to give each paper enough time to do a good job.
- The words of the day are ‘constructive criticism’. Especially for papers you grade as ‘reject’, give the authors suggestions that, if adopted, would have moved the paper towards the ‘accept’ column.
- Don’t be biased by poor English skills. Judge the merit of the ideas. If needed, the program committee chair can assign a ‘shepherd’ to work with the authors to improve the phrasing and presentation. (There is usually a way for you to provide reviewer comments that are not given to the authors — you can flatly say “if we accept this paper then such-and-such must be fixed before publication.”
- Be fair, in that it’s easy to say “well that’s obvious” and assign a paper a low score. Was it obvious *before* you read the paper? Every manuscript doesn’t have to be groundbreaking. Rather, every paper should advance the community’s understanding of important issues/concepts in a way that can be externally validated and built upon in future work.
- The Golden Rule applies: I try to give the kind of feedback (or: do the kind of thoughtful evaluation) that I wished other reviewers did on the papers I submitted. That doesn’t mean I accept everything, of course; historically speaking I think I recommend ‘accept’ for only about 20% of papers.
- If you are working with junior academic types it can be a good idea to farm out a paper or two to them, both to give them experience/exposure and to give you sometimes a better evaluation of the paper than you might have done yourself. Be sure to convey the importance of confidentiality and ethics (not stealing unpublished work). Also be sure to read the paper yourself, and file your own review if there are important points not stated by your ‘external reviewer’.
- Peer review is part of the fuel that makes our scientific engines work. I’ve always thought of it as an honor to be asked to review a paper (and a *big* honor the few times I’ve been asked to be on a Program Committee). So I try to deliver reviews that ‘feed the scientific engines’, so to speak.