So you’ve been asked to review a paper. Where do you begin?
Accept or Not?
Journal often invite reviewers based upon their expertise related to a single manuscript.
Ask yourself:
- Do I know this area well enough to give a good review?
- Do I have the time to give a good review by the deadline?
- Do I want to provide my free labor to this publisher/organization?
CS Conferences invite reviewers writ-large to a “Program Committee” (PC) in this case you have accept to be on the PC. Then you bid on papers you wish to review.
You may or may not be assigned papers that you bid on. You are requested to review all papers regardless.
Before you begin Reading
Determine the review requirements for this journal/conference? Most venues have specific instructions on what to look for. Some care about quality and innovation, others care about methods. It depends.
Know the quality of the venue. Your review standards may adjust depending on the history of the journal. In other words, the bar for acceptance will be different for different venues.
Read the paper
Then read the paper with an eye on the questions that you’ll need to answer.
I recommend printing it and using a pen to mark your comments in the margin from the first read-through. Later I will type these comments up into the formal review.
Basic checklist:
- Title: is the title informative? Too long/short? Does it accurately describe the content of the article?
- Keywords: are the keywords appropriate?
- Abstract: Is the abstract structured correctly? Does it include the objectives, hypothesis and conclusions. (The abstract is not a preface)
- Introduction: Dot he authors explain the background of the problem and tie it in with relevant studies
- Methods: Is the algorithm, study-design clearly described. Is the contribution ground-breaking innovative or minor step forward (there is a subjective gradient here)
- Experiments: Is there sufficient experimental data? What bias exists in the data (there is always some bias). Are the statistical tests well-conceived, articulated clearly and convincing?
- Results: Are figures and tables clearly marked. Are the statistical analysis somehow shown in the figures? Are the captions clear.
- Discussion/Conclusions: Are the stated conclusions appropriate, i.e., is there appropriate evidence for their conclusions. Do the authors provide and clear and sober analysis of the limitations of the study
- References: Are references up to date and formatted correctly?
Note that a few things are intentionally missing from this list:
- Grammar: Please be kind to non-native writers. The ability to communicate is important, however the grammar is not (typically) part of scientific merit. However, reviewers should point out weirdness in word choice or grammar as they see it to help improve the paper.
- Did the authors not cite your relevant paper? Too bad. Please don’t be the reviewer that asks authors to cite your papers. It’s annoying.
Decision time
Reviewers are typically ask to provide a recommendation:
For many journals the decision is reject, major revision, minor revision, accept
- If a reviewer suggests reject, then the paper will likely be rejected.
- If a all reviewers suggest major revision, then the paper will likely undergo revision and will likely be eventually accepted (though, not always)
- Minor revision means that the editor may often choose not to send the paper back to the reviewers and simply handle the revision them self.
- Accept means that the paper can be accepted without further revision
Reviewers are also provided an anonymous feedback box so that they can communicate with the editor privately with additional notes.
There is often a box for reviewer confidence.
CS Conferences sometimes have a sliding scale from -2 (strong reject) to 0 (borderline) to +2 (accept). Or maybe 0-6 or -3 to +3 or various versions of this.
Then the associate editor or SPC makes a decision
CS Conferences are accept or reject (long vs short, oral presentation vs poster)