News
Peer Review
Written: 19.01.2022 14:19
Written By: Aleksandar Bojchevski
Dear Students,
The peer review part of the seminar is next. On the materials page you will find the link to the Review Assignment document where you will see a list of the seminar papers that were assigned to you, and a link to download them. The deadline to submit your reviews is 28.01.22. Note that the the PDF file for the "Instance-level explanations" paper is potentially corrupted and may need to be fixed. The author should contact me to resolve the issue. We will upload a fixed file as soon as possible. If you notice other issues with the submissions or the assignments don't hesitate to get in touch.
General Advice. Start by carefully reading each paper, using the 3 pass approach and taking notes along the way. Think about possible criticism and concrete ways of improving upon the paper as you read it. Give specific, actionable feedback and balance the positive and negative. Be critical, but kind. Please make your review as informative and substantiated as possible. Superficial, uninformed reviews are worse than no review as they may contribute noise to the review process.
In general, your review should concentrate on the seminar paper itself, not the presented methods.
Some of the questions you might ask during the review are:
- Did the author give a good overview of the field?
- Do you feel like you get the main ideas now?
- Do you understand the methods?
- Did they make every sentence count or did they just waste your time with empty babbling?
- Do you know which methods seem promising and which ones were just a sham?
- Do you know what the current challenges in the field are, what needs to be overcome for the next big thing? Do you maybe even have an idea of how this could be done?
The reviews you write will be considered in your final grade. The reviews you receive do not influence your grade. However, the paper that receives the most positive, convincing, high-quality reviews will receive the participant-decided best paper award.
Review Form. Your review should be in in plain text format and should have 6 sections as detailed below.
1. Summary. Demonstrate that you've read and understood the paper in detail by summarizing what you feel are the most important points. Aim for precision and conciseness. This is not the place to critique the paper or to evaluate it's strengths. The authors should generally agree with a well-written summary.
2. Detailed Comments. List the strong and weak points of the paper, but also provide further input to whether (and why) you think the strengths or the weaknesses are dominating. For each point, indicate the importance of the point at hand: is this a major (important, critical) strength/weakness, or a minor one?
Make sure that you address the following aspects:
- Clarity: Is the submission clearly written? Is it well organized? If not, please make constructive suggestions for improving its clarity. Does it adequately inform the reader?
- Literature: Is the related literature appropriately discussed? If not, be specific about what is missing. Note that oftentimes it is a question of judgement of whether a result should be mentioned due to page limits. The must-mention results should be directly relevant to the topic of the paper.
- Soundness: A paper ideally makes claims, which should be well supported, either by theoretical arguments, by experimental results, or by appropriate references. Either say, the paper is sound, or list the problems. Any problem listed needs a justification. Do not just say that something is incorrect, include an explanation of why you think it is incorrect.
- Miscellaneous minor issues: List any typos, grammar, and other issues which you view as minor but should be addressed in the final version of the paper.
All other relevant details (general flaws, corrections, style, etc.) and (constructive) criticism and feedback that could help improve the work or its presentation can also go here.
3. Overall Score.
- 1. Outstanding paper: it deserves the best paper award
- 2. Very good paper: there are only a few minor flaws
- 3. Borderline paper: but has merits that outweigh flaws
- 4. Borderline paper: but the flaws may outweigh the merits
- 5. Below the standard: there are multiple major flaws
- 6. Unacceptable: there are many wrong or misleading claims
4. Score Justification. Beyond what you've written above as strengths and weakness, what were the major considerations that led you to your overall score for this paper?
5. Time Spent Reviewing. How much time did you spend reviewing this paper (in hours)?
6. Confidence Score.
- 5. You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
- 4. You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission.
- 3. You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission. Math/other details were not carefully checked.
- 2. You are willing to defend your assessment, but it is quite likely that you did not understand central parts of the submission. Math/other details were not carefully checked.
- 1. Your assessment is an educated guess. Math/other details were not carefully checked.
How to Write a Good Review. If you want to learn how to write a good review you can have a look at the reviewer tutorial below and the resources listed therein. Keep in mind that these are aimed at a slightly different audience, namely, reviewers evaluating novel submissions at a conference. Nonetheless, a lot of the advice is still quite useful.
Resources on Reviewing:
Good luck with you reviews!