Trustworthy Graph Neural Networks

• Did the author give a good overview of the field?
• Do you feel like you get the main ideas now?
• Do you understand the methods?
• Did they make every sentence count or did they just waste your time with empty babbling?
• Do you know which methods seem promising and which ones were just a sham?
• Do you know what the current challenges in the field are, what needs to be overcome for the next big thing? Do you maybe even have an idea of how this could be done?

• Clarity: Is the submission clearly written? Is it well organized? If not, please make constructive suggestions for improving its clarity. Does it adequately inform the reader?
• Literature: Is the related literature appropriately discussed? If not, be specific about what is missing. Note that oftentimes it is a question of judgement of whether a result should be mentioned due to page limits. The must-mention results should be directly relevant to the topic of the paper.
• Soundness: A paper ideally makes claims, which should be well supported, either by theoretical arguments, by experimental results, or by appropriate references. Either say, the paper is sound, or list the problems. Any problem listed needs a justification. Do not just say that something is incorrect, include an explanation of why you think it is incorrect.
• Miscellaneous minor issues: List any typos, grammar, and other issues which you view as minor but should be addressed in the final version of the paper.

• 1. Outstanding paper: it deserves the best paper award
• 2. Very good paper: there are only a few minor flaws
• 3. Borderline paper: but has merits that outweigh flaws
• 4. Borderline paper: but the flaws may outweigh the merits
• 5. Below the standard: there are multiple major flaws
• 6. Unacceptable: there are many wrong or misleading claims

• 5. You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
• 4. You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission.
• 3. You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission. Math/other details were not carefully checked.
• 2. You are willing to defend your assessment, but it is quite likely that you did not understand central parts of the submission. Math/other details were not carefully checked.
• 1. Your assessment is an educated guess. Math/other details were not carefully checked.

• Reviewer Tutorial
• Last Minute Reviewing Advice
• How to Review a Paper

Good luck with you reviews!

• Focus on getting the main ideas across. The goal is to summarize existing research by either: (i) categorizing and comparing many papers in a mini survey, or (ii) focusing on a few papers and going more in depth. The paper should be able to teach the main ideas in the field to a novice reader. Since the topics are rather large and the seminar paper is short, you will have to condense the knowledge as much as possible. This means you will have to focus on the most important aspects and optimize your wording to explain these in the most compact way that is still comprehensible.
• Highlight the advantages and disadvantages of different approaches and your criticism. Are there any trade-offs, e.g. between accuracy and robustness, or between running time, memory, and performance? What are the open challenges? Does the approach generalize well? Did they overly simplify the problem? Is the theory solid or is it just based on unrealistic toy problems? Are the experiments convincing (setup, baselines)? Is it really a new model or just a minor modification of an old one? Remember, you only have very little space! So concentrate on the important criticism, don't lose yourself in the details!
• Make sure there is a consistent story and line of argument. Write short and direct sentences. Make sure there is one message per paragraph.
• Keep the exposition related to graphs and GNNs brief. Since all topics deal with GNNs you can assume that the reader is familiar with the basics of how a GNN works. However, it does make sense to formally introduce the notation you are using.
• Formal (mathematical) definitions of the main problem being tackled and the related background concepts are encouraged. Try to keep them concise but precise. You are welcome to include any theorems if they help you to make some point, or if they are particularly illustrative. However, avoid adding them just for the sake of making the paper more "mathy". There is also no need to rewrite existing proofs (just provide a reference), unless you think that the proof technique is interesting and worth sharing.
• If possible you can consider summarizing the main algorithm (or several similar algorithms) in pseudo-code, or in a single illustrative figure.
• If you are comparing experimental results from several different papers make sure that the results are comparable (e.g. same train/validation/test setup) or point out the relevant differences.
• If you did everything right you've now become a little expert in your niche and you've likely developed your own opinion. Therefore, you are welcome to include a short (optional) section with your research ideas for future work. Think about the major weaknesses of previous works and how you could solve them. This can both be general directions and specific ideas.

Important: The first version is not a supposed to be just a rough draft. The paper should be mostly finished at this point and most changes should be based on the comments from your supervisor and your peers. Of course, if you have additional ideas in the later stage you are welcome to add them.

Feel free to get in touch if you have any questions. I wish you a great start to the new year and good luck with your seminar papers!

Best,
Aleksandar