Topics in Adversarial Machine Learning Xiao Zhang

Course Objective: In this seminar, we will focus on understanding the security threats adversaries pose to machine learning systems (evasion and poisoning attacks) and the recent algorithmic advancements of building more robust machine learning systems to mitigate those threats. In addition, we will look into several theoretical works on understanding and characterizing the fundamental limits of adversarial machine learning. Registration for the seminar is not possible directly. Please use the CS department assignment system to indicate your interest and to register your bid.

Expected Background: Previous background in mathematics, statistics, machine learning, and security would be beneficial but optional as long as you are motivated and able to learn relevant fundamentals. Students in the seminar should have either a strong machine learning background or a strong security background, but you are not expected to have an extensive background in both areas. The seminar is open to ambitious undergraduate students (with permission of the instructor) and graduate students interested in adversarial machine learning research and other related topics in trustworthy machine learning. To self-assess whether this is the right course, you can read the following papers to check how much you can understand these papers and whether you are interested in the topics or not:

• Intriguing Properties of Neural Networks
• Explaining and Harnessing Adversarial Examples
• Poisoning Attacks against Support Vector Machines

Instructor: Xiao Zhang (xiao.zhang@cispa.de). My office is Room 3.12, CISPA 0, Stuhlsatzenhaus 5.

Course Expectations

Students enrolled in this seminar are expected to:

• Lead discussions on assigned topics during class meetings. Each week, a team of two-three students will prepare and present a research topic in adversarial machine learning, then lead the discussion and answer questions from the audience. After the presentation, the team will also be responsible for writing a blog summary to document the in-class activities. The presenting team should deeply understand the research papers related to the topic to deliver a well-structured presentation and lead an engaging discussion. I will discuss this in the kick-off meeting, assign topics, and form teams based on interests.
• Participate actively in class meetings. Each week, students who are not presenting will read two assigned research papers related to the presenting topic, write a short review for one of the papers, and prepare three well-thought questions that can contribute to in-class discussions. Reviews and questions will be shared among the group (in particular with the team charged with the presentation).
• Contribute fully to a team that develops a course-long project. Each team will hand in a final seminar paper, either a research project or a systematization of knowledge (SoK) project. I will explain this more in the kick-off meeting. Also, you may want to discuss your team project with the instructor at an early time.

Deliverables

Review (30%). Write a review for one of the weekly assigned research papers (the team charged with the presentation does not need to write the review). Throughout the semester, you should expect to write six reviews (each consisting of 5% of your final grades). The review should try to address the following questions:

• What is the problem addressed by the paper?
• What was done before, and how does the paper improve prior works?
• What are the strengths and the weaknesses of the paper?
• What part of the paper was difficult to understand?
• What are possible improvements or further implications of the paper?

There are no requirements for the length of the review. However, you may want to go through the ICML 2023 Reviewer Tutorial to see an example of how to write a good review. In addition to the weekly review, you should prepare three well-thought questions you want to ask the presenters about the topics during the class.

Presentation and Blog Post (40%). You will form a team of two-three students and deliver a 45-min presentation followed by a 30-min Q&A session on the topics assigned to you in an early class. After each presentation, the responsible team must write a blog post summarizing the presentation and the discussions. Throughout the semester, your team should expect to take charge of two presentations and write two blog summaries (each consisting of 20% of your final grades).

Seminar Paper (30%). Develop a course-long team project and write a seminar paper on a topic in adversarial machine learning. This could be a research project or a systematization of knowledge (SoK) project. It must not be longer than eight pages, not counting references and appendices. Papers can be shorter, but generally, the provided page limit indicates how long a typical paper should be.

Bonus Report (5%). Write a report on a bonus topic in adversarial machine learning, summarizing your takeaways from relevant research papers and potential future research directions. The quality of the written report will determine how many bonus points you will receive (5% is the maximum).

Important Details

1. Kick-off meeting in the first week of the semester (TBD)
2. We will use an online poll to assign topics and form teams based on interests. 
3. We will use an online poll to decide the weekly meeting time that works for everyone.
4. Each paper review and three questions are due one day before each presentation.
5. The blog summary is due one week after each presentation day and will be posted on the course website.
6. We plan to have in-person meetings as long as possible and switch to fully online if needed. Attendance and contributions to discussions in all class meetings are mandatory.
7. You may want to discuss your course-long project with the instructor earlier in the semester (by appointment).
8. All seminar papers are due on (TBD). Based on your submission, you will receive feedback within one week and will have until (TBD) to improve your paper. Note that the first submission must already be good enough for the instructor to review. Otherwise, you will not receive full credits.
9. Reports for bonus points are due on (TBD).

List of Topics and Papers

We plan to include the following research topics in adversarial machine learning:

1. Adversarial Examples & Robustness Evaluation
2. Robustness Certification Methods
3. Robust Overfitting & Mitigation Methods
4. Robust Generalization & Semi-Supervised Methods
5. Intrinsic Limits on Adversarial Robustness
6. Targeted Poisoning Attacks & Certification
7. Indiscriminate Poisoning & Backdoor Attacks
8. Adversarial ML Beyond Image Classification

Here is the google spreadsheet of all the topics and papers (including bonus topics) planned for this course. Note that the list of research topics might be subject to change.

Presentation Schedule

The weekly class meeting time will be determined via an online poll earlier in the semester. A tentative schedule for all the class meetings is summarized below. Note that this schedule might be subject to change.

---

Week	Topic	Presenters
01.05 - 05.05	Adversarial Examples & Robustness Evaluation
08.05 - 12.05	Robustness Certification Methods
15.05 - 19.05	Robust Overfitting & Mitigation Methods
22.05 - 26.05	Robust Generalization & Semi-Supervised Methods
29.05 - 02.06	Intrinsic Limits on Adversarial Robustness
12.06 - 16.06	Targeted Poisoning Attacks & Certification
19.06 - 23.06	Indiscriminate Poisoning & Backdoor Attacks
26.06 - 30.06	Adversarial ML Beyond Image Classification

---

Honor and Responsibilities

We believe in the value of a community of trust and expect all students in this class to contribute to strengthening that community. The course will be better for everyone if everyone can assume everyone else is trustworthy. The course instructor starts with the assumption that all students deserve to be trusted. In this course, we will be learning about and exploring some vulnerabilities that could be used to compromise deployed systems. You are trusted to behave responsibly and ethically. You may not attack any system without the permission of its owners and may not use anything you learn in this class for evil. If you have any doubts about whether or not something you want to do is ethical and legal, you should check with the course instructor before proceeding.