Topics in Adversarial Machine Learning Xiao Zhang

Course Objective: In this seminar, we will focus on understanding the security threats adversaries pose to machine learning systems (evasion and poisoning attacks) and the recent algorithmic advancements of building more robust machine learning systems to mitigate those threats. In addition, we will look into several theoretical works on understanding and characterizing the fundamental limits of adversarial machine learning. Registration for the seminar is not possible directly. Please use the CS department assignment system to indicate your interest and to register your bid.

Expected Background: Previous background in mathematics, statistics, machine learning, and security would be beneficial but optional as long as you are motivated and able to learn relevant fundamentals. Students in the seminar should have either a strong machine learning background or a strong security background, but you are not expected to have an extensive background in both areas. The seminar is open to ambitious undergraduate students (with permission of the instructor) and graduate students interested in adversarial machine learning research and other related topics in trustworthy machine learning. To self-assess whether this is the right course, you can read the following papers to check how much you can understand these papers and whether you are interested in the topics or not:

• Intriguing Properties of Neural Networks
• Explaining and Harnessing Adversarial Examples
• Poisoning Attacks against Support Vector Machines

Instructor: Xiao Zhang (xiao.zhang@cispa.de). My office is Room 3.12, CISPA Main Building, Stuhlsatzenhaus 5.

Meeting Time: 14:15 - 15:45 on each Wednesday

Meeting Location: Conference Room 0.07, CISPA Main Building, Stuhlsatzenhaus 5

Course Website: https://trustml-cispa.gitlab.io/seminar_aml_ss23/. All course materials will be posted on the course website. Students will be expected to provide materials to add to this site.

Course Expectations

Students enrolled in this seminar are expected to:

• Lead discussions on assigned topics during class meetings. Each week, a team of students will prepare and present a research topic in adversarial machine learning, then lead the discussion and answer questions from the audience. After the presentation, the team will also be responsible for writing a blog summary to document the in-class activities. You should get familiar with blogging mechanics for instructions on how to create a blog post. The presenting team should deeply understand the research papers related to the topic to deliver a well-structured presentation and lead an engaging discussion. I will discuss this in the kick-off meeting, assign topics, and form teams based on interests.
• Participate actively in class meetings. Each week, students who are not presenting will read two assigned research papers related to the presenting topic, write a short review for one of the papers, and prepare three well-thought questions that can contribute to in-class discussions. Reviews and questions will be shared among the group (in particular with the team charged with the presentation).
• Contribute fully to a team that develops a course-long project. Each team will hand in a final seminar paper, either a research project or a systematization of knowledge (SoK) project. I will explain this more in the kick-off meeting. Also, you may want to discuss your team project with the instructor at an early time.

Deliverables

Review (30%). Write a review for one weekly assigned research paper (the team charged with the presentation does not need to write the review). Throughout the semester, you should expect to write six reviews (each consisting of 5% of your final grades). The review should aim to address the following questions:

• What is the problem addressed by the paper?
• What was done before, and how does the paper improve prior works?
• What are the strengths and the weaknesses of the paper?
• What part of the paper was difficult to understand?
• What are possible improvements or further implications of the paper?

There are no requirements for the length of the review. However, you may want to go through the ICML 2023 Reviewer Tutorial for detailed instructions on how to write a good review. In addition to the weekly review, you should prepare three well-thought questions you want to ask the presenters during the class.

Presentation and Blog Post (40%). You will form a team of students and deliver a 45-min presentation followed by a 30-min Q&A session on the topics assigned to you in an early class. After each presentation, the responsible team must write a blog post summarizing the presentation and the discussions. Throughout the semester, your team should expect to take charge of two presentations and write two blog summaries (each consisting of 20% of your final grades).

Seminar Paper (30%). Develop a course-long team project and write a seminar paper on a topic in adversarial machine learning. This could be a research project or a systematization of knowledge (SoK) project. It must not be longer than eight pages, not counting references and appendices. Papers can be shorter, but generally, the provided page limit indicates how long a typical paper should be.

Bonus Report (5%). Write a report on a bonus topic in adversarial machine learning, summarizing your takeaways from relevant research papers and potential future research directions. The quality of the written report will determine how many bonus points you will receive (5% is the maximum).

Important Details

1. Kick-off meeting in the first week of the semester
   • Time: 16:15 - 17:30 on 13.04.2023 (Wednesday)
   • Location: Conference Room 0.07, CISPA Main Building, Stuhlsatzenhaus 5
2. We will use an online poll to assign topics and form teams based on interests. 
   • We will conduct an online poll after the kick-off meeting (19.04 - 24.04)
   • Teams will then be formed based on interests
3. We will use an online poll to decide the weekly meeting time that works for everyone.
   • We will conduct an online poll after the kick-off meeting (19.04 - 24.04)
4. Each paper review and three questions are due one day before each presentation.
5. The blog summary is due one week after each presentation day and will be posted on the course website.
6. We plan to have in-person meetings as long as possible and switch to fully online if needed. Attendance and contributions to discussions in all class meetings are mandatory.
7. You may want to discuss your course-long project with the instructor earlier in the semester (by appointment).
8. All seminar papers are due on 11.07. Based on your submission, you will receive feedback within one week and will have until 21.07 to improve your paper. Note that the first submission must already be good enough for the instructor to review. Otherwise, you will not receive full credits.
9. Reports for bonus points are due on 21.07.

List of Topics and Papers

We plan to include the following research topics in adversarial machine learning:

1. Adversarial Examples & Robustness Evaluation
2. Robustness Certification Methods
3. Robust Overfitting & Mitigation Methods
4. Robust Generalization & Semi-Supervised Methods
5. Intrinsic Limits on Adversarial Robustness
6. Targeted Poisoning Attacks & Certification
7. Indiscriminate Poisoning & Backdoor Attacks
8. Adversarial ML Beyond Image Classification

Here is the Google spreadsheet of all the topics and papers (including bonus topics) planned for this course. Note that the list of research topics might be subject to change.

Presentation Schedule

The weekly class meeting time will be determined via an online poll earlier in the semester. The finalized schedule with detailed presenter information will be posted on the course website here: https://trustml-cispa.gitlab.io/seminar_aml_ss23/schedule/.

Honor and Responsibilities

We believe in the value of a community of trust and expect all students in this class to contribute to strengthening that community. The course will be better for everyone if everyone can assume everyone else is trustworthy. The course instructor starts with the assumption that all students deserve to be trusted. In this course, we will be learning about and exploring some vulnerabilities that could be used to compromise deployed systems. You are trusted to behave responsibly and ethically. You may not attack any system without the permission of its owners and may not use anything you learn in this class for evil. If you have any doubts about whether or not something you want to do is ethical and legal, you should check with the course instructor before proceeding.