Pruning deep neural networks for lottery tickets

Seminar: Pruning deep neural networks for lottery tickets

Deep learning has achieved major breakthroughs in a variety of tasks. Yet, it comes at a considerable computational cost, which is exaggerated by the recent trend towards ever wider and deeper neural network architectures. Instead, many problems can be solved with the help of extremely sparse neural network architectures but finding and training them is a non-trivial task. According to the recent lottery ticket hypothesis, such sparse architectures can be identified by pruning large randomly initialized neural networks. In this seminar, we will present recent algorithmic advancements in this direction, gain theoretical insights into the existence of lottery tickets, identify open problems, and discuss common challenges in the quest for winning lottery tickets.

Organization

In this seminar, students will learn to present, discuss, and summarize papers related to the lottery ticket hypothesis. Specifically, each student will get a single topic assigned to them, consisting of two papers (a lead and follow-up paper). Each student will

• write a short seminar paper on the topic assigned to them, for which the two papers on the topic serve as the starting point;
• prepare a presentation on the topic assigned to them;
• write three short reviews on papers from a different topic, and prepare questions to ask the to the presenter of this paper/topic. The reviews will be shared among the group (in particular with the presenter of the topic).

Important Dates

• Kick-off meeting in the first week of the semester (tbd) (to be held online, via zoom).
• The reviews (and questions) must be submitted during the semester, one review per month.
• The presentations will be organized in a block format during the semester break (dates to be fixed at the kick-off meeting). Participation is mandatory.
• Hand-in of report: tbd, ideally one week after the block course.

Deliverables

• 3 short reviews: (each contributes 10% of your final grade): Write a short review (max 1 page) on one of the papers (not the one that you are presenting) that addresses the following questions:
  1. What is the problem addressed by the paper?
  2. What was done before, and how does the paper improve on previous work?
  3. What are the strengths and the limitations of the techniques in the paper
  4. What part of the paper was difficult to understand?
  5. What are possible improvements or extensions of the techniques in the paper?

  In addition to your review you will have to submit 3 questions that you will ask the presenter of the paper.

• Presentation: (40%). You will prepare and deliver a 30 min presentation (followed by 15 mins question/discussion) of the paper assigned to you. You will have the possibility to get feedback on your slides before the presentation.

• Seminar Paper: (30%) You will write a seminar paper on the topic that you have presented. It must not be longer than 6 pages, not counting references and appendices. Note that appendices are not meant to provide information that is absolutely necessary to understand the paper, but rather to provide auxiliary material. Papers can be shorter, but in general the provided page limit is a good indicator of how long a paper should be.

Papers that were discussed last year

https://docs.google.com/spreadsheets/d/1kvCrZO5AjVad6ya_1ayw0HqcOEBpt1qfKtPDc9ljmVQ/edit?usp=sharing