Do we really need overparameterization in deep learning? Rebekka Burkholz

Do we really need overparameterization in deep learning?

Deep learning continues to impress us with breakthroughs across disciplines and is a major driving force behind a multitude of industry innovations like ChatGPT. Most of its successes are achieved by increasingly large neural networks that are trained on massive data sets and still achieve zero training loss. This recent trend to overparameterize neural networks defies classic concepts of statistical learning theory that suggest to avoid overfitting by reducing the number of trainable parameters. We will look into recent explanations of this puzzling phenomenon, discuss related insights, and challenge the modern belief that scaling up neural networks is always the best way to move forward.
Are the simplest models always the best choice? And is counting parameters really the best way to measure model complexity? Please join the seminar if you enjoy thinking about this kind of questions.

Important Dates

* Kick-off meeting in the first week of the semester (Apr 24, 10:15) (to be held online, via zoom).
* Two reviews (and 3 questions) must be submitted during the semester. Submission deadlines: June 12th, July 10th.
* The presentations will be organized in a block format during the semester break (Aug 24-25th). Participation is mandatory.
* Hand-in of report: Oct 2nd.

Deliverables

* 2 short reviews: (each contributes 15% of your final grade): Write a short review (max 1 page) on one of the papers (not the one that you are presenting) that addresses the following questions:
    1. What is the problem addressed by the paper?
    2. What was done before, and how does the paper improve on previous work?
    3. What are the strengths and the limitations of the techniques in the paper
    4. What part of the paper was difficult to understand?
    5. What are possible improvements or extensions of the techniques in the paper?
* In addition to your review you will have to submit 3 questions that you will ask the presenter of the paper (ideally in the same document).
* Presentation: (30%). You will prepare and deliver a 20 min presentation (followed by 10 mins question/discussion) of the paper assigned to you. You will have the possibility to get feedback on your slides before the presentation.

* Discussion: (10%). How actively you engage in discussions during the seminar will also influence your grade.
* Seminar Paper: (30%) You will write a seminar paper on the topic that you have presented. It must not be longer than 6 pages, not counting references and appendices. Note that appendices are not meant to provide information that is absolutely necessary to understand the paper, but rather to provide auxiliary material. Papers can be shorter, but in general the provided page limit is a good indicator of how long a paper should be.