Let’s Role Play in the Deep: Do we really need overparameterization in deep learning?

Let’s Role Play in the Deep: Do we really need overparameterization in deep learning?

It's time to connect to your inner kid in this fun and engaging seminar format (https://colinraffel.com/blog/role-playing-seminar.md.html): Several students read the same paper, but each student takes on a specific role, which defines the lens through which they contribute to the discussion. Students cycle through roles throughout the course of this seminar on overparameterization in deep learning.

 

Content: Do we really need overparameterization in deep learning?

Deep learning continues to impress us with breakthroughs across disciplines and is a major driving force behind a multitude of industry innovations like ChatGPT. Most of its successes are achieved by increasingly large neural networks that are trained on massive data sets and still achieve zero training loss. This recent trend to overparameterize neural networks defies classic concepts of statistical learning theory that suggest to avoid overfitting by reducing the number of trainable parameters. We will look into recent explanations of this puzzling phenomenon, discuss related insights, and challenge the modern belief that scaling up neural networks is always the best way to move forward. Are the simplest models always the best choice? And is counting parameters really the best way to measure model complexity? Please join the seminar if you enjoy thinking about this kind of questions.

 

Important Dates

* Kick-off meeting in the second week of the semester (Oct 28, 2024).
* The presentations will be organized as a bi-weekly meeting during the semester. We usually meet at 17:15 in E9 1 (CISPA main building) in Room 0.07. Participation is mandatory. Please find the schedule below:

Date	Mentor	Papers	Code
Nov 18	Celia (celia.rubio-madrigal@cispa.de)	Are Wider Nets Better?	https://github.com/google-research/wide-sparse-nets
Dec 2	Advait (advait.gadhikar@cispa.de)	Deep Double Descent: Where Bigger Models and More Data Hurt	https://gitlab.com/harvard-machine-learning/double-descent/
Dec 16 (virtual)	Tom (tom.jacobs@cispa.de)	Omnigrok: Grokking Beyond Algorithmic Data	https://github.com/KindXiaoming/Omnigrok
Jan 6	Gowtham (gowtham.abbavaram@cispa.de)	Rethinking Bias-Variance Trade-off for Generalization of Neural Networks	https://github.com/yaodongyu/Rethink-BiasVariance-Tradeoff
Jan 20	Chao & Dong (chao.zhou@cispa.de, dong.sun@cispa.de)	On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima	https://github.com/keskarnitish/large-batch-training
Feb 3	Rahul (rahul.nittala@cispa.de)	Sharpness-aware Minimization for Efficiently Improving Generalization	https://github.com/google-research/sam / https://github.com/davda54/sam

 

Deliverables

*Short presentations from the perspective of your respective role and regular presentations of your findings: (40% of your final grade).

*Discussion: (40%). How actively you engage in discussions during the seminar will also influence your grade.

*Seminar Paper: (20%) You will write a seminar paper (max. 2 pages) on your role play experience and how your perception of papers changed depending on your role