Optimization for Machine Learning

Covered Topics & Materials

Covered topics and suggested (complementary) reading materials.

Date	Contents	Suggested reading material (see also bibliography in the slides)
16.4.24	Intro & Convexity	Chapter 2 of Lecture Notes Boyd: Sec. 2-3
23.4.24	Convergence Analysis of GD	Chapter 3 of Lecture Notes Nesterov's Book: Sec. 2.1
30.4.24	SGD	Chapter 6 of Lecture Notes Alternative SGD Convergence Proof
7.5.24	SGD with bounded variance assumption & Coordinate Descent	Chapter 8 of Lecture Notes
14.5.24	Adaptive Gradient Methods	Optional: [Li & Orabona] Optional: [Reddi, Kale & Kumar]
21.5.24	Distributed Optimization I, Mini-Batch & Asynchronous SGD	Recommended: [Critical Parameters: Stich, Mohtashami & Jaggi]
28.5.24	Distributed Optimization II, Local SGD	Optional: [Local SGD: Stich] More general proof: [Koloskova et al.]
11.7.24	Distributed Optimization III	Optional: [Scaffold] & [Mime]
18.7.24	Variance Reduction, SVRG	Optional: [SVRG: Johnson & Zhang] Optional: [Loopless SVRG: Hofmann, Lucchi, Lacoste-Julien, McWilliams]

 

Master Thesis Project

If you are interested in writing your master's thesis in my group, please get in touch with me with a project proposal and your most recent transcript of records.

Formal procedure: As a prerequisite for the thesis, the student is expected to attend and pass the thesis seminar, which consists of a kick-off presentation about the thesis topic and a written project proposal/outline to be delivered when registering for the kick-off presentation. See e.g. Andreas Zellers' and Christian Rossows's proposal guidelines to get an idea of what a thesis proposal could look like (we will agree on a format before you start).

Research assistant (HiWi)

I regularly offer (paid - up to 19hrs/week) research projects. These projects can be practical (pytorch) or theoretical. Please contact me with a project proposal and your most recent transcript of records.