Optimization Methods for Large-Scale Machine Learning Sebastian Stich

Optimization Methods for Large-Scale Machine Learning

Optimization lies at the heart of many machine learning algorithms. This proseminar teaches how to give scientific presentations, and gives an overview of modern mathematical optimization methods for applications in machine learning and data science. We will closely follow the book "Optimization Methods for Large-Scale Machine Learning" by  Bottou, Curtis and Nocedal. (available on arXiv: https://arxiv.org/abs/1606.04838).

Topics include (to be finalized in the kick-off meeting):
- Formal Machine Learning Procedure
- Formal Optimization Problem Statements
- Stochastic vs. Batch Optimization Methods
- Analyses of Stochastic Gradient Methods 
- Noise Reduction Methods
- Second-Order Methods
- Other Popular Methods
- Methods for Regularized Models

Note that a particular focus will be put on the proof framework commonly used in this domain (e.g. assumptions on the stochastic noise, assumptions on the objective functions, convergence statements and proofs). Students are expected to (learn to) explain such technical concepts.

As a proseminar's primary purpose is to learn presentation skills, the seminar will feature two presentations from each student. The first presentation will be on a chapter/section of the book, and the second one on a paper of their choice that is related to the book section that they presented already. Both presentations will be graded.

The first presentation will count towards 30% of the overall grade, the choice of the second reference will count for 10% and the second presentation itself will count for 50% of the overall grade. 10% of the grade will be determined by oral participation in the sessions. Attendance in the proseminar meetings is mandatory.

The seminar will be organized in an online format (zoom), with a few hybrid sessions allowing for in-person presentations. The presentations will be scheduled in the middle and towards the end of the semester (similar to a block format - there is no regular weekly meeting).

Kick-off meeting (online): Tuesday, April 18, 5.15-6pm

Tentative slots for the presentations: Tuesday, 4-6pm, Thursday 4-6pm (not every week, I will propose some dates in the kick-off meeting)