Optimization for Machine Learning Sebastian Stich

Optimization for Machine Learning

This course teaches an overview of modern mathematical optimization methods, for applications in machine learning and data science. In particular, scalability of algorithms to large datasets will be discussed in theory and in implementation.

This advanced lecture aims to prepare students to conduct research on this topic. An interest and the ability to understand and apply mathematical proofs is essential.

The first lecture will take place on April 17 (hybrid), building E9 1 (CISPA), room 0.02 ("showroom"). The zoom link will be send by email to registered participants.

The following lectures are scheduled to take place in the (a bit smaller, but much more comfortable) room 0.01.

 

Learning Prerequisites

• Previous coursework in calculus, linear algebra, and probability is required.
• Familiarity with optimization and/or machine learning is useful.

It is recommended to register for this course only as a master student, but attendance is also possible for bachelor students in their last semester. There are no strict rules or regulations, but the students are required to acquire missing fundamentals on their own. Please note that this course will be offered on a regular basis (yearly).

 

Program

Materials will be posted under the materials tab.

Preliminary schedule (subject to change - materials will be updated under program):

• April 17 (hybrid), L1, Introduction, Convexity
• April 24 (hybrid), L2, Gradient Descent I
• May 1, holiday, no lecture
• May 8 (only online!), L3, Gradient Descent II
• May 15 (hybrid), L4, Stochastic Gradient Descent, Non-Convex Optimization
• May 22 (only online!), L5, Non-Convex Optimization, Accelerated Gradient Descent
• May 29, holiday, no lecture
• June 5, (hybrid), L6, Coordinate Descent
• June 12 (only online!), L7, Adaptive Methods and Delays
• June 19 (only online!), L8, Distributed Optimization I
• June 26 (TBA), L9, Distributed Optimization II
• July 3 (TBA), L10, Variance Reduction
• July 10 (TBA), L11, Distributed Optimization III
• July 17 (TBA), L12, Opt for ML in Practice (overview)

Exam: (it will be announced later if the exam is written or oral, depending on the number of participants in the course)

Registration

• Registration opens on March 28.
• Students are required to register for the lecture and exam on LSF (the registration deadline is a few weeks after the first lecture, and will be visible on LSF). No assistance can be provided if you miss this deadline - no exceptions.

 

Organization

There will be one lecture per week, taking place on Mondays, 16:15 - 17:45. The first lecture is on April 17.

There will be a Q&A session with the teaching assistant (Xiaowen Jiang), on Mondays 15:00 - 16:00, room 3.21. First office hour: April 24.

We might also offer office hours on zoom, if requested.

We will use a hybrid format. All lectures can be attended online, some lectures will be held in building E9 1 (CISPA), room 0.01 (presentation room, ground floor).

 

Grading and Exam

The assignments (exercises) will in general not be graded. Nevertheless it is strongly recommended to do the exercises every week! To encourage regular work on the assignments, each student must submit 4 assigned exercises, and pass 3 of them with 'Pass' to take part in the exam.

A research project (theoretical or practical) will focus on either practical implementation and the real-world performance of one of the studied optimization algorithms or variants, or a theoretical investigation with a small extension (with proof) of one of these schemes. The project is mandatory and done in groups of 2-3 students. The projects will be graded in scale of Fail, Pass, Good (top 30%, 0.3 bonus), Excellent (top 10%, 0.6 bonus). You are required to pass the project to take part in the exam. Failed projects can be resubmitted within 2 weeks.

The final exam will be (oral or written, TBA), and will cover all the material discussed in the lectures and the topics from the assignments/exercises (but not the research projects). If you pass the exam, eventual bonus points from the project will be considered to determine the final grade.