Optimization for Learning

What is this course about?

This is a mathematical course that focuses on optimization theory and the role of optimization as a foundation for classical and contemporary supervised learning applications. This means that if you are interested in this type of mathematical theory, this is a course for you. If you are interested in applying (deep) learning to real-world problems, this is not a course for you.

Course program

The course program contains the schedule and relevant information regarding lectures, exercises, deadlines, etc. All relevant information will also be posted on this canvas page.

Before the course

Mathematical prerequisites. The course is fairly mathematical. We will in the teaching assume that you feel reasonably comfortable with the content of this mathematical prerequisites document.

Coding environments. We will program in Python. Have a look at this introduction to Python if you are unfamiliar.

Schedule

Check TimeEdit for the most up-to-date schedule.

Lecture videos

Some lecture videos (that are short flipped classroom type videos, not lecture recordings) can be found here. The videos are intended for active listening. This means pausing, skipping 15 s back (or forward) within video to repeat, e.g., an argument, skipping between videos to recall concepts, changing playback speed, and maybe taking notes and verifying calculations while watching,

Lecture slides

The slides and videos may be updated during the course.

• L0 - Intro
• L1 - Convex sets (videos)
  
• L2 - Convex functions (videos)
  
• L3 - Subdifferentials and the proximal operator (videos/videos)
  
• L4 - Conjugate functions and duality (videos/videos)
  
• L5 - Proximal gradient method - Basics (videos)
  
• L6 - Algorithms and convergence (videos)
• L7 - Proximal gradient method - Theory (videos)
• L8 - Least squares (lecture video from last year) / Logistic regression (lecture video from last year)
  
• L9 - Support vector machines (lecture video from last year)
• L10 - Deep learning
• L11 - SGD - Qualitative convergence
• L12 - SGD - Implicit regularization
• Recap

All lecture slides are collected in two different pdfs that both contain eight slides per page. In the first one, the "animation slides" have been condensed to take up one slide in the pdf. This pdf can be downloaded here and contains 83 pages. The other one is the same except that it shows all slides of all animations in sequence. This pdf can be downloaded here and contains 122 pages. You will not loose important information by using the shorter pdf, but some slides may look strange.

Extra material (not in the course this year):

• Coordinate gradient descent
• Newton's method and quasi-Newton methods

Exercise material

• Exercise compendium (New exercises will be added on a weekly basis.)
• Mathematical prerequisites
• Introduction to Python

Suggested exercises

Assignments

• Assignment 1 (Deadline for the second resubmission: 1 November)
• Assignment 2 (Deadline for the resubmission: 1 November)

If your submission is not passed: Two resubmissions are allowed on the first assignment. Only one resubmission is allowed on the second assignment. The resubmission deadlines will be posted as notifications here in Canvas.

Previous exams

• 2022-10-25 (with solutions)
• 2021-10-26 (with solutions) 
• 2020-10-26 (with solutions)
• 2019-10-28 (with solutions)

Contact information

Additional material

• Smoothness and strong convexity duality