Exercise 1 (out of 14)

Getting started with supervised learning, incl. installation of some tools.

We will use Google Colab Links to an external site., which allows you to run (small) notebook examples. This requires no installation on your computer (though you need to have a Google account).

However, it will also be useful for you to install some software tools that are used in the course. This allows you to run code on your own computer and learn to handle such tool infrastructure. The installation procedure depends on which OS you are running, but you can easily find good installation instructions for Unix, macOS, and Windows, and we will help you with installation on the exercise if you have problems. One recommendation is to install Conda (Anaconda/Miniconda), which is a tool for handling virtual environments and packages in Python.

Some additional notes/instructions on working with Conda virtual environments in Python can be found here: virtual_environment_intro.pdf Download virtual_environment_intro.pdf (virtual_environment_intro.md Download virtual_environment_intro.md)

Exercise 1.1 Install the following tools (if you have problems, proceed to the next exercise)

• Anaconda Links to an external site. a distribution of python with many data science tools ready to go
• Scikit-learn Links to an external site. a machine learning library for Python
• Pandas Links to an external site. a  library providing easy-to-use data structures and data analysis tools
• matplotlib Links to an external site. a 2D-plotting library for python
• seaborn Links to an external site. a Python data visualization library based on matplotlib
• jupyter Links to an external site. which allows you to run jupyter notebooks in your browser

Be prepared to run into some software issues that might require some work to resolve. Unfortunately, this is something you have to get used to in this line of work.

You can also use the discussion forum to ask questions and help other students out if you think you know the answer.

If you get stuck with the installation and need help, you can move on and use Google Colab for the moment.

Exercise 1.2 Run the notebook ex3_python_intro.ipynb Links to an external site. either on google colab or on your local jupyter installation. It will go through some basics in a few of the packages we will be using.

(Hint: Start by copying the file to your own Google Drive (assuming you have such). This will allow you to save changes you make to the code. )

Exercise 1.3 On the lecture we studied overfitting and cross-validation, see ex3_cosinus.ipynb Links to an external site.. In the code, replace the function used with $y_{\textrm{true}} = \cos(1.5\pi x) + \sin(3\pi x)$, n_sample=50, leave-on-out cross-validation (cv=50), and replace the x-range from [0,1] to [-1,1]. Fit polynomials of order 1,...,20 and evaluate the true MSE and compare with the  estimate you get from cross-validation. Try different values of the random seed and study how stable the result is

Exercise 1.4  Assume we have a linear dynamical system satisfying

$y(t) +a_1y(t-1) + \ldots + a_n y(t-n) = b_1u(t-1)+\ldots+b_nu(t-n)+e(t)$

where $y$ is a measured output, $u$ a known input, and $e$ an unknown disturbance. Describe how estimation of system parameters $a_1,\ldots, a_n$ and $b_1,\ldots, b_n$can be formulated as a linear regression problem as on Lecture 1. (We will return to dynamical system identification later in the course.)

Exercise 1.5 We will prove the formula for the optimal parameter when ridge regression (=Tikhonov regularisation) is used for least squares optimization:

Introduce the matrix  $\overline{X} := \begin{pmatrix} X \\ \gamma^{1/2} I_p\end{pmatrix}$ and vector $\overline{y} := \begin{pmatrix} y \\ 0_p\end{pmatrix}$.

a) Show that $\Vert \overline{y}-\overline{X}\theta \Vert_2^2 =\Vert y-X\theta \Vert_2^2 + \gamma \Vert \theta \Vert_2^2$. (Hint: Use the matrix result that $\Vert \begin{pmatrix} u \\ v\end{pmatrix} \Vert_2^2 = \Vert u \Vert_2^2 + \Vert v \Vert_2^2$)

b) Show that solving the (extended) least squares problem  $\textrm{argmin}_{\theta}\; \Vert \overline{y}-\overline{X}\theta \Vert_2^2$ shows that

$\textrm{argmin}_{\theta}\; \Vert y-X\theta \Vert_2^2 + \gamma \Vert \theta \Vert_2^2 =\left(X^TX+\gamma I\right)^{-1}X^Ty$

The left hand side can be rewritten as a traditional LS problem $\textrm{argmin}_{\theta}\; \Vert \bar y-\bar X\theta \Vert_2^2$ by clever choice of extended matrices  $\bar y \textrm{ and }\bar X$. Then use the old LS-formula $\widehat{\theta} = \left(\bar X^T\bar X\right)^{-1}\bar X^T\bar y = ...$ and simplify the result.

Solutions: sol1.pdf Download sol1.pdf