Exam Aug 2023

Solutions are available here: exam2023Aug_solutions.pdf Download exam2023Aug_solutions.pdf

----------

Instructions:

Name files handed in to Canvas using your anonymization code, such as NR.zip or NR-problem1.pdf etc (this facilitates matching of any handwritten solutions to the correct person in canvas during the anonymized exam grading).  Any handwritten solutions  should be marked with both your anonymization code and a personal identifier chosen by you.

All solutions must be well motivated.  Code should be understandable, commenting will help

Preliminary limits for grades:   3: 25, 4:33, 5: 42.

Good luck !

------------------------------------------------------------------------------------------------------------------------

1. Dimensional Analysis [6p]

When a flowing liquid breaks up into small droplets, such as when using a spray can, the resulting droplet diameter $d$ [m] is assumed to depend on

• liquid velocity $v$ [m/s]
• density $\rho$ [kg/m^3]
• viscosity $\mu$ [kg/(m*s)
• surface tension $\sigma$ [kg/s^2]
• outlet diameter D [m]

Show that dimensional analysis indicates a relationship of the form

$\frac{d}{D} = f(\Pi_1, \Pi_2)$

with two dimensionless variables $\Pi_1$and $\Pi_2$ (that depend on $v,\rho, \mu,\sigma, D$, but do not depend on d).

(You should not aim to determine the function f, it needs to be found from experiments).

------------------------------------------------------------------------------------------------------------------------

2. Logistic Regression for Classification [6p]

(In this problem you do not need to write any code, the problems can be solved by pen and paper).

The data in the picture below is used to fit a logistic regression model

$Prob(y=1 | x, w) = \sigma(w_0 + w_1x_1 + w_2x_2)$

where $\sigma(z) = 1/(1+\exp(-z))$ denotes the sigmoid function.

The two input features are $x_1$ and $x_2$. The output is known to be $y=0$ for the red samples(o) and $y=1$ for the blue(+).

a) Explain briefly how one can fit the model above by maximum likelihood,  i.e. find the weight vector $w=[w_0 ; w_1 ; w_2]$ through minimizing a certain loss function

$J(w) = -L(w,D_{train})$

Describe $L$. Also describe how the obtained model can be used for classification of future data $x$.

b) Sketch a possible decision boundary corresponding to a good weight vector $w$, when trained as in a) on the data in the figure. How many classification errors does you method make on the training set? (You can make a copy of the figure and draw the decision line by hand).

c) Now suppose we regularize the $w_0$ parameter, i.e. we minimize

$J_0(w) = -L(w,D_{train}) + \lambda w_0^2$

Assume $\lambda$ is a very large number, so $w_0$ is regularized down to 0, but the other parameters are unregularized. Sketch a possible decision boundary. How many classification errors does you method make on the training data?

d) Now suppose we instead heavily regularize the $w_1$ parameter, i.e. we minimize

$J_1(w) = -L(w,D_{train}) + \lambda w_1^2$

Sketch a possible decision boundary. How many classification errors does you method make on the training data?

e) Same question as in c, and d) but with heavy regularization on the $w_2$ parameter.

$J_2(w) = -L(w,D_{train}) + \lambda w_2^2$

Sketch decision boundary and determine number of classification errors on the training data.

------------------------------------------------------------------------------------------------------------------------

3. Supervised Learning - linear and nonlinear regression [12p]

This google colab file Links to an external site. studies prediction of the strength of concrete based on 8 different input features. In total 824 different experiments have been performed. The data is given in the form of a 824*9 matrix, where the first 8 columns contain input features and the last column is the output,  a numerical value y related to the the concrete strength.  The goal is to predict $y$ from the corresponding input features.

The file contains two initial attempts: A linear regression achieving mean absolute prediction error (MAE) around 8 and a kernel-based linear regression with Tikhonov regularization (so called kernel ridge regression) giving MAE around 13.

a) Improve the code and the prediction performance (accuracy) of the Kernel Ridge method.

b) Also try at least one more method, of your own choice.

------------------------------------------------------------------------------------------------------------------------

4. Causal Inference [8p]

The following linear equations describe a structured causal model.

Here na,nx,ny,nb,nc are N(0,1) independent random variables and c1,c2,c3,c4,c5 unknown coefficients. In this problem we are interested in the causal effect of X on Y.

a) Draw a DAG describing the model, and determine $\frac{\partial }{\partial x}E[Y \mid \mathbf{do}(X:=x)]$, i.e. the causal effect of X on Y (as a function of c1, c2,c3,c4, c5).

Which of the following ordinary least squares identifications will give a correct estimation for this causal effect (found as the estimated coefficient before the X variable)

b) Y ~ X

c) Y ~ X + A

d) Y ~ X + B

e) Y~ X + C

f) Y ~ X + A + B

g) In all the suggested LS models above, a bias will be included in the estimation. Is this a problem for the correct estimation of the causal effect? Note that no such bias was present in the correct structural causal model.

Motivate your answers.

(Feel free to generate data yourself to verify your conclusions)

------------------------------------------------------------------------------------------------------------------------

5. System Identification [12p]

The file sysiddata230817.mat Download sysiddata230817.mat contains some data from a linear system with one input u and one output y sampled at the rate h=0.01.

The code sysidproblem230817.m Download sysidproblem230817.m contains an initial investigation of the data and some not so successful identification of ARX models.

Identify a discrete time linear model of the system. Aim for using few model parameters. Be sure to describe your methodology, including outlier analysis, choice of suitable model structure and model order, and include model validation with residual analysis. Also hand in your matlab code.

(Hint: Useful commands might include help ident, systemIdentification, arx,oe,armax,bj, present, compare, resid, bodeplot,pzmap,...)

------------------------------------------------------------------------------------------------------------------------

6. System Identification Theory [6p]

We want to estimate parameters $\alpha$and $\beta$ in a nonlinear input output relation of the form

$y(t) = \alpha \exp(u(t)) + \beta \exp(-u(t)) + e(t)$

where  the input $u(t) \in N(0,1)$ and the measurement noise $e(t)\in N(0,1)$ are independent normally distributed samples. The signals $y(t)$ and $u(t)$ are measured for $t=1,\ldots N$.

a) [1p] Describe how to use least squares linear regression to estimate the true parameter vector $\theta_0 = \begin{bmatrix} \alpha \\\beta \end{bmatrix}$.

b) [5p] Describe the asymptotic statistical properties of the estimation error $\hat \theta_N - \theta_0$, when $N\to \infty$: Will the estimate be bias-free? Will it be Gaussian? Also calculate the asymptotic behavior of the error  covariance matrix  $E[(\hat \theta_N-\theta_0)(\hat \theta_N-\theta_0)^T]$ as $N\to \infty$.

Hint: For a normally distributed variable   $u(t) \in N(0,1)$  it holds that $E(\exp(\gamma u)) = \exp(\gamma^2/2)$.