Linear and Nonlinear Regression

We seek to build a simple linear regression model. We will have some training data, and we want to build a model for this data.

Definition

$$\hat{y}=h_{\theta }(x)={\theta }_0+{\theta }_{1}x$$

Where:

• $\hat{y}$ is the predicted $y$ values
• $h$ is the hypothesis function
• $\theta$ are the model parameters (weights). ${\theta }_j$ is the $j^{th}$ model Parameter (including the bias term ${\theta }_0$ and the feature weights ${\theta }_1$).
• $x_i$ is the feature value (High temp in this case).

We also have to choose a Loss Function, in this case we usually use something called Mean Squared Error. Our objective when fitting a model is to minimize the cost function.

We then seek to make a prediction using a learned model:

$$\hat{y}=h_{\hat{\theta }}(x)$$

And then test error using squared loss. Why squared loss? it is often not explained well, but we want to heavily penalize large errors, and squaring the error achieves this. There is a good video by Artem Kirsanov called What Textbooks Don’t Tell You About Curve Fitting

$$(h_{\hat{\theta }}(x)-y{)}^2=(\hat{y}-y{)}^2$$

In terms of our linear function equation, we are seeking to find ${\theta }_0$ and ${\theta }_1$ by minimizing MSE.

---

Linear Regression with Multiple Variables

We can extend this to generalize for any number of features:

$$\hat{y}=h_{\theta }(x)={\theta }_0\stackrel{\text{always 1 }}{\overbrace{x_0}}+{\theta }_1{x}_1+{\theta }_2{x}_2+\cdots +{\theta }_n{x}_n$$

Where $x_j$ is the $j^{th}$ feature. Our objective is to minimize the cost function, which is defined as:

$$J({\theta }_0,{\theta }_1,{\theta }_2,\dots ,{\theta }_n)=J(\theta )=MSE(x,h_{\theta }(x))=\frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^i)-y^i{)}^2$$

Simplifying our definition:

$$\underset{\text{parallelized over all samples}}{\underbrace{J(\theta )=\frac{1}{m}(X\theta -Y{)}^T(X\theta -Y)}}$$

We usually use Gradient Descent for higher order learning, but there is a closed form math solution for two variables:

Write $\theta$s, $x$s and $y$s in vector form.

$$\vec{\theta }=\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\\ {\theta }_2\\ ⋮\\ {\theta }_n\end{array}\right]\vec{x}=\left[\begin{array}{c}1\\ x_1\\ x_2\\ ⋮\\ x_n\end{array}\right]\vec{y}=\left[\begin{array}{c}{y}^{(1)}\\ y^{(2)}\\ y^{(3)}\\ ⋮\\ y^{(n)}\end{array}\right]$$

Then ${\vec{\theta }}^T\vec{x}$ becomes:

$$\left[\begin{array}{ccc}{\theta }_0& {\theta }_1& {\theta }_2\end{array}\right]\left[\begin{array}{c}1\\ x_1\\ x_2\end{array}\right]$$

And we achieve $\hat{y}$:

$$\hat{y}={\vec{\theta }}^T\vec{x}={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2$$

And our loss can finally be defined as $J(\vec{\theta })$:

$$J(\vec{\theta })=\frac{1}{m}\sum _{i=1}^m({\vec{\theta }}^T{\vec{x}}^{(i)}-y^{(i)}{)}^2$$

And our closed form solution for $\hat{\theta }$:

$$\hat{\theta }=(X^{T}X{)}^{-1}{X}^{T}Y$$

This is great, but there are a few things that make this a costly operation at large $n$. For one, taking the inverse of a matrix is a costly operation, and at very high points this closed form solution is somewhere between $O(2.4)$ and $O(3)$ (although, to be honest, it will probably still instantly, this is a two parameter operation). Additionally, if $X^{T}X$ is not invertible (i.e. singular or non-square), then this method fails. In these cases, we usually use Gradient Descent, which we will discuss next class.