Proof of Linear Regression Formulas

Nerd Cafe

Goal: Find The Best Fit Line

We want to model a linear relationship between variables:

$$\hat{y_{i}}=mx_{i}+b$$

Where:

• $\hat{y_{i}}$ is the predicted value,

• $x_{i}$ is the observed input (independent variable),

• $y_{i}$​ is the actual output (dependent variable),

• $m$ is the slope,

• $b$ is the intercept.

Objective: Minimize the Total Squared Error

The error (residual) for each point is:

$$e_{i}=y_{i}-\hat{y_{i}}=y_{i}-(mx_{i}+b)$$

We want to minimize the sum of squared errors:

$$E=\sum_{i=1}^{n}(y_{i}-(mx_{i}+b))^{2}$$

Step 1: Minimize Error Function (E)

We treat $E$ as a function of $m$ and $b$ :

$$E(m,b)=\sum_{i=1}^{n}(y_{i}-mx_{i}-b)^{2}$$

To minimize $E$, take partial derivatives of $E$ with respect to $m$ and $b$, and set them to zero.

Step 2: Partial Derivative with Respect to m

$$\frac{\partial E}{\partial m}=\frac{\partial }{\partial m}\sum_{i=1}^{n}(y_{i}-mx_{i}-b)^{2}$$

Use the chain rule:

$$=\sum_{i=1}^{n}(2)(-x_{i})(y_{i}-mx_{i}-b)=-2\sum_{i=1}^{n}x_{i}(y_{i}-mx_{i}-b)$$

Set this derivative to 0:

$$-2\sum_{i=1}^{n}x_{i}(y_{i}-mx_{i}-b)=0\Rightarrow \sum_{i=1}^{n}x_{i}(y_{i}-mx_{i}-b)=0\;\;\;\;(1)$$

Step 3: Partial Derivative with Respect to 𝑏

$$\frac{\partial E}{\partial b}=\frac{\partial }{\partial b}\sum_{i=1}^{n}(y_{i}-mx_{i}-b)^{2}$$

Use the chain rule:

$$\sum_{i=1}^{n}(2)(y_{i}-mx_{i}-b)(-1)=-2\sum_{i=1}^{n}(y_{i}-mx_{i}-b)$$

Set this to zero:

$$\sum_{i=1}^{n}(y_{i}-mx_{i}-b)=0\;\;\;\;(2)$$

Step 4: Solve the System of Equations

Equation (2):

$$\sum_{i=1}^{n}(y_{i}-mx_{i}-b)=0\Rightarrow \sum_{i=1}^{n}y_{i}-m\sum_{i=1}^{n}x_{i}-nb=0\Rightarrow b=\frac{\sum_{i=1}^{n}y_{i}-m\sum_{i=1}^{n}x_{i}}{n}$$

Plug last into (1):

Equation (1) becomes:

$$\sum_{}^{}x_{i}y_{i}-m\sum_{}^{}x_{i}^{2}-b\sum_{}^{}x_{i}=0$$

Substitute 𝑏 from equation (3):

$$\sum_{}^{}x_{i}y_{i}-m\sum_{}^{}x_{i}^{2}-(\frac{\sum_{}^{}y_{i}-m\sum_{}^{}x_{i}}{n})\sum_{}^{}x_{i}=0$$

Multiply the right-hand term:

$$\sum_{}^{}x_{i}y_{i}-m\sum_{}^{}x_{i}^{2}-\frac{\sum_{}^{}x_{i}\sum_{}^{}y_{i}}{n}+m\frac{(\sum_{}^{}x_{i})^{2}}{n}=0$$

Now collect terms with 𝑚 together and simplify:

$$m(\frac{\left( \sum_{}^{}x_{i} \right)^{2}}{n}-\sum_{}x_{i}^{2})=\frac{(\sum_{}^{}x_{i})(\sum_{}^{}y_{i})}{n}-\sum_{}^{}x_{i}y_{i}$$

Multiply both sides by −1 to clean the left-hand term:

$$m=\frac{n\sum_{}^{}x_{i}y_{i}-\sum_{}^{}x_{i}\sum_{}^{}y_{i}}{n\sum_{}^{}x_{i}^{2}-(\sum_{}^{}x_{i})^{2}}$$

Final Formulas

Slope:

$$m=\frac{n\sum_{}^{}x_{i}y_{i}-\sum_{}^{}x_{i}\sum_{}^{}y_{i}}{n\sum_{}^{}x_{i}^{2}-(\sum_{}^{}x_{i})^{2}}$$

Intercept:

$$b=\frac{\sum_{i=1}^{n}y_{i}-m\sum_{i=1}^{n}x_{i}}{n}$$