Linear Regression

Introduction to Linear Regression

Recall that the equation of a straight line is given by \[y=a+bx,\] where \(b\) is called the slope of the line and \(a\) is called the \(y\)-intercept (the value of \(y\) where the line crosses the \(y\)-axis).

As mentioned above, some quantities are related to others in a linear way. For example, the price of mangos. In the United States, the price per kilogram of mango is approximately \(\$1.80\), so the price for \(2\) kilograms would be \(\$3.60\). Thus, the relationship between the price and the weight of the mango is given by the equation \[y=1.80x,\]

where \(x\) is the number of kilograms (the independent variable) and \(y\) is the price (the dependent variable).

Now, suppose you are interested in people's addiction to cell phones. So in your afternoon arts class, you asked \(5\) people how many text messages they sent over the duration of the class, and you got the following information.

Table 1. Ages and number of text messages data.

Is there a relationship between a person's age and the number of text messages they send? Is the relationship linear? How can I find the equation that relates them?

Definition of Linear Regression

Before defining what linear regression is, let's look at the following scatter plot showing the distribution of the data obtained in the text message example.

Figure 1. Scatter plot of the relationship between age and text messages sent

At a glance, you can see that you can draw several lines that can approximate the behavior of the points.

With linear regression, you can make a prediction of the data you do not know, from the behavior of the data you obtained in the sampling.

Correlation coefficient

One way to know if two sets of data are linearly related is to look at the scatter plot. The other is by calculating the Pearson correlation coefficient or correlation coefficient.

Let \(x\) and \(y\) be the independent and dependent variable, respectively. If \(\mu_{x}\) and \(s_x\) are the mean and the standard deviation of the sample of \(x\) and \(\mu_{y}\) and \(s_y\) are the mean and the standard deviation of the sample of \(y\).

Then, if the sample has size \(n\), the correlation coefficient is calculated by:

\[r=\frac{\sum z_xz_y}{n-1

where

\[z_x=\frac{x-\mu_{x}}{s_x}\,\text{ and }\, z_y=\frac{y-\mu_{y}}{s_y}\]

Properties of the correlation coefficient

1. The correlation coefficient takes values from \(-1\) to \(1\).

2. If \(|r|=1\) then the relationship between the variable \(x\) and \(y\) is completely linear.

3. If \(r=0\), then there is no linear relationship between \(x\) and \(y\).

4. If \(r>0\), then when \(x\) increases, \(y\) tends to increase and when \(x\) decreases, \(y\) tends to decrease (also called positive correlation).

5. If \(r<0\), then when \(x\) increases, \(y\) tends to decrease and when \(x\) decreases, \(y\) tends to increase (also called negative correlation).

❗❗ Correlation does not imply causation.

Assumptions for Linear Regression

To apply linear regression, you first have to check the following conditions:

1. Quantitative variable condition: Correlation only applies if both variables are quantitative.

2. Straight enough condition: Look at the scatter plot and make sure your data has an approximately linear relationship. Correlation only measures the strength of a linear association.

3. Outlier condition: Outliers can ruin the correlation. When outliers are present, it is best to calculate one correlation including the outliers and another excluding the outliers.

Model of Linear Regression

The regression line is not perfect. It does not pass through all the points, some points will be above and some will be below, but it is the best in the sense that the sum of squares of the residuals (see the Residuals article for more information) is the smallest possible.

The calculations for finding the regression line are often tedious and time-consuming, that's why there are statistical software and graphing calculators that you can use to help you do the calculations.

Equation of Linear Regression

The line of best fit is called the least square regression line.

In the previous line equation, the value \(\hat{y}\) is the predicted value of \(y\) that results from substituting a particular value \(x\) into the equation. Since \(\hat{y}\) is only a prediction, the difference between the value \(\hat{y}\) and the actual value of \(y\) is called residual and is given by:

\[\varepsilon=y-\hat{y}\]

The linear regression line should only be used to predict values that are within the domain of the \(x\) values in the sample. Otherwise, in the example of text messages, you could erroneously conclude that a \(1\)-year-old sends \(55\) text messages!

Example of Linear Regression

Let's look at an example where outliers can change the regression line.

Multiple Linear Regression

Multiple linear regression is used to estimate the relationship between one dependent variable and two or more independent variables.

For example, it is known that the cost of a house depends on its size, but it can also depend on the square meters of construction, the age of the property. In this case, the dependent variable is the cost of the house, while the independent variables are the size, the square meters of construction and the age of the property.

For these cases, you can also apply regression, but because the procedure is similar, this topic will not be covered in this article.