What is linear regression?

Linear regression is the most basic and commonly used predictive analysis. Regression estimates are used to describe data and to explain the relationship between one dependent variable and one or more independent variables.

At the center of the regression analysis is the task of fitting a single line through a scatter plot. The simplest form with one dependent and one independent variable is defined by the formula y = a + b*x.

Sometimes the dependent variable is also called endogenous variable, prognostic variable or regressand. The independent variables are also called exogenous variables, predictor variables or regressors.

However Linear Regression Analysis consists of more than just fitting a linear line through a cloud of data points. It consists of 3 stages – (1) analyzing the correlation and directionality of the data, (2) estimating the model, i.e., fitting the line, and (3) evaluating the validity and usefulness of the model.

There are 3 major uses for Regression Analysis – (1) causal analysis, (2) forecasting an effect, (3) trend forecasting. Other than correlation analysis, which focuses on the strength of the relationship between two or more variables, regression analysis assumes a dependence or causal relationship between one or more independent and one dependent variable.

Firstly, it might be used to identify the strength of the effect that the independent variable(s) have on a dependent variable. Typical questions are what is the strength of relationship between dose and effect, sales and marketing spend, age and income.

Secondly, it can be used to forecast effects or impacts of changes. That is regression analysis helps us to understand how much will the dependent variable change, when we change one or more independent variables. Typical questions are how much additional Y do I get for one additional unit X.

Thirdly, regression analysis predicts trends and future values. The regression analysis can be used to get point estimates. Typical questions are what will the price for gold be in 6 month from now? What is the total effort for a task X?

There are several linear regression analyses available to the researcher.

• Simple linear regression
1 dependent variable (interval or ratio) ? 1 independent variable (interval or ratio)

• Multiple linear regression
1 dependent variable (interval or ratio) ? 2+ independent variables (interval or ratio)

• Logistic regression
1 dependent variable (binary) ? 1+ independent variable(s) (interval or ratio)

• Ordinal regression
1 dependent variable (ordinal) ?1+ independent variable(s) (nominal)

• Multinominal regression
1 dependent variable (nominal) ? 1+ independent variable(s) (interval or ratio)

• Discriminant analysis
1 dependent variable (nominal) ?1+ independent variable(s) (interval or ratio)

When selecting the model for the analysis another important consideration is the model fitting. Adding independent variables to a linear regression model will always increase the explained variance of the model (typically expressed as R²). However adding more and more variables to the model makes it inefficient and over fitting occurs. Occam’s razor describes the problem extremely well – a model should be as simple as possible but not simpler. Statistically if the model includes a large number of variables the probability increases that the variables test statistically significant out of random effects.

The second concern of regression analysis is under fitting. This means that the regression analysis’ estimates are biased. Under fitting occurs when including an additional independent variable in the model will reduce the effect strength of the independent variable(s). Mostly under fitting happens when linear regression is used to prove a cause-effect relationship that is not there. This might be due to researcher’s empirical pragmatism or the lack of a sound theoretical basis for the model.