Explaining The Variance of a Regression Model

Question

This may be a simple explanation (I'm hoping anyway).

I have done some regression analysis in Matlab using the regression toolbox. However, I have come across a study that states this:

"Using regression analysis, it was possible to set up a predictive model using only four sonic features that explain 60% of the variance"

The link to the article is here if needed: Article

I'm not 100% sure what this means, but I'm hoping its something simple. Also is 60% a good thing? I have tried to search for this but because there is always a percentage before the word 'variance', its hard to find answer.

Answer 1

I will try to explain this in simple terms.

The regression model focuses on the relationship between a dependent variable and a set of independent variables. The dependent variable is the outcome, which you’re trying to predict, using one or more independent variables.

Assume you have a model like this:

Weight_i = 3.0 + 35 * Height_i + ε

Now one of the obvious questions is: how well does this model work? In other words, how well the height of a person accurately predicts – or explains – the weight of that person?

Before we answer this question, we first need to understand how much fluctuation we observe in people’s weights. This is important, because what we are trying to do here is to explain the fluctuation (variation) in weights across different people, by using their heights. If people’s height is able to explain this variation in weight, then we have a good model.

The variance is a good metric to be used for this purpose, as it measures how far a set of numbers are spread out (from their mean value).

This helps us rephrase our original question: How much variance in a person’s weight can be explained by his/her height?

This is where the “% variance explained” comes from. By the way, for regression analysis, it equals the correlation coefficient R-squared.

For the model above, we might be able to make a statement like: Using regression analysis, it was possible to set up a predictive model using the height of a person that explain 60% of the variance in weight”.

Now, how good is 60%? It’s hard to make an objective judgement about this. But if you have other competing models – say, another regression model that uses the age of a person to predict his/her weight – you can compare different models based on how much variance is explained by them and decide which model is better. (There are some caveats to this, see ‘Interpreting and Using Regression’ -- Christopher H. Achen http://www.sagepub.in/books/Book450/authors)

Answer 2

The authors are referring to the $R^2$ value for the model which is given by the formula

$$ \frac{\sum_{i=1}^{n} (\hat{y}_i - \bar{y})^2}{\sum_{i=1}^{n} (y_i - \bar{y})^2} $$

where $y_i$ is the observed value, $\hat{y}_i$ the least squares fitted value for the $i^\text{th}$ data point and $\bar{y}$ is the overall mean. We sometimes think of $R^2$ as a proportion of variation explained by the model because of the total sum of squares decomposition

$$ \sum_{i=1}^{n} (y_i - \bar{y})^2 = \sum_{i=1}^{n} (\hat{y}_i - \bar{y})^2 + \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 , $$

the latter term being residual error that is not accounted for by the model. The $R^2$ basically tells us how much of the overall variation has been "absorbed into" the fitted values.