How to compare variance explained in two regression models?

Question

Let's say I have a measurement of height of 200 individuals as $y_1$ and fit a simple linear regression model as:

$y_1 = x_1 + x_2 + \epsilon_1$

where $x_1$ and $x_2$ are some subject metadata

I would use $R^2$ or $R^2adjusted$ to quantify the variance of $x_1$ and $x_2$ explained by $y_1$

Let's say I have a new measurement of height of the same 200 individuals (from a different measurement technique) as $y_2$ and a fit a simple linear regression model as:

$y_2 = x_1 + x_2 + \epsilon_2$

Same as above, I would use $R^2$ or $R^2adjusted$ to quantify the variance of $x_1$ and $x_2$ explained by $y_2$

Here, since $x_1$ and $x_2$ are subject metadata they are same in both models.

How should I compare the difference in variance explained by $y_1$ and $y_2$? Simply, I could use the difference in $R^2$ but I am not sure if this approach is correct and also could not think of a proper way to test the significance of the difference in $R^2$.

In other words, how would I pick a measurement method based on the variance (of $x_1$ and $x_2$) explained?

Answer 1

First, note that $R^2$ describes the proportion of variance in $y$ explained by the $x$ features.

Getting to your question, I would use an indicator variable $m$ (for “measurement technique”) to indicate which measurement technique was used. Then you wind up with $400$ observations and a regression like:

$$ y=\beta_0+\beta_1x_1+\beta_2x_2+\beta_3m+\epsilon $$

Then you test if the $m$ variable is a significant contributor. This is the $t$-test of the coefficient on $m$, and this is common to have in software outputs.

With $400$ points, you might have enough observations to interact $m$ with the $x$ features, resulting in a regression like:

$$ y=\beta_0 +\beta_1x_1 +\beta_2x_2 +\beta_3m +\beta_4x_1m +\beta_5x_2m +\epsilon $$

A reasonable test in this case would be a “chunk test” of the coefficients that involve $m$, the common one in this situation being a partial $F$-test of nested models with a null model $y=\beta_0+\beta_1x_1+\beta_2x_2$ nested within the above equation.

In fact, the $t$-test also can be seen as a chunk test, with the same null model nested within $y=\beta_0+\beta_1x_1+\beta_2x_2+\beta_3m$.

To measure the effect size, you have the $R^2$ (or adjusted $R^2$) from the model with $m$ (either alone or with interactions) and the model that excludes $m$. Consider their difference.

Throughout all of this, whether you model with $m$ or not, you use all $400$ observations.

In more generality than just your question, understanding chunk tests reveals a lot of beauty in statistics, as many common procedures are special cases of testing nested models.