2

I have been reading that Simple OLS regression falls into two main cases:

  1. $Y,X$ are both RVs (Random Variables) , i.e., neither value can be predicted with accuracy, but only probabilistically, e.g., height vs weight where we choose pairs of values $(h_i, w_i)$ without controlling either, i.e., we just sample random pairs from observations at given moments.

  2. When Y is random but X is Mathematical (aka "Fixed"), i.e., the values are "controllable" e.g., age (controllable, we can choose any value we want) and weight.

What are the main differences between these two? I guess, e.g., correlation of Y vs X in the first case does not mean much. Do we also care about joint distribution, of having a bivariate normal $(X,Y)$ in the first case as we do in the second?

Estimate the estimators
  • 720
MSIS
  • 549
  • 3
  • 11
  • There is no joint distribution in the second case, because $X$ is not a random variable. – whuber Jun 23 '20 at 21:46
  • @whuber Perhaps your point is that this would only make sense in the first case, but then how is $\mathbb{E}[Y\vert X=x]= x^T\beta$ meaningful? – Dave Jun 24 '20 at 02:00
  • The opposite of random is "fixed". I have not come across the use of the term "Mathematical" in this sense. – Tim Mak Jun 24 '20 at 02:48
  • @Dave When $X$ does not represent a random variable, then expressions like "$E[Y\mid X=x]$" are understood as parameterizing a set of random variables $Y$ with values $x$ rather than as conditional expectations. The context usually (but not always) makes it clear what the meaning is. – whuber Jun 24 '20 at 13:11
  • @Whuber: If there is no joint, how do we conclude that the correlation is 0? Say the case of points on the standard unit circle, and take the variables X, Y=\sqrt (1-X^2). This is often cited as an example of two variables that are dependent but uncorrelated. In order to show dependence we must assume some joint to test for this, right? – MSIS Jul 20 '20 at 22:34
  • That's why correlation is a different thing than regression. Correlation analyses characterize a property of a bivariate distribution while regression analyses characterize either (a) a conditional distribution or (b) a parameterized distribution, depending on whether you view the parameter as a random variable (in a) or just a parameter (in b). – whuber Jul 20 '20 at 22:41
  • @whuber: Sorry, I meant to say that when we say X,Y as above are not independent, we have concluded that there are events E_X, E_y in the distribution of X,Y respectively so that ## P( E_X| E_Y) \neq P(E_X) ## Isn't this assuming the existence of a join? Otherwise how do we determine the dependence without a joint? – MSIS Jul 20 '20 at 22:47
  • I hope I didnt say something too stupid or otherwise wrong. Sorry if I did. – MSIS Jul 22 '20 at 01:28

1 Answers1

1

Linear regression works fine in both the situations you mention, at least in the sense that you will learn the best linear unbiased model for predicting Y when given a value of X.

Note that standard linear regression does not estimate the joint distribution p(X,Y), it only estimates the conditional distribution p(Y|X). There is no model for p(X) in standard linear regression, which would be required to produce the joint distribution p(X,Y). This means that linear regression cannot predict which future values of X you are likely to see, it can only predict the value of Y for a given X (in machine learning terminology, we hence say that linear regression is a discriminative model rather than a generative model).

James
  • 546