What is $\mu_i$ in a GLM / link function

Question

Is it so that:

• $y_i$ is not a discrete value, but a range with probability density function
• Which means for the same predictor(s) value $y_i$ could have different results
• In linear regression this distribution can only be normal
• In GLM, this distribution can be any distribution from the exponential family
• distribution of a single $y_i$ has nothing to do with distribution of all $y(s)$
• $\mu_i$ is expected value of $y_i$
• In practical use, $\mu_i$ is the predicted value $y_i$, specially if dataset has only one y for given predictor(s)

Are above correct? Where am I wrong?

Based on the above I've tried simulating glm with lm in R, and it kinda works:

library(boot)
download.file("https://dl.dropbox.com/u/7710864/data/ravensData.rda", 
              destfile="./ravensData.rda",method="curl")
load("./ravensData.rda")
# download manually and loadhere if above fails
# load("/yourpath/ravensData.rda")

# calling logit(ravensData$ravenWinNum) results in 
# [1]  Inf  Inf  Inf  Inf  Inf -Inf  Inf  Inf  Inf  Inf -Inf  Inf  Inf  Inf  Inf -Inf
# [17] -Inf -Inf  Inf -Inf
# that's way too much, as inv.logit goes to 1 at 20
# so we'll write our own dummy "logit" routine
# this will give us 5 when winNum=1 and -5 when it's zero
win <- ravensData$ravenWinNum*10-5

# now we can do a simple lm
fit <- lm(win~ravensData$ravenScore)

# and get probability of win using inv.logit
fitwin <- inv.logit(fit$fitted.values)
plot(ravensData$ravenScore, fitwin)

# now glm
fitglm <- glm(ravensData$ravenWinNum ~ ravensData$ravenScore, family="binomial")
plot(ravensData$ravenScore,fitglm$fitted)

Answer 1

In GLM, your dependent variable is drawn from a distribution that depends on your independent variables. More specifically, the mean of the distribution from which the DV is drawn is related to a linear combination of your independent variables by the link function (actually the inverse link, $\mu_i = g^{-1}(\theta^Tx_i)$).

In ordinary least squares regression, the DV is modeled as a normal distribution whose mean is related to a linear combination of the independent variables by the identity function.

$$ y_i \sim \mathcal{N}(\theta^T x_i, \sigma^2). $$

In logistic regression, the DV is modeled as being drawn from a Bernoulli distribution whose mean is related to a linear combination of the independent variables by the inverse logit (expit) function.

$$ y_i \sim \mathcal{Ber}(g^{-1}(\theta^T x_i), \sigma^2), $$

where $g$ is the logit function.

In your case I think the issue is that you're trying to draw large numbers from a Bernoulli distribution with an unrealistic mean. If $$ y = x^2 + \epsilon, \epsilon \sim \mathcal{Ber}(0.5), $$ this does not mean that $y$ is being drawn from a Bernoulli distribution with mean $x^2$. It means that $y-x^2$ is being drawn from a Bernoulli distribution with mean 0.5.

Edit: Responding to comment

It is a bit more complex than that. GLM is generalized from linear models in two ways. The distribution that the data comes from can be an distribution in the exponential family, and the relationship between the mean of that distribution and the IVs can be non-linear. These two generalizations are not entirely disconnected in that certain types of distributions are intrinsically connected to certain link functions. This resource handles the subject very well, http://data.princeton.edu/wws509/notes/a2.pdf.

You have a bunch of data $\{(x_i, y_i) : 1 \leq i \leq N \}$. Our assumption is that there is a family of distributions with various means, we'll call them $D_{\mu}$ from which the $y_i$ were drawn. We believe that the mean was different for the different $y_i$, so we say specifically, $y_i \sim D_{\mu_i}$. What's more we believe that $\mu_i$ has a deterministic relationship with $x_i$, so $\mu_i = f(x_i)$. What's even more, we believe that this function lives in a family that we have parameterized by $\theta$, so for some $\theta^{*}$, $\mu_i = f_{\theta^{*}}(x_i)$.

OUR GOAL: Estimate $\hat{\theta} \approx \theta^{*}$, so that for future $x$ we can predict a corresponding $\hat{y} = f_{\hat{\theta}}(x)$, the mean of the distribution that the corresponding $y$ will be drawn from.