8

Consider the nested logistic regression model’s following interpretation.

A person repeatedly chooses between two different options. These options feature different levels of the same features (e.g. car characteristics). In some cases, the person will choose option 1, in some cases the person will choose option 2.

The standard formula of the logistic regression

$$P(Y=1|X=x_{i}) = \frac{e^{\beta x_{i}}}{1-e^{\beta x_{i}}}$$

could be interpreted such that the person evaluates options according to an utility function $U(x_{i})$, assigning utility components $\beta$ to the characteristics of the options, and choosing the option which maximizes utility. In a logistic regression we would thus estimate the implicit values of the characteristics $x_{i}$.

Now suppose the setting would be such that the person does not make a discrete choice. The person could, for example, say that both options are equally attractive, that option 1 is much more attractive than option 2, or that option 2 is just slightly more attractive than option 1 etc.

We would thus compute a logit specification model with the dependant variable not being 0 or 1, but within the interval $(0,1)$.

Is it mathematically possible to do that? I believe it is.* If you take a look at the ML-process of calculating a logistic regression, it is nowhere required for y to be dichotomous, nor is it in fact required to be in the interval of $(0,1)$. You can fit regression models with the ML-method for any objective functions you desire, question is if it will make sense or not. It won’t exactly be logistic regression anymore, but it might be something different, similar.

So the question remains: Does that make any sense? Is it valid to think of a binary choice problem the way I portrayed it, and to make use of additional information on relative weighting of preferences (if available) by assuming y \e (0,1)? Would that make better use of available information, as in the case of y ~ 0.5 we know that the valuation of the options is similar?

chl
  • 53,725
mzuba
  • 1,228
  • 1
    Err, actually you can only do ML estimation on objective functions that are also likelihoods. That's why there's an 'L'. This rules out quite a lot of functions. What you mean, I think, is that you can minimize any parameterized function you like. Indeed you can. – conjugateprior Mar 26 '12 at 14:01
  • More generally than minimizing parameterized functions, you can solve any estimating equations (EEs) you like. The EEs from logistic regression are efficient, among the class of EEs which are linear in Y, so long as $Var(Y)$ is proportional to $Mean(Y)(1-Mean(Y))$. If you have more information on the mean-variance relationship, you might buy yourself a little efficiency using EEs which reflect it. – guest Mar 26 '12 at 15:43

2 Answers2

4

There are situations where this sort of thing makes sense, for instance say you are trying to determine if someone is likely to like some particular icecream as a function of the ingredients, then you could get a sample of say 100 people and get them to taste each icecream and say whether they like it or not. If you assume the sample is from some population, then whether any particular individual likes the ice cream is a Benroulli trial, with probability that depends on the ingredients. You could either build your model with a dataset with one pattern per individual for each flavour of ice cream, or you could just have one pattern for each ice cream where the propotion of subjects that liked it was the proportion of the panel that liked it. The log-loss is the same (up to a muliplicative constant) either way. I have done this before (in a protein binding problem, that is much more difficult to explain than ice cream) and it worked reasonably well.

This suggests that the logit model may be appropriate for modelling some probabilities and some proportions, as long as they can be intepreted as arising from some form of Bernoulli experiment.

Dikran Marsupial
  • 54,432
  • 9
  • 139
  • 204
3

It definitely makes sense in the case of values between 0 and 1. Consider if you have training data with identical X but different Y. If you average the Ys for those X (and keep the proportion of those samples the original data set unchanged), you'll arrive at the same optimal solution.

Another way of thinking about it is that your labels are inherently probabilistic. For example, you are trying to summarize or speed up an existing complicated function with a log linear one. Say you have an expensive Monte Carlo simulation solution to a problem, and you want to make a fast approximation to it. You could use the simulation to generate data to train a logistic regressor, and here your labels are not going to be exactly 0 or 1.

On the other hand, trying to predict outcomes outside of the [0, 1] interval seems wrong, since they are out of the domain of the logistic function.

rrenaud
  • 1,088
  • 5
  • 11