8

In one of his lectures Geoff Hinton explains that a big problem of sigmoid belief nets is the explaining away phenomenon. I didn't fully understand this. I see that the induced width of the graph representing Variable Elimination might increase due to the fill-in edges being introduced at the V structures. But is there a more intuition-based way to see this, rather than an appeal to the rigor of the induced graph?

Consider a simple example where the RVs are: did an earthquake occur, did a burglary occur, and did an alarm trigger. The graph is a V structure with the Earthquake and the Burglary as the parents and the Alarm as the child. If a human being were to try and reason about the true $P(Alarm|Earthquake)$ or $P(Alarm|Burglary)$ what sort of difficulty would this V-structured relationship cause?

Andre Silva
  • 3,080
user3246971
  • 425
  • 2
  • 9
  • What is the data/information that is being used to make the inference? – probabilityislogic Sep 24 '14 at 05:40
  • lets say we can collect with regularity data-vectors of the form: was there an earthquake in the last T hours?,was there a burglary in the last T hours?, did the alarm ring in in the last T hours? – user3246971 Sep 24 '14 at 06:27
  • Well, if that is your data - then any inference is difficult (sample size is 1). You have also incorporated the notion of time into your data, but not into your model. – probabilityislogic Sep 24 '14 at 10:09
  • There is no need of any notion of time, Its just an example of how you would actually, physically go about collecting such data. Thus there is the usual assumption of stationarity (though we don't need to analyse that dimension of the problem ). Also, as mentioned in the previous comment, we "collect with regularity", so you don't have any dearth of data. – user3246971 Sep 24 '14 at 10:26
  • Although I don't have an answer to this, as I too am looking for the answer, here are two links which might be helpful: https://www.quora.com/Why-does-the-phenomenon-of-explaining-away-make-inference-difficult-in-directed-belief-nets, https://www.quora.com/What-is-an-intuitive-explanation-of-the-explaining-away-problem-in-directed-graphical-models. – Sayan Pal May 26 '16 at 16:07

1 Answers1

3

The only problem I can see from such a "V structure" is that you have two "causes" - this means that there may be difficulty untangling the effect of each "parent". In terms of your example, it could be the case that you observe "earthquake" and "burglary" together (both "on" or both "off"). So you would get a good estimate for the value $ P (alarm|earthquake \cup burglary) $ but not the marginal effects. This is the equivalent of multicolinearity in linear regression. If you are using the vector $ (e_i, b_i, a_i) $ (e for earthquake, etc) and assuming a binomial model your probability for the ith data point is $ p_i=e_i\theta_e + b_i\theta_b $. The probabilities in question are $ Pr (alarm|earthquake)=\theta_e + Pr (burglary|earthquake)\theta_b $ and then $ Pr (alarm|burglary)=\theta_b + Pr (earthquake|burglary) \theta_e $

If you only observe them together $ (e_i=1, b_i=1) $ or $(e_i=0, b_i=0) $, then both probabilities are $\theta_b + \theta_e $. This means that the data is unhelpful in choosing which variable to keep.

probabilityislogic
  • 24,971