7

Take as an example the well-known Boston housing dataset. In the original paper, the regression model was used to model the amount people were willing to pay for improved air quality.

But the 'people' whose behaviour was being modelled was never specified. Presumably, it was not limited to 'residents of Greater Boston'. Only implausibly would it be 'humans anywhere on the planet'. Nor, 'all US residents'.

So what inference are we invited by the authors to draw about the scope they intend for their theory? Perhaps 'all residents of large US metro areas'. But - what about other First World metro areas? The first two sections of the Boston paper scrupulously avoid any reference which might suggest that they had a particular 'population' to which their 'sample' was to apply.

There is also the dimension of time - the data came from the 70s, whereas the theory is has no time limitations spelt out.

Plus - going back to first principles - how could the Boston data be treated as a sample of a population including, say, the Chicago metro area when no data from the Chicago area was considered?

Besides, lots of the covariates in the Boston regression weren't samples at all, but totals - so, for example, the proportion of blacks was supplied from census stats.

My question: how can we conceive the Boston data as supporting the geographically unlimited propositions in the paper? Is there a mathematical way of showing equivalence or approximation of the Boston data analysis to other cities? Or is it just assumed as a matter of common sense - such that the paper's authors would be amazed to think anyone could have any doubts on the matter?

DavidP
  • 183
  • 2
  • 5

1 Answers1

5

I asked myself the very same question some time ago. In particular I was reading a paper of fellow researchers. They did a study with students and in their paper they said the population was people aged 0-99.

So...

Thinking about the problem, the population to which you can generalize depends largely on the experiment. You can find examples for both very special and very general questions yourself.

The two ways of supporting generalization I can think of are as follows:

  1. Extent the experiment to cover the target population: Sound, empirical, expensive, and sometimes impossible.
  2. Argue.

Argue means that you come up with a model (A) about which factors have the most influence on your hypothesis. Then you extend the population to those subjects that don't vary that much (B) from your sample in terms of these factors.

E.g. if you think that age is an important factor in your question (and other factors don't matter that much), and you have an experiment with students of age 20-29, then you can very well generalize to a general population of people aged 20-29. But generalizing to all age groups is not well supported.

You see, that this method contains two largely hypothetical components A and B that are usually not supported empirically or theoretically themselves. Also (not having read much, not to say almost none) publications with statistical content myself, I don't think authors explicitly argue along this way: providing A and B explicitly.

But

That's the way it goes. Apparently there is no other good way. So in the end, I think the best you can do is this:

  1. Descriptively state how your sample was obtained. This includes at least a description of your sample (e.g. students aged 21-37) along with all peculiarities that might matter (e.g. all were studying chemistry) and a description of your inclusion/exclusion guidelines (e.g. people aged 15-99).
  2. Optionally state to which population your data might generalize (e.g. people aged 15-99). Argue why you think this is the case.

Disclaimer

I have never done an experiment myself, nor did I learn how to do it, nor did I read many empirical works.

ziggystar
  • 1,594
  • [Continuation!] As a statistical novice, I struggle sometimes to understand how some procedures can be rigid with phalanxes of high-level maths (that I don't understand) up to a point, only to need resolving at the sharp end by what amount to informed hunches. – DavidP Jul 28 '13 at 21:35
  • @DavidP Maybe see it that way: If you don't use those statistical procedures, you have one more source of error. In particular, making use of statistics lets you avoid conclusions that are provably wrong (i.e. due to chance for almost certain). Even mathematical proofs often involve a lot of informal thinking and intuition. Simply put: Common sense is a major part of science. – ziggystar Jul 29 '13 at 07:09