9

I am sure that everyone who's trying to find patterns in historical stock market data or betting history would like to know about this. Given a huge sets of data, and thousands of random variables that may or may not affect it, it makes sense to ask any patterns that you extract out from the data are indeed true patterns, not statistical fluke.

A lot of patterns are only valid when they are tested in the samples. And even those that are patterns that are valid out of samples may cease to become valid when you apply it in the real world.

I understand that it is not possible to completely 100% make sure a pattern is valid all the time, but besides in and out of samples tests, are their any tests that could establish the validness of a pattern?

Graviton
  • 1,015
  • Maybe you are asking a bit to much to statistic: you said "patterns that you extract out from the data are indeed true patterns, not statistical fluke" and then you ask if there is a statistical procedure to answer the question... can I send that to xkcd :) ? more seriously, I think you should try to see if the pattern are meaningfull by trying to understand if they mean something – robin girard Jul 25 '10 at 11:29
  • @robin, that would be the hard part. Sometimes when thousands of variables influencing something, it's hard to make sense of the patterns that emerged. – Graviton Jul 27 '10 at 00:09

3 Answers3

16

If you want to know that a pattern is meaningful, you need to show what it actually means. Statistical tests do not do this. Unless your data can be said to be in some sense "complete", inferences draw from the data will always be provisional.

You can increase your confidence in the validity of a pattern by testing against more and more out of sample data, but that doesn't protect you from it turning out to be an artefact. The broader your range of out of sample data -- eg, in terms of how it is acquired and what sort of systematic confounding factors might exist within it -- the better the validation.

Ideally, though, you need to go beyond identifying patterns and come up with a persuasive theoretical framework that explains the patterns you've found, and then test that by other, independent means. (This is called "science".)

walkytalky
  • 1,898
  • coming up with a persuasive theoretical framework that explains the patterns is very hard, especially when there are thousands of variables involved. Real world is a complicated business. – Graviton Jul 20 '10 at 04:33
  • 2
    @Ngu Nobody promised it would be easy. Expecting meaningful patterns just to magically emerge from a sea of data is what gives data miners a bad name. – walkytalky Jul 20 '10 at 05:29
  • +1 for this answer. I'll embellish it by saying that "meaningful" should also be considered in the context of decision making. If a given pattern is reliably extracted from the data generating process, is there a business/research decision that will be made differently by its presence? If so, you can estimate the Value of Information with respect to the effort required to do the data mining and the potential (monetary) gain from confidently making a different decision. – Josh Hemann Sep 12 '10 at 03:56
6

You could try:

Shane
  • 12,461
3

One way to test patterns in stock market data is discussed here. A similar approach would be to randomise the stock market data and identify your patterns of interest, which would obviously be devoid of any meaning due to the deliberate randomising process. These randomly generated patterns and their returns would form your null hypothesis. By statistically comparing the pattern returns in the actual data with the returns from the null hypothesis randomised data patterns you may be able to distinguish patterns which actually have some meaning or predictive value.

babelproofreader
  • 4,819