1

I ran some experiments first. Afterwards, I looked for the parameter $p_{max}$ for which I can claim the chance a matrix has the ESP-property is $p_{max}$ or lower to a significance of exactly 99%. Is this correct on a more philosophical basis?

I'm doing research in computer science, and have a test to determine whether a random matrix holds the ESP-property or not.

If you run this Bernouilli experiment repeatedly, so I get a binomial distribution of matrices holding or not holding the ESP-property of which I estimate the $p$ parameter. This chance $p$ of not holding the ESP-property is really low (like 0.001%). So what I do is to find the chance $p_{max}$ a matrix holds the ESP-property, with a significance level of exactly 0.99.

Finding this value is no problem. However, are there good reasons this is a bad approach to the problem? Because basically, it is a more advanced approach of this: Is it ever good to increase significance level?

The difference in this case is I keep the significance level constant, while looking for a parameter $p$ which best explains my experimental results.

Community
  • 1
317070
  • 333
  • 1
  • 8
  • 3
    I'm having trouble understanding your question. Are you asking if it's acceptable to set your alpha level for significance at <99%, as long as you do this beforehand? – gung - Reinstate Monica Feb 09 '13 at 15:12
  • When you say "the chance of not holding the ESP-property is really low" are you referring to the chance that it holds absolutely no ESP property an in a chi-square where all O = E? – russellpierce Feb 09 '13 at 16:12
  • Are you trying to make a decision about matrices one at a time? Are you doing only one test then, or are you testing a whole bunch of matrices? – russellpierce Feb 09 '13 at 16:13
  • Are you perhaps trying to construct a 99 percent confidence interval for the underlying probability in the data generating process for creating matrices; with the constraint that the lower bound of the confidence interval is 0 (and hence you need only estimate the upper bound)? If so you need to know that your use of the term "p-value" is highly confusing. But the good news is that if I have interpreted you right, there is absolutely no problem with estimating this upper bound and reporting it as the top of a 99% confidence interval. – Peter Ellis Feb 10 '13 at 00:28
  • You are correct, I edited it and removed every reference to 'p-value'. It is the parameter p of a Bernouilli/Binomial distribution I am interested in. – 317070 Feb 11 '13 at 01:07

1 Answers1

3

I think you have confused the parameter you are trying to estimate ("p", the probability of a randomly chosen matrix having the property you are interested in) with the p-value or alpha value in significance testing in the question you have linked to. There is no relation between these different uses of "p" at all.

Further confusion is whether your "p" is the real probability, your best estimate of that proability, or the upper limit of the confidence interval that you hope contains that probability. You may need to invest in some more letters.

It sounds like you are trying to create a 99% confidence interval, with zero as the lower limit and an unknown number p as the upper limit, which contains the true probability of a matrix having that probability with 99% confidence (ie repeating the procedure infinite times, it would correctly include the true value 99% of the time).

There is absolutely nothing wrong with choosing the value of p that gives you 99% confidence in the coverage of the interval [0,p].

Peter Ellis
  • 17,650
  • Thanks, so I will report this $p$ in the publication. Your interpretation of my question was correct, sorry for the confusion. I do want to create a 99% confidence interval of my parameter. – 317070 Feb 11 '13 at 01:09