Help understanding the results of the Lilliefors test

Question

I have a set of annual rainfall data for Thailand which is gridded, so I have approximately 30x18 grid squares. I am trying to test whether the gamma distribution is suitable for my data, so I am doing the Lilliefors test on the rainfall data from each grid square. I appear to have implemented this fine in Matlab. The problem I am having is trying to understand my results. Here is an example from one grid square:

Test statistic = 0.0782
Critical value (1% significance level) = 0.0958
Critical value (5% significance level) = 0.0811
Critical value (10% significance level) = 0.0738

So from this, I think I reject the null hypothesis at the 10% level, but not at the 1% & 5% levels (as the null hypothesis is rejected if the test statistic exceeds the critical value).

Thing is, I don't really understand how I can accept something at the 1% and 5% level but not the 10% level. I am really not a statistician, so I do struggle to get my head around this stuff. I do tests for statistical significance in my work, where something can be statistically significant at the 10% level but not at the 5% or 1% level, so I think this is why I can't understand it being the other way around.

What does my result say about the goodness of fit using the gamma distribution for my data?

What's "the other way round"? If the rainfall measure really does follow a gamma distribution (that's the null hypothesis) & you decide to say it doesn't when the test statistic (which measures discrepancy of the data with the gamma distibution), is over 0.0738, then you'll be wrong 10% of the time. If that's too often you can choose a more stringent criterion, like it's being over 0.0958, & be wrong only 1% of the time. — Scortchi - Reinstate Monica, Oct 15 '14 at 15:35
It's not your question but I fit gamma distributions and I would never assess them in this way. I would use a quantile-quantile plot and look for systematic deviations from the expected structure. I would also try to think of other distributions that might do better. In this particular case, it is positive that many workers have found gammas useful for rainfall but negative that (a) your data are hardly independent of each other, which may throw off the P-value any way (b) most work seems to have fit gammas to values for specific stations, not to spatially distributed data. — Nick Cox, Oct 15 '14 at 15:41
BTW, the issues discussed here are generally relevant to testing goodness-of-fit to a theoretical distribution; &, if not for you, for others, it might be useful to note that the gamma distribution has a shape parameter & you therefore have to estimate the distribution of the test statistic by bootstrapping as explained here. — Scortchi - Reinstate Monica, Oct 15 '14 at 15:47
@Scortchi Why 'must' in relation to bootstrapping rather than (say) simulation? Lilliefors certainly didn't use the bootstrap. I fear there's something I missed that would make it necessary. — Glen_b, Oct 15 '14 at 16:15
@Glen_b: Sorry, I should've said "parametric bootstrap", i.e. simulating from the best-fit model. Lilliefors only had to simulate once from a standard exponential distribution & once from a standard normal to get his tables for each because for a family of distributions defined by scale & location parameters that are estimated from the data the distribution of the KS test statistic doesn't depend on the true values of those parameters. (And I was assuming the gamma shape parameter is being estimated from the data in this case.) — Scortchi - Reinstate Monica, Oct 15 '14 at 16:27
@NickCox it would be impossible for me to look at quantile-quantile plots for the amount of data I have - I would need to look at over 500 plots, and that is just for annual data. I also need to look at monthly and seasonal data. I need to do a test that I can do automatically for a large number of grid squares. I am not sure I completely follow the rest of what you and Scortchi said! — emmalgale, Oct 15 '14 at 16:34
@Scortchi your first comment helps me understand what the percentage values are actually referring to. I didn't realise they meant that I have a x% chance of being wrong if I reject the null hypothesis at the x% level. — emmalgale, Oct 15 '14 at 16:40
I understand now that you are looking at separate annual series for 540 grid squares, so some of my comments need modification. But I really don't see a difference. If you are willing to look at 540 Lilliefors test results (and it is to be hoped, follow up any strange results individually), then the graphs sound much more fun. You have a computer to automate a loop, group graphs into pages, etc. Conversely, if you want the tests to make automated decisions for you, and not to look at the data, then good luck, but I advise against. — Nick Cox, Oct 15 '14 at 16:45
@NickCox My program calculates the critical values and test statistics for each grid square, and then sees whether the test statistic exceeds each critical value, all automatically. It then produces a map of each % level showing which squares reject the null hypothesis. Everything done in less than 15 minutes. I will be looking at those squares that reject the null hypothesis in more detail to see why that occurs. This isn't the main focus of my work, just a small thing that helps to do something else. — emmalgale, Oct 15 '14 at 16:49
@scortchi Yep, there was actually something I missed, which I was kind of sensing before, but I was too tired to think my way through it then. — Glen_b, Oct 15 '14 at 21:32

Alecos Papadopoulos · Accepted Answer · 2014-10-15T15:54:03.383

Regarding the interpretation of the "rejection/not rejection" issue:
The $\alpha$% significance level is "the probability that the test will reject a true null hypothesis". So

"Reject at 10% level": we reject the null hypothesis knowing that there is a 10% chance that we are rejecting it wrongly.

"Not reject at 5% level": we do not reject the null hypothesis remarking that "if we were to reject it, there would be a 5% chance of the rejection being wrong".

But wait: what is the meaning of this last phrase, it seems nonsensical: why do we care about the probability of a wrong rejection since we do not reject the null in the first place?

We care, because it is the other way around: the reason of no-rejection is exactly the decision from our part to bear a probability of wrong rejection of only 5% (and not higher). In other words, exactly because we are being very "conservative", very "intolerant" towards the prospect of a wrong rejection, we find ourselves unable to reject the null.

Whether we reject the null hypothesis or not, is not only a matter of the data and the statistical methods we employ: it is as much a matter of a choice we make, a decision we take, about how high a probability of a wrong rejection we want to accept. The "usual" significance levels, 1%, 5%, 10%, do not have any statistical or in general "objective" justification whatsoever (only historical-sociological, and perhaps philosophical such).

"we reject the null hypothesis knowing that there is a 10% chance that we are rejecting it wrongly" ... if the null were actually true (and the assumptions also held). — Glen_b, Oct 15 '14 at 16:13
@Glen_b Although I think that the "wrongly" in "reject wrongly" necessarily implies "the null is assumed true", indeed, it is better to include the clarification. — Alecos Papadopoulos, Oct 15 '14 at 16:35
Thanks, I think that makes sense. I was planning on using the 5% level as that appears to be used in the literature in my field (and I use that level for my statistical significance work). At that level, I only reject the null hypothesis in 4% of the grid squares. — emmalgale, Oct 15 '14 at 16:38

Help understanding the results of the Lilliefors test

1 Answers1