7

I am using lmfit to fit a function in Python. This is my fit:

Log scale: enter image description here

Lin scale: enter image description here

where the histogram is my data and the dashed line the fit. The error bars are simply the sqrt of the number of counts.

I would like to decide on whether it is a good fit automatically, for this I am using the chi squared test. Despite the fit looks decent to me, I always have to reject it when performing the chi squared test. So either I am doing the test wrong or my 'eye calibration for what a good fit looks like' is wrong.

I am obtaining the value of chi square from the fit by doing result.chisqr as stated in the documentation of lmfit. For this fit I get a value of 162.8 = result.chisqr. Now to select the critical value for chi I am using scipy.stats.chi2.ppf(.95, degrees_of_freedom) where degrees_of_freedom = result.nfree (for this example it is 22 = degrees_of_freedom). This gives me a critical chi squared value of 33.9 and since result.chisqr is larger, I have to consider that this is a bad fit.

Is this actually a bad fit or am I doing the test wrongly?

I noticed that if I use result.redchi, i.e. the reduced chi squared statistic, then this seems to work. I am not sure if I have to use this or it is just a coincidence.

user171780
  • 229
  • 1
  • 4
  • 1
    The (existing) answers point in the wrong direction. The chi-squared distribution is inapplicable here due to the many bins with tiny expected counts. What one usually does is to anticipate where the left and right tails might be (you don't have to be too accurate) and make one bin out of each tail, thereby assuring that most expected counts are sufficiently large (5 is a standard rule of thumb). Alternatively, you can use the chi-squared statistic but you have to compute the p-value with simulation. (This is an option automatically offered by chisq.test in R, for instance.) – whuber Apr 13 '23 at 13:45

3 Answers3

8

In a way, the "eye test" measures an effect size but not a p-value. Looking at the histogram versus the fit line can give you a sense of "how far" the two are apart, but the p-value will depend very much on the sample size represented in the graph, which is not as immediately obvious. With sufficient sample size, arbitrarily small deviations from the true distribution may be deemed significant, which you would never be able to discern by the eye test. What you're seeing here is a "pretty good" fit with a decent amount of data - the effect size of the deviation from the chi-squared distribution is small, but we have sufficient data to be sure it's not by chance.

Oftentimes, downstream analysis that makes particular assumptions about data distribution will still work reasonably well even if the data is not truly from the assumed distribution. The eye test may mislead you about whether a difference is of statistical significance, but it's usually a reasonable indicator of whether the difference is of practical significance.

Rather than looking at p-value, you could examine a measure of effect size like phi or Cramer's V. These measures can quantify "how similar" the observed and expected distributions are, rather than "how sure" you are that they are different. It may be trickier to justify an effect size cutoff for rejecting/accepting fits, but you could perhaps use the eye test to identify meaningful differences and see if that represents a common effect size range where you'd reject by eye.

Nuclear Hoagie
  • 9,297
  • 1
    And how would you create a practical hypothesis test to accept or reject fits based on practical significance? I have reasked here. – user171780 Apr 12 '23 at 14:46
  • 3
    @user171780 You could calculate phi or Cramer's V as a measure of effect size, which will express the overall strength of association between the observed and fitted values. This will be a measure of "how far" apart they are rather than "how sure" you are that they're different - you'll reject things that seem quite different even if you don't have much data to support it, but will keep things that appear similar even if you are quite sure they're not actually identical. – Nuclear Hoagie Apr 12 '23 at 15:03
  • This comment seems like it would be good to edit into the answer, since it seems to be getting at what the OP was actually intending to ask about. – Glenn Willen Apr 13 '23 at 00:56
  • How are you treating empty bins? Are they excluded from the fit or are they assigned an error of 0? – Nicolas Busca Apr 13 '23 at 07:32
  • @NicolasBusca I am including them with 0 value. – user171780 Apr 13 '23 at 13:56
4

Your fit is pretty good. It is not perfect. In some sense, you are correct that your eye calibration is off, but that is not such a terrible issue. Basically every model is at least a little bit wrong. With sufficient data, the slightest of deviations can be caught. However, the eye test is often a powerful and useful approach when “good enough for subsequent work” falls into that “I know it when I see it” category.

It seems that you are experiencing the same kind of issue encountered with a lot of testing for distribution normality.

Dave
  • 62,186
  • 1
    For a single (or few) fits the 'eye test' is good enough, but what happens when we have hundreds or thousands of fits? This is what I am trying to solve now, some way to automate the 'eye test'. Is the chi squared test not intended for this? As you stated I am just interested in knowing whether the fit is 'good enough for subsequent work', I am not trying to spot small deviations from an expected underlying distribution but whether the fit converged to a 'decent' set of parameters or it ended up in some local minimum which sometimes gives really bad fits. – user171780 Apr 12 '23 at 13:13
3

You are keeping bins with 0 counts and assigning them an uncertainty of 0. When you measure 0 counts, 0 is a poor estimate of the uncertainty. 0 is a valid outcome even when you expect a few counts. Common practices are to remove those bins or assign them a non-zero value for the std (I would use 1.14 since it's when you expect 0 at 32% probability).

Your chi2 value should be much closer to the expectation value, and you will be able to use a chi2 test to determine goodness of fit.

Nicolas Busca
  • 171
  • For quick reference, this solved the issue: hist_errors = hist**.5 (quick estimation of error bars in histogram), now modify bins with 0 counts hist_errors[hist_errors==0] = 1.14, finally pass this to the lmfit.model object as weights=1/hist_errors. – user171780 Apr 14 '23 at 05:31