1

I have one experimental data set and two fits to it. Both fits were generated from the same model - the fits are different because the parameters of the model are different.

The fit from set 1 is better in terms of chi^2 of fit, but visually the difference doesn't seem big AND this set of parameters has one parameter being a bit out of range of what is usually observed experimentally. Close enough to still be realistic, but far enough to require some solid arguments to accept it.
Because we had doubts about this, I performed another modelling run in which the offending parameter was hard restricted to stay in expected range. This resulted in set 2 with very similar values of parameters to set 1 and the fit is a bit worse in terms of chi^2.
Note, that fitting procedure is computationally very expensive and requires a lot of manual work - the parameter/fit space is hard to navigate.

What I would like to know then: is the difference in quality of fit statistically significant? If it is not, then I would reject the set 1 on the basis of unphysical parameter and my life would be simple and easy.
If the difference IS significant however, then we need to start to question our understanding of the meaning of that parameter.
NOTE: while the value of the offending parameter is 'well known' from simple samples, it its not currently possible to measure it directly from our complicated multi-component sample. If set 1 is significantly better, then we need to start thinking if it is possible for the interactions of the components to affect the value of that parameter - in fact, i think it is possible, but i need to convince my boss ;)

My search so far made me only confused:
I do not have however much experience or education in statistics, and the language is very opaque for me. Log-likelihood test seems to be used for nested models - but i only have ONE model. Chi^2 again requires difference in number of parameters - but the number of parameters is the same, it is THE EXACT SAME model, only the values of parameters differ, and only slightly, and in the second case the range of one parameter was restricted.

How can I test if the quality of fit is better from set 1 than set 2? Links to resources introducing the procedure would be greatly appreciated.

Maciej
  • 111
  • Do you have other fit statistics available, such as RMSE (proxy for average magnitude of error) or R-squared (the model's explanation of dependent data variance)? – James Phillips Mar 24 '19 at 14:40
  • If the $\chi^2$ values are similar, I would go for the second model. Nevertheless, sometimes the values of the parameters of the model may not be easily interpreted, e.g. due to interaction terms being present, or due to some degree of collinearity, or due to the intercept (in a regression) being in a region far away from the "centre" of the data. – Ertxiem - reinstate Monica Mar 25 '19 at 00:22
  • @JamesPhillips I have experimental standard deviations for each datapoint - the datacurve is in fact an average of a few dozen - few hundred curves. – Maciej Mar 25 '19 at 08:49
  • @Ertxiem - but then the question is, what is 'similar'. It would be better to have some hard arguments :) – Maciej Mar 25 '19 at 08:58

0 Answers0