0

I am trying to identify positive detections on the readings of a metal-detector antenna, and for that I am employing the well known Student t-test.

I am continuously filling a rotating buffer with incoming data (the test sample), and testing it against the calibration sample: t calculation then 2-tailed p significance.

My idea is that the calibration sample can have a solid cardinality (done one-off as the system starts, eg. around.. 300 samples?), while the test sample must react quickly, hence its cardinality is small (around maybe 10 or 20 samples?).

I want to test a significant deviation of the mean of the test distribution away from the calibration reference, but the p value is way too sensible and quickly goes to ~1.

QUESTION I wanted to know some opinions from the community about my current setup, along with maybe some guiding on the direction to take.

I am using the t-test for independent samples with equal variance assumptions described here. I did not try the Welch t-test yet, I have not tested the assumption on variances yet, but I see would give even higher significance, hence it would not help.

ANECDOTE: while developing the test, I was wrongly inserting a 0 value in the calibration sample, that was increasing the overall samples variance by a factor of ~100x. This way I had really satisfactory results, hence I am tempted to manually add a constant FACTOR to the calculation of my pooled variance but I really feel this is awesomely unorthodox.

Campa
  • 163

1 Answers1

1

My first thought was that significance was the problem and you just had a problem with the usual amount of false positives. However, as you say that p-value often goes to 1, there must be another problem. Here are a couple of possibilities:

  • There may be a difference between test and calibration populations and that difference can be of statistical significance but not of practical significance. If your device is good enough and its variance is low due to the instrument being faithful, it will detect very small differences without practical interest. This can be solved by modifying your alternative hypothesis: instead of testing for inequality, you can test for difference larger than a given threshold. However, that is unlikely to be the problem with such a small sample (20 units).
  • Your samples may be correlated. For example, if there is an slow drift due to small changes of temperature or orientation between the moment you take the calibration and test samples, you may be underestimating variance. A simple check could be taking several calibration samples at different moments and comparing them.
Pere
  • 6,583
  • a) "instead of testing for inequality, you can test for difference larger than a given threshold" : that is interesting: I guess this is as simple as adding an offset to the formula right? t=((E(X_cal)+threshold)-E(X_test))/(pooled_var). b) that is unlikely to be the problem with such a small sample : I observed such noisy behavior also with large test samples. c) Do you mean that a drift that is relevant within the time window of a calibration is causing the calibration variance to be smaller that what it actually is? In any case, I see no relevant drift. – Campa Mar 29 '18 at 12:43
  • a) Yes, this offset is a way to account for the threshold - assuming you are performing an unilateral test. c) A drift would be an extreme case of correlated measures. You can test for autocorrelation to make sure that your sample is independent. – Pere Mar 29 '18 at 12:55
  • Thank you very much for mentoring, @Pere, very useful. I'll wait a little more before placing the award for the best answer anyway. Thanks a lot again .. On the drift thing, I posted an other question here if you're interested! – Campa Mar 29 '18 at 13:05
  • I am not actually sure how to test for a null-hypothesis that the absolute difference between the 2 means is more than a threshold: could you please put some more detail in your answer? Is it: t=((abs(E(X_test)-E(X_cal)) - threshold))/(pooled_var) then deduce its 1-tailed significance, maybe..? – Campa Mar 29 '18 at 14:17