0

I am working on a dataset that is unfortunately messy. The data collection period covers from 2019 to 2022 but not every participant has data in all the data collection intervals in this period. In the beginning, the data was gathered monthly but towards the end, it is collected weekly. It kind of seems each subject has reporting periods that is unique to them and was not the same with other subjects! I have grouped subjects into three larger groups (G1, G2, and G3) based on their specialties and have used Tableau and a date scaffold to create the following plots on three of the metrics that I have to report. The date scaffold is very similar to the idea presented here: https://tarsolutions.co.uk/blog/tableau-scaffolding-dates-calculating-deferred-revenue/

Here is my question: I can see that the line plot for example for "Time in Notes per Appointment" for G1 is higher than that of G2 and that is higher than G3. What sort of analysis should I use to test the statistical significance of diff. between them?

I am not sure if I have to treat it as time series or not and use something like ANOVA. What do you smart people think about this?

enter image description here

kjetil b halvorsen
  • 77,844
Vera
  • 1
  • what about Kolmogorov-smirnov 2 samples test? see this answer for reference – Mayeul sgc Jan 06 '23 at 01:44
  • Thank you very much for suggesting the KS test. I was not familiar with it and it seems very interesting. Can I use this 2-sample test to compare my 3 groups (like G1 vs G2, G1 vs G3, G2 vs G3)? – Vera Jan 09 '23 at 20:26
  • I would guess so yes – Mayeul sgc Jan 10 '23 at 01:14
  • The KS test will ignore the autocorrelation structure in the time series and treat data as i.i.d., which they are quite certainly not. Taking this into account would mean far more sophisticated modelling, and it may ultimately depend on assumptions what result you get. What do you want to achieve making a significance claim here? If you're just talking about the observed values, the visual display clearly shows a difference, so this is clearly there, regardless of any formal test. Significance tests have recently been widely criticised for being overused and misinterpreted. – Christian Hennig Jan 10 '23 at 18:36
  • Note also that with the many series you have, visual selection of which hypothesis exactly to test will bias the test outcome (as you will pick what looks most clear, so it's not random). To avoid this, any test comparing pairs of series would need to be applied on all pairs of series that could potentially be of interest, with a suitable correction for multiple testing. – Christian Hennig Jan 10 '23 at 18:39
  • A test could be applied if you had a generative model for the time series that models that "they are statistically the same", i.e., same model parameters and just random differences. Then it could be tested whether the series are compatible with it if that's what you're interested in. This should be doable but may not be easy. However, if you just see a difference and want some kind of "significance stamp" that you think you need for claiming that this is meaningful, this is not how it works. – Christian Hennig Jan 10 '23 at 18:44
  • Thank you for your answer. I see your point about autocorrelation and KS. I also get what you are saying about the bias in the visual selection. The fact is we are preparing a report about interactions of a bunch of users with an online reporting tool. We have broken the users down into 3 groups. We are comparing them over about 20 metrics. You can see 3 of these metrics in the image that I have posted. Some of them are clearly different between the 3 groups (like the green plots). Some are not really that different between the 3 groups (like the yellow line plots) – Vera Jan 12 '23 at 20:23
  • I also understand what you say about significance tests being criticized for being overused and the "significance stamp". However, my team doesn't really like it if we put "After eyeballing the results important differences were observed" in our report. We are leaning towards using mixed ANOVA or repeated measure ANOVA for this report. Do you have any opinions about those tests or any better opinions? Your help is much appreciated. – Vera Jan 12 '23 at 20:32

0 Answers0