I previously posted a question about comparing distributions of categorical outcomes between groups. I haven't found a good answer to the time-dimension part of it, but tried going with simple χ2 for each time period as a starting point, to compare whether the distribution of outcomes (3 levels) is the same between comparison group[s] and a baseline group.
Background / Goal: Basically, there is a relatively well-established underlying process (everyone agrees that it is happening) that generates the distribution of outcomes in the baseline group. There is speculation that different processes may be generating the distribution of outcomes in the comparison groups - but much less agreement on whether those different processes are actually happening, and they're not directly observable in any reliable way. A priori, though, it is plausible that the exact same process generating the outcomes in the baseline group is also at work in the comparison groups, but nobody has closely examined the data. Establishing whether it is the same process or a different one (or, for instance, historically the same but recently diverging) is an important empirical question.
My argument is that if it is the exact same process, then we would expect the distribution of outcomes in the baseline and comparison groups to be very similar. That said, data quality here are generally poor and not everything is observable, and it's not strictly an either-or (same vs. different process). Also, my sample sizes are quite large (~10,000-50,000) (and also, if it matters, somewhat uneven between groups - baseline group is several times larger than comparison groups), so formal tests like χ2 result in miniscule p values that are pretty uninformative (see also these previous answers, also this discussion and this commentary).
After further thought and some helpful discussion with folks here, it seems I should instead focus on some way to quantify how similar or different the outcome distributions in baseline vs. comparison groups are. For instance, it would be meaningful if I could say, these distributions are very similar (or are not at all similar), or have been very similar for some time but are starting to diverge, for instance. Is there a good formal way to quantify / express the similarity of these categorical distributions?
(I'm also working on visually highlighting the similarity, but that's less challenging; also, I'd like a more rigorous way to demonstrate it.)
[Edit: adding example data] For instance, taking one cross-section of my data:
Outcome A B C
Group
0 5086 27817 1858
1 2160 9996 491
2 1505 8477 1664
Or, as proportions of each group:
Outcome A B C
Group
0 0.146 0.800 0.053
1 0.171 0.790 0.039
2 0.129 0.728 0.143
For this time point, for instance, it seems (and looks, when graphed) like groups 0 and 1 have very similar distributions, while group 2 is a little more different (but not wildly so) from 0. But can I put a number on these hand-wavy comparisons?
