1

I have a dataset where I'm comparing overall ratings between multiple "products." It looks something like this:

Product 1: 3.5

Product 2: 4.1

Product 3: 3.7

...

Product 7: 3.3

What I'm doing is comparing Product 1 to all the others and I'd like to highlight whether or not there's a statistical difference in the ratings (if that's a thing) between Product 1 and the others. I'm not sure if that's possible.

I read that maybe using an independent t-test is the way to go, but I'm not sure if that's 100% accurate since the dataset I have isn't all from the same raters. For example, the data on Product 1 wasn't rated by the same people as Product 2.

Is there a more accurate way to do this?

DD1229
  • 111
  • Do you have the individual ratings? – Dave Nov 19 '21 at 17:47
  • Is all you have the per-product summary rating? You won't be able to do statistical analysis without knowing anything about the sample size. If a single person rated Products 1 and 2 as a 3.5 and 4.1, you can't make any statistical inference at all. If a handful of people rated them this way, it's probably not significant, but If thousands of people independently rated them in that manner, it likely will be. You won't be able to do much with just the summary ratings. – Nuclear Hoagie Nov 19 '21 at 17:49
  • @Dave I don't - just the final results/ratings of each. – DD1229 Nov 19 '21 at 17:49
  • How were you planning to do an independent t-test? – Dave Nov 19 '21 at 17:52
  • @NuclearHoagie - yeah, I just have the summary rating. We can assume that there were thousands of people rating the product, but we don't have the per-person ratings, just the overall. – DD1229 Nov 19 '21 at 17:52
  • @Dave I wasn't sure yet - I was just reading about this sort of scenario at the moment. – DD1229 Nov 19 '21 at 17:53
  • @NuclearHoagie would it be accurate to just highlight where the other products were above/below product 1? Like product 2 > product 1 we can show that it's rated higher? – DD1229 Nov 19 '21 at 17:55
  • @DD1229 That's reasonable, but be careful with interpretation. You can say the observed mean rating of one product is higher than another, but you'll lack statements about statistical significance. Without significance measures, you won't be able to comment on whether or not the evidence suggests that the "true ratings" of the products are actually different. The observed ratings may be different, but you won't be able to say if that's a real difference or just a random fluctuation in a small sample size. – Nuclear Hoagie Nov 19 '21 at 19:08

2 Answers2

0

Without having the individual measurements, I cannot see a way to do this as an ANOVA-style comparison between groups. However, analysis of your data is not impossible. The way I might proceed is to give the relative positions of the products. Of the four you listed, product $1$ is middle-of-the-pack.

Particularly if you have more than seven products (preferably more like $>100$), you can be formal about this by giving the percentiles of the products. Wouldn't it be nice to tell your boss that product $1$ is in the bottom $10\%$ of ratings but product $90$ is in the top $2\%$ of ratings?

Dave
  • 62,186
0

If the average ratings you have are based on thousands of responses, it is virtually certain that the differences are statistically reliable (i.e., statistically significant) and not due to chance or random fluctuation. The question then is are these differences practically meaningful. I suggest you use a two prong approach to reach an opinion on the question of meaningfulness of the differences in which you have an interest. First, look at the rating scale. What are the definitions of the rating scale anchors for 3 and 4. If one anchor is "acceptable"and the other "less than acceptable" that gives some meaning to the means of 3.5 and 4.1. Second, if you know anything about any two of the products that were rated, you might be able to say that the difference between these two known products is this big, so any other difference of that size has some meaning in terms of what you know about the two known products.

Joel W.
  • 3,306