Let's there are 2 groups of user. Based on their query I return some search results to them (p1,p2,p3). The search results could vary based on either the group that user belongs to or some user specific parameter. I want to measure, whether the search results to the users are significantly different to each other or not for a given query (let's say when there is difference of more than 7 results out of first 10 results).
Are there any online learning algorithm/test to do this?
Here is what i am planning so far, group all the requests by groupid, query. Calculate the frequency distribution of search results by groupid for a given query. Measure how different are the frequency distributions (open question here). Are there other techniques this can be achieved? What are the pitfals to be aware of for example what if there is a skew in number of users lying in group1 vs group2. What if there are n groups instead of just 2.
Literature suggestion are also welcome.
