5

I have a bunch of data points (about 300 samples in total), each sample is a number from 1 to 3 (from {1, 2, 3}).

These numbers are supposed to come from a survey. Are there any statistical methods to determine whether these sample are generated by human graders, or are they machine generated?

wrek
  • 185
  • 1
    Can you say more? is the data supposed to be random? – Estimate the estimators Sep 17 '22 at 06:08
  • No, the data is not supposed to be random. I am just wondering what would be the realistic survey data looks like, do they follow some kind of normal distribution? – wrek Sep 17 '22 at 17:51
  • Would it be sufficient to just test if the two results are from two different groups? or do you have a dataset simulated by a "machine" that you could use to comapre against? – Estimate the estimators Sep 17 '22 at 21:27
  • 4
    There isn't enough information here to give any more than a general pessimistic answer. To make progress you need to characterize "human" responses as well as machine-generated responses. – whuber Sep 20 '22 at 12:08

3 Answers3

5

Without any further information on the stipulated sampling method for the survey or the meaning of the three outcomes, any possible response could have some from humans or a machine, and there is no statistical test to check the difference.

It is only possible to test for a difference between human and machine-generated data if you are willing to impose some assumptions on what characterises each of these things. Any test you do on that basis would only be as good as your characterisation of what looks like a human or machine-generated answer. As an example, if you were to stipulate that the survey sampling was supposed to be random (e.g., simple random sampling without replacement), it would be possible to perform a test of exchangeability to check if there is evidence in the data to falsify this. If there is strong evidence for the alternative (non-exchangeability) then you might argue that this is evidence of a machine-generated answer. Contrarily, you might decide that the opposite is true --- i.e., a human-generated survey will have some departure from exchangeability, whereas a machine-generated set of data will be "too perfect".

Ben
  • 124,856
2

There are many ways in which this could go wrong in real life, but given that it sounds like a test question for students, conceptually simplifying it is a reasonable thing to do. If the survey questions had been randomized (such that the answers (1,2, or 3) were equally likely, this would be a situation where run length testing would be useful. https://www.itl.nist.gov/div898/handbook/eda/section3/eda35d.htm

girolamous
  • 21
0

If you have labelled instances (i.e. you know if a sample is generated by machine or by human) you can train a machine learning classifier to find the pattern which separates the two classes.

Not sure if you would count a ML method as "statistical method".

Ggjj11
  • 1,237