Grouping trials decreases standard error?

Question

My apologies is this is too rudimentary to be asked here. If it does not belong here, could someone recommend a more appropriate place to ask?

A little context. I am in a Senior Physics Lab class in college. Our first "experiment" involved rolling dice and calculating the probability of rolling each face. (The purpose is purely for practice in applying and understand statistical analysis.) I chose to roll two dice separately with the intention of testing whether the two dice are identical. I know now I should have taken more samples, but I am not able to collect more data at this time.

I originally recorded my data in trials of 20 rolls, and I took 15 trials for each die. I then calculated the counts of each face within each trial and divided by 20 to obtain an estimate of the probability in each trial. I then averaged these probabilities over all the trials and calculated the standard error (calculated as the standard deviation of the sample). However, this produced very large standard error, from about 40% to 60% for each face. Here is the data for Die 1.

Face  Avg. Probability  Error  Percent Error
1     0.13              0.08   59.6
2     0.20              0.08   39.0
3     0.18              0.09   47.9
4     0.13              0.08   64.8
5     0.18              0.10   57.1
6     0.18              0.10   55.4

However, I discovered that by grouping the trials into larger sets, the error decreased. That is, I grouped trials 1 through 5, 6 through 10, and 11 through 15, and I calculated the probability of obtaining each face as the count of the face within the "grouped trial" divided by 100. This produced the following values for Die 1:

Face  Avg. Probability  Error  Percent Error
1     0.13              0.03   26.6
2     0.20              0.01   5.0
3     0.18              0.04   20.0
4     0.13              0.06   43.5
5     0.18              0.06   32.9
6     0.18              0.02   11.1

Have I definitely done something wrong, or is the change in the averages and the decrease in error explainable? Would I be doing something to invalidate my results by using the larger "group trial" samples? I can provide my original data in CSV format if desired, and of course, if I am saying things that make no sense, I am open to correction.

Edit:

I've found the mistake in my calculations with the groups of 20 rolls, and the means are now identical as expected. However, this hasn't resolved the question of why the error is so much larger when I group my data into a greater number of samples, which was really the question I meant to ask.

Yes, I understand that I could simply consider this problem using other means, but I'm still wondering why the error/standard deviation decreases. I think that question applies more generally than to this specific probability question. Or is the approach of trying to estimate the probability this way entirely invalid?

Answer 1

For each $i=1, 2, \ldots$, define the random variable $X_i$ as follows: $X_i = 1$ if the $i$th roll of the die is a 1, and $X_i = 0$ otherwise. Suppose that for your die, the probability of rolling a 1 on any given roll is $p$. Then $P(X_i = 1) = p$ and $P(X_i = 0) = 1-p$. Because the rolls of the die are independent, the $X_i$ are independent random variables.

For each $i$, the expected value of $X_i$ is $E(X_i) = 1 \times p + 0 \times (1-p) = p$. The expected value of $X_i^2$ is $E(X_i^2) = 1^2 \times p + 0^2 \times (1-p) = p$. The variance of $X_i$ is $\text{Var}(X_i) = E(X_i^2) - E(X_i)^2 = p - p^2 = p(1-p)$.

If you roll your die $n$ times, you get $X_1, \ldots, X_n$ and you estimate $p$ by $$ \hat{p}_n = \frac{1}{n} \sum_{i=1}^n X_i. $$ If $Y$ is any random variable and $c$ is any number, then $\text{Var}(cY) = c^2 \text{Var}(Y)$, so \begin{align*} \text{Var}(\hat{p}_n) &= \text{Var} \left( \frac{1}{n} \sum_{i=1}^n X_i \right) \\ &= \frac{1}{n^2} \text{Var} \left( \sum_{i=1}^n X_i \right) \\ &= \frac{1}{n^2} \sum_{i=1}^n \text{Var}(X_i) \qquad{\text{because the $X_i$ are independent}} \\ &= \frac{np(1-p)}{n^2} \\ &= \frac{p(1-p)}{n}. \end{align*} So the standard deviation of $\hat{p}_n$ is $\sqrt{p(1-p)/n}$. This is what you're estimating by taking the standard deviation of your sample proportions. If the die is fair, then $p=1/6$, and you first have $n=20$ and then $n=100$.

I've explained this for the proportion of ones, and the same applies to the other numbers 2,3,4,5,6.

Answer 2

The fact that you group your rolls into 15 trials of 20 rolls each is not relevant to estimating averages and standard errors. It is much more straightforward to treat this just as 300 rolls of each die.

The result of a n rolls of a die is a random variable with multinomial distribution. The variance of the number of times the i th face comes up is $np_i(1-p_i)$. You don't know the real value of $p_i$ but can estimate it from your data or just use $1/6$ - for these purposes you will get similar (not identical) results.

As you are interested in scale-free proportions, not the number of times out of 300, you need to divide the number of occurences by n (in this case 300); and to get the variance of that estimated proportion you divide by $n^2$ (because if you multiply a random variable by a constant like $1/n$, its variance increases by the constant squared). To convert the variance of that proportion into a standard error you take its square root, hence $\sqrt{\frac{p(1-p)}n}$ as suggested by mark999. It looks like you were using the wrong value of n.

Answer 3

I am not sure I understand completely what you are doing but I believe this could be it. Mathematically has already been explained above so just to dumb it down to prosa:

You roll out a sample of 300, each roll associated with an outcome. Then, you can group those results as you like. You also have the expected values for a "fair" dice and compare it to those outcomes by calculating standard errors, right?

What you are actually doing is assuming that the dice is fair and then doing a regression like experiment: You add up all results from the experiment (or a segment thereof), calculate the frequency and take this as an estimator - which you know should be 1/6. However you have a random part in your model. Either your dice may not be fair, or it may not fall to its expected value "by chance". In either case, this "error" can be expected to be zero on average.
It is added to each result (which would be expected as 1/6 chance to be either side), making the results differ.

Well using a lot of statistical results which you will learn, one can see that if the amount of data points increase, this error term tends to diminish if it has certain properties (which are probably fulfilled in an experiment such as this). In short: The expected value of that error term is zero and with more data points the variance of the regression residuals decrease.

So if you segment your groupings, you have less information about this error term. Your "estimate" of the true parameter (which should be 1/6), is simply more influenced by the error. If you combine your data points, you have much more information in your sample and those random errors start to even out more and more.

In fact, if you would have an infinite sample, you'd end up with no errors at all in your estimates.

The point beeing that your estimators are probably unbiased. Their expected value (so the expected value of the estimators) are in fact 1/6 (mostly because you error is expected to be zero). However in reality you will be off. And this residual error decreases with more data points. Remember that you add your data points together which means that you estimators for each segments are not from the same data. Adding those together you have more information - or said differently the error points for each sample - which are centered around zero - even out, pinning your estimate closer to the actual probability of the fair dice you threw.