How to calculate uncertainty of probability derived from random samples?

Question

I'm running many simulations based on random samples, each of which produces a True or False result. The goal is to calculate the probability that the result will be True, which I can easily calculate using (number of True results / total number of tests).

As I increase the number of simulations, this probability value varies less and less, converging to a specific number that depends on the thing I'm testing. How do I calculate the "uncertainty" of this probability estimate as I run the simulations? In other words, as the number of simulations increases, this "uncertainty" number would get smaller and smaller, while the "probability" number varies less and less.

Do I have to do multiple trials of a certain number of simulations each, calculate the probability for each trial, and then measure the uncertainty of those probabilities as a group? Or is there a way to calculate it directly along with the overall average from running n simulations in a row?

In other words, the more simulations I run, the more accurate my result is. How do I calculate how accurate it is?

Answer 1

You can use a Confidence Interval to get a measure for the accuracy of your estimation.

Basically you are drawing from a Binomial distribution, i.e. your $n$ True/ False values correspond to $X_i\sim Bi(p,1), \quad i=1...n$. and you are trying to estimate $p$ by $\hat{p}= \frac{1}{n}\sum_{i=1}^nx_i$.

As $n$ increases you can use the Central limit theorem to approximate the distribution of $\frac{1}{n}\sum_{i=1}^nX_i$. So for a large enough $n$ it holds approximately that $\frac{1}{n}\sum_{i=1}^nX_i \sim N(p, \sigma^2)$. Now you can create a confidence interval with this approximation. This requires an estimation of the variance of the normal distribution by $\hat{\sigma}^2= \frac{1}{n}\hat{p}(1-\hat{p})$. The confidence interval then turns out to be $[\hat{p}-z_\frac{\alpha}{2}\hat{\sigma}, \hat{p}+z_\frac{\alpha}{2}\hat{\sigma}]$, where $z_\frac{\alpha}{2}$ is the $\frac{\alpha}{2}$-Quantile of the standard normal distribution. I.e if you want a $0.95$ confidence interval, $z_\frac{\alpha}{2}$ would be $1.96$.

The factor $\frac{1}{n}$ for the estimated variance (or $\frac{1}{\sqrt{n}}$ for the standard deviation) also explains why your accuracy get better for increasing $n$, as the confidence interval gets smaller and smaller.

What this method assumes is that

1. for each sample the probability of getting an overall true/false outcome is the same.
2. the samples are independent.
3. n is large enough for the approximation of the Central Limit theorem to work.