Estimating "population p-value" $\Pi$ using an observed p-value

Question

I asked a similar question last month, but from the responses, I see how the question can be asked more precisely.

Let's suppose a population of the form

$$X \sim \mathcal{N}(100 + t_{n-1} \times \sigma / \sqrt{n}, \sigma)$$

in which $t_{n-1}$ is the student $t$ quantile based on a specific value of a parameter $\Pi$ ($0<\Pi<1)$. For the sake of the illustration, we could suppose that $\Pi$ is 0.025.

When performing a one-sided $t$ test of the null hypothesis $H_0: \mu = 100$ on a sample taken from that population, the expected $p$ value is $\Pi$, irrespective of sample size (as long as simple randomized sampling is used).

I have 4 questions:

1. Is the $p$ value a maximum likelihood estimator (MLE) of $\Pi$? (Conjecture: yes, because it is based on a $t$ statistic which is based on a likelihood ratio test);

2. Is the $p$ value a biased estimator of $\Pi$? (Conjecture: yes because (i) MLE tend to be biased, and (2) based on simulations, I noted that the median value of many $p$s is close to $\Pi$ but the mean value of many $p$s is much larger);

3. Is the $p$ value a minimum variance estimate of $\Pi$? (Conjecture: yes in the asymptotic case but no guarantee for a given sample size)

4. Can we get a confidence interval around a given $p$ value by using the confidence interval of the observed $t$ value (this is done using the non-central student $t$ distribution with degree of freedom $n-1$ and non-centrality parameter $t$) and computing the $p$ values of the lower and upper bound $t$ values? (Conjecture: yes because both the non-central student $t$ quantiles and the $p$ values of a one-sided test are continuous increasing functions)

Answer 1

I think I may have found a possible answer for you.

In Computational Statistics Handbook with MATLAB by Wendy L. Martinez and Angel R. Martinez, they state:

Let $\theta$ represent a population parameter that we wish to estimate, and let $T$ denote a statistic that we will use as a point estimate for $\theta$. The observed value of the statistic is denoted as $\hat{\theta}$. An interval estimate for $\theta$ will be of the form $$\hat{\theta_{Lo}}<\theta<\hat{\theta_{Up}}$$ where $\hat{\theta_{Lo}}$ and $\hat{\theta_{Up}}$ depend on the observed value $\hat{\theta}$ and the distribution of the statistic $T$.

If we know the sampling distribution of $T$, then we are able to determine values for $\hat{\theta_{Lo}}$ and $\hat{\theta_{Up}}$ such that $$P\left(\hat{\theta_{Lo}}<\theta<\hat{\theta_{Up}}\right)=1-\alpha$$ where $0<\alpha<1$. [The preceding equation] indicates that we have a probability of $1-\alpha$ that we will select a random sample that produces and interval that contains $\theta$. [$\hat{\theta_{Lo}}<\theta<\hat{\theta_{Up}}$] is called a $\left(1-\alpha\right)\cdot100\%$ confidence interval. \dots It should be noted that one-sided confidence intervals can be defined similarly [Mood, Graybill and Boes, 1974].

$\dots$

the procedure for Monte Carlo hypothesis testing using the $p$-value approach is similar. Instead of finding the critical value from the simulated distribution of the test statistic, we use it to estimate the $p$-value.

Procedure—Monte Carlo Hypothesis Testing (P-Value)

1. For a random sample of size $n$ to be used in a statistical hypothesis test, calculate the observed value of the test statistic $t_0$.
2. Decide on a pseudo-population that reflects the characteristics of the population under the null hypothesis.
3. Obtain a random sample of size $n$ from the pseudo-population.
4. Calculate the value of the test statistic using the random sample in step 3 and record it as $t_i$.
5. Repeat steps 3 and 4 for $M$ trials. We now have values $t_i=1,\dots,t_M$, that serve as an estimate of the distribution of the test statistic, $T$, when the null hypothesis is true.
6. Estimate the $p$-value using the distribution $\dots$, using the following.

Lower Tail Test$$\hat{p}-value=\frac{\left(t_i\leq t_0\right)}{M}$$ for $i=1,\dots,M$

UpperTail Test$$\hat{p}-value=\frac{\left(t_i\geq t_0\right)}{M}$$ for $i=1,\dots,M$

It seems reasonable then, that you could use this same method to report the limits of the sampled $p$-values in some meaningful way to represent a confidence interval of the test statistic.