6

I believe the heuristic definition of a Sufficient Statistic makes sense to me - when you take a sample in order to make an inference about the parameter related to the probability distribution, and you reduce that realised sample to a (sufficient) statistic - you do not lose any information or "inference power" regarding the parameter. (However, we still do not know what the actual true value of the parameter is, but I could make an estimate of the range of values that the parameter may be in just as well with the statistic as I could with the original sample).

(Please correct me if you feel there is anything inaccurate about what I said above)

However where I am struggling to connect the dots is how the mathematical definition of Sufficient statistics is saying same thing.

Definition: A statistic $t = T(X)$ is sufficient for underlying parameter $θ$ precisely if the conditional probability distribution of the data $X$, given the statistic $t = T(X)$, does not depend on the parameter $θ$. (Wikipedia)

Why does $\theta$ disappearing in the conditional probability distribution mean the same thing as $T(X)=t$ and $X=x$ having the same "inference power" about $\theta$.

The intuition I have interpreted about this is "Knowing $T(X)$ means you know everything about $\theta$ from the information supplied by the sample $X$" - but my confusion is why does this mean that the conditional distribution should not have $\theta$ once we have $T(X)$ -> it's not like we know the actual value of $\theta$ when we obtain $T(X)$, right? I just imagined $T(X)$ is all that is required when we try to make an inference - a "guess" about the range of values $\theta$ may be in. But what does this have to do with the conditional probability being independent of $\theta$?

Tortar
  • 356
user523384
  • 569
  • 1
    Some intuition: The conditional distribution not depending on the parameter $\theta$ means that regardless of the parameter value $\theta$, the conditional distribution will be always the same. In other words, you get the same conditional distribution if $\theta = 1$ or if $\theta = -1$ (or for that sake any other value). Would you be able to make any inference regarding $\theta$ using this conditional distribution after observing a sample? Clearly not. So, the conditional distribution contains no information, so to say, regarding the value of $\theta$ that has generated the data. – baruuum Feb 21 '20 at 05:08
  • 1
    Thanks @baruuum! That clarifies the idea of the conditional distribution being independent of $\theta$, yet I still have trouble seeing the connection between independence of $\theta$ and preserving all information about $\theta$? – user523384 Feb 21 '20 at 09:52
  • Similar: https://stats.stackexchange.com/q/84226/119261 – StubbornAtom Feb 21 '20 at 13:36

3 Answers3

4

I think a common way to motivate the mathematical definition is the following.

Say you have the sufficient statistic $T(X)$, and I just have the data/random sample $X$. By the mathematical definition of sufficiency, $$ p(X|T(X), \theta) = P(X|T(X)).$$ The r.h.s. is a probability distribution you, again per definition and in theory, know, and can use to do inference. In particular, if you draw a sample $X' \sim P(X'|T(X))$, then this sample is also a draw from the distribution of interest, in particular, the distribution that depends directly on $\theta$. But you have now recovered a sample using just the sufficient statistic that contains as much information as the original sample $X$. Thus just using your sufficient statistic you can infer just as much (confidence intervals, for example) as me who enjoyed access to the full sample $X$, and it is therefore sufficient to know $T(X)$.

Forgottenscience
  • 1,226
  • Thanks for the answer! I wanted to clarify something. So it seems to me that "$P(X|T(X),\theta) = P(X|T(X))$" is saying "Once you know $T(X)$, I know what the probability distribution of $X$ is (since there is no more unknown $\theta$)."

    So the distribution of $X$ is unknown due to $\theta$, but when we find out $T(X)$, then the distribution of $X$ becomes known?

    This feels strange to me because based on one sample (turned into a statistic), we managed to get the distribution of $X$?

     – user523384 Feb 21 '20 at 12:35
  • Also, in the last sentence of the solution in: https://online.stat.psu.edu/stat414/node/282/

    They claim: "We have just shown that the conditional distribution of $X_1, X_2, ..., X_n $ given $Y$ does not depend on $p$. Therefore, $Y$ is indeed sufficient for $p$. (-HERE->) That is, once the value of $Y$ is known, no other function of $X_1, X_2, ..., X_n$ will provide any additional information about the possible value of $p$."

     – user523384 Feb 21 '20 at 12:48
  • The place marked "(-HERE->)" is where I lose the link in understanding – user523384 Feb 21 '20 at 12:49
  • Two facts: $\theta$ is unknown, while the distribution of the data is not unknown - it is the statistical model we have defined, we call it $p(X|\theta)$. The definition states that, given a sample $X \sim p(X|\theta)$ and a sufficient statistic calculated from that sample, $T(X) = t$, the equation $p(X|T(X) = t, \theta) = p(X|T(X) = t)$ holds. It states nothing about $\theta$ being known. In the example you link to, they explicitly show that, given $T(X)$ (the sample mean), the sufficiency condition holds - namely, $\theta (=p)$ is gone from $p(X|T(X)=t, \theta)$. – Forgottenscience Feb 21 '20 at 13:46
2

"A statistic t=T(X) is sufficient for underlying parameter θ precisely if the conditional probability distribution of the data X, given the statistic t = T(X), does not depend on the parameter θ."

If the sampling distribution for some data $X$ does not depend on $\theta$ then how can that data say anything about $\theta$?

It would be like estimating some value by observing something irrelevant (that does not depend on the value to be estimated).

This is a general statement. In this case we have more specifically as data 'the rest of the data conditional on the sufficient statistic'. And that is confusing because the sample distribution for the rest of the data does depend on the parameter to be estimated. It is only that the conditional distribution of that data does not depend on the parameter to estimated.

example 3 (different outcomes of data, but with the same probability for a given $\theta$)

(edit: based on the comments I came up with a much more simple/intuitive explanation)

Say you do a urn problem trying to estimate the fraction of blue balls in an urn. You perform an experiment by drawing balls with replacement.

Say you got "$x_1 = red$, $x_2 = blue$, $x_3 = red$, $x_4 = blue$, $x_5 = blue$"

That is 3 blue balls in total (the total is the sufficient statistic). You could base on this a point estimate of 0.6 for the fraction of blue balls in the urn. (in reality you should take a bigger sample if you want to make a confidence interval with a narrow bandwith, but that makes this example difficult to write down)

Now, does it matter (for the fraction) which particular balls $x_i$ where blue (beyond the fact that we already know the total number 3)? Is the estimate gonna be different for "$x_1 = blue$, $x_2 = red$, $x_3 = red$, $x_4 = blue$, $x_5 = blue$" or any other different observation that also has 3 blue balls in total? Each of these outcomes, with a total of 3 blue balls, are equally possible. So they will not give more information about the fraction of balls in the urn.

We could tabulate all the different outcomes and how the probability of observing them depends on $\theta$ (the fraction of blue in the vase)

observation        probability of observing given theta

bbbbb              (1-theta)^0(theta)^5

rbbbb              (1-theta)^1(theta)^4
brbbb              (1-theta)^1(theta)^4
bbrbb              (1-theta)^1(theta)^4
bbbrb              (1-theta)^1(theta)^4
bbbbr              (1-theta)^1(theta)^4

rrbbb              (1-theta)^2(theta)^3
rbrbb              (1-theta)^2(theta)^3
rbbrb              (1-theta)^2(theta)^3
rbbbr              (1-theta)^2(theta)^3
brrbb              (1-theta)^2(theta)^3
brbrb              (1-theta)^2(theta)^3
brbbr              (1-theta)^2(theta)^3
bbrrb              (1-theta)^2(theta)^3
bbrbr              (1-theta)^2(theta)^3
bbbrr              (1-theta)^2(theta)^3

rrrbb              (1-theta)^3(theta)^2
rrbrb              (1-theta)^3(theta)^2
rbrrb              (1-theta)^3(theta)^2
brrrb              (1-theta)^3(theta)^2
rrbbr              (1-theta)^3(theta)^2
rbrbr              (1-theta)^3(theta)^2
brrbr              (1-theta)^3(theta)^2
rbbrr              (1-theta)^3(theta)^2
brbrr              (1-theta)^3(theta)^2
bbrrr              (1-theta)^3(theta)^2

brrrr              (1-theta)^4(theta)^1
rbrrr              (1-theta)^4(theta)^1
rrbrr              (1-theta)^4(theta)^1
rrrbr              (1-theta)^4(theta)^1
rrrrb              (1-theta)^4(theta)^1

rrrrr              (1-theta)^5(theta)^0

Notice that in this table above there are groups of potential observations/outcomes for which the probability to be observed have exactly the same dependency on $\theta$. This means that it doesn't matter whether you observe rbrbb or brrbb, they relate to $\theta$ in the same way. All the observations with three blue balls can be considered to provide the same information about $\theta$.

This is sort of what the sufficient statistic does. It groups together the observations whose Likelihood dependency on $\theta$ is the same.

I have deleted examples 1 and 2, because it makes the post very large, but you can still see them in the history of this post

Sextus Empiricus
  • 77,915
  • Thanks for your response! Could you please guide me through this, I'm getting a little lost through this process:
    • We consider a sample $\mathbf{X} = (X_1, X_2,...,X_n) $ from a population. $X_i \sim f(x_i;\theta)$, where $\theta$ is non-random but unknown
    • The realised sample becomes $\mathbf{x} = (x_1, x_2,...,x_n)$. This sample can be used to form a inference (e.g. confidence interval) for $\theta$
    • We claim that a sufficient statistic $t =T(\mathbf{x})$ can be used to form the same sample inference (e.g. confidence interval) for $\theta$ without $\mathbf{x}$
     – user523384 Feb 21 '20 at 10:25
  • I'm not sure how this relates to the conditional distribution (is it $f(x_{i}; \theta | T(\mathbf{x}) = t)$ ?) – user523384 Feb 21 '20 at 10:27
  • @user523384 in the first example the sufficient statistic is the maximum of the points. That is $X_{max}$ is the largest from the $(X_1,X_2,...,X_n)$. How the other $X_i$ are distributed can be seen as fully dependent on this maximum $X_{max}$. – Sextus Empiricus Feb 21 '20 at 10:46
  • We could say $T \sim f(t;\theta)$ and $X_{i} \sim g(x_i;T)$. – Sextus Empiricus Feb 21 '20 at 10:48
  • Or $f(x_{i}; \theta | T(\mathbf{x}) = t) = f(x_{i}; T(\mathbf{x}) = t) $. The distribution of the $x_{i}$, when it is conditional on $T$, can be described without $\theta$. – Sextus Empiricus Feb 21 '20 at 10:50
  • Yes, I understand that this does occur - but I wonder why it is so "groundbreaking" that $f(x_i | T(\mathbf{x}) = t) $ can be described without $\theta$? And why does this mean that $T(\mathbf{x})= t$ conveys the same information about $\theta$ as $\mathbf{x}$?

    Thanks

     – user523384 Feb 21 '20 at 10:52
  • 1
    It is not groundbreaking. It is just that a summary of the data, the statistic $T(x)$, may be sufficient. Different values of $x$ can give the same value of $T(x)$, but how these values $x$ are distributed for a given value of $T(x)$ is independent from $\theta$. – Sextus Empiricus Feb 21 '20 at 10:57
  • Right, so in your example, for $X_i \sim U(0,a) $ with parameter $a$, say our sufficient statistic is $T(X) = X_{max} = m$, then the rest of the data $X_{i} \sim U(0, m) $, i.e., they no longer depend on $a$

    So thanks for this example, before I did not even understand the idea of being independent of the parameter $a$. However, I still struggle to see the conection between this and the heuristic definition as on wikipedia -

     – user523384 Feb 21 '20 at 11:03
  • "a statistic is sufficient for a family of probability distributions if the sample from which it is calculated gives no additional information than does the statistic, as to which of those probability distributions is that of the population from which the sample was taken." – user523384 Feb 21 '20 at 11:03
  • It may become more intuitive when you restrict the value $\theta$ to only two values. Say you have a vase with some fraction of blue balls. Say you take a sample of size five and you get 'blue, red, black, blue, yellow' then we have 2 blue balls. In order to estimate the fraction of blue balls does it matter to describe the other colours? In order to describe the fraction of blue balls does it matter to describe the positions of the blue balls, or is it sufficient to just use the total number of blue balls? – Sextus Empiricus Feb 21 '20 at 11:04
  • Neither should matter - the total number of blue balls would be "sufficient" – user523384 Feb 21 '20 at 11:07
  • @user523384 and what now if the order of the balls, for a given fraction, would be somehow dependent on $\theta$? Say, if $\theta >0.5$ then the first ball has a hundred percent probability to be blue. – Sextus Empiricus Feb 21 '20 at 11:20
  • Another example: say instead of estimating a uniform distribution between 0 and a we estimate an exponential distribution that is truncated at $\lambda$. In the former case the maximum is enough and the distribution of the other points is irrelevant. In the latter case the distribution of the other points is still dependent on the parameter. – Sextus Empiricus Feb 21 '20 at 11:25
  • Let us continue this discussion in chat. – user523384 Feb 21 '20 at 11:26
0

Here is a very simple example that might make things clear. For a normal model $X\sim N(\theta,1)$ with a sample size of $n=1$, $X$ is sufficient for $\theta$. Without conditioning on $T(X)=X$, then $P(X\le x)=\Phi(x-\theta)$ depends on $\theta$. If I condition on $T(X)=X=c$ then $$ P(X\le x|X=c)=\left\{\begin{array}{ll} 0 & \text{if } x < c\\ 1 & \text{if } x \ge c \end{array} \right.$$ does not depend on $\theta$. This shows that once $T(X)$ is known there is no other information available for estimating $\theta$. If there was additional information available then $P(X\le x|T(X)=c)$ would depend on $\theta$ just like $P(X\le x)$ did (and certainly would not be degenerate). Let me known if I have made any mistakes.

Geoffrey Johnson
  • 2,800