What is the Statistical Method?

Question

I apologize in advance for the possibly philosophical nature of the question, however I would like to get answers from this website rather than from Philosophy exchange at the moment. I also come from a Pure mathematics background, so I might have some misconceptions about how the field of Statistics works. The scientific method and the axiomatic method of mathematics try to address the questions of distinguishing science from pseudoscience, and mathematics from "pseudo-mathematics" respectively. I wish to know an answer for the analogous question of distinguishing Statistics from "Pseudo Statistics". I will present below a potential answer to the previous questions, and I would like to hear from statisticians on this website whether I have the right answer or not.

My idea of the scientific method is basically the flowchart below:

For example, a scientist may be living on two dimensional surface where she always observes that the sum of angles of any triangle add up to 180 degrees. From this observation, our scientist might make the hypothesis that the surface has a euclidean geometry. This hypothesis will make a prediction that for example Pythagoras theorem is true in our surface. The scientist will test the validity of this prediction by making more experiments . If the prediction turns out to be false the hypothesis is thrown away. If the prediction turns out to agree with further experiments, then the scientist's faith in her model increases a bit and she uses the hypothesis to make further prediction and tests them in an ongoing process.

I try to modify the above scenario for statistics.

1. Step 1: We collect some observations about a real world phenomenon. The phenomenon can be flipping a coin
2. Step 2: The hypothesis could be that this real world phenomenon of flipping a coin $n$ times can be modelled as Bernouli process $B(n,\frac{1}{2})$. I think of hypothesis making as the process of matching mathematical structures to real world phenomena. In the above scenario, we were concerned with matching the real world phenomenon of the surface we live on to some mathematical Geometric structure, and here we are concerned with matching the real world phenomenon of flipping a coin to some mathematical probabilistic structure.
3. Next step is to make a prediction. We agree by convention on some significance level before the start of the investigation, which could be say $99$%. If some event $E$ in the probabilistic model of our hypothesis has $P(E)\geq 0.99$, then it counts as a prediction. In our case for example, Chebyshev's inequality will give that $$P(\text{proportion of heads after $10,000$ trials will lie in the interval $]0.45,0.55[$})\geq 0.99$$. Thus, we have our prediction that if we flip the coin $10,000$ times then proportion of heads will be between 0.45,0.55
4. Step 4: We test our prediction experimentally. We flip the coin actually 10,000 times. If the prediction agrees with empirical results then our faith in our hypothesis increases. If prediction does not agree with empirical results then hypothesis is thrown away and further investigation should take place. Maybe the coin is biased and our probabilistic model should have been $B(n,p)$, maybe the coin tosses affects the later tosses if for example the coin temperature changes or its shape changes thus changing its mechanics, hence we the assumption of independence of coin tosses is inappropriate and does not yield good predictions...etc.

Question : Do I have the correct idea about how statistics works ? If yes, on what basis do we choose the significance level ?

Answer 1

What is the Statistical Method?

There is no the statistical method. Your title question is asking for something that doesn't exist.

I try to modify the above scenario for statistics.

Your scenario looks much like a Popperian view of the scientific method. It is not a general view of the scientific method (in the recent century several philosophers have been disagreeing with Popper) and it is certainly not statistics. You can involve statistics in it like in your modified scenario but it is not equivalent.

---

The term statistics originates from the word 'state' and it used to refer to 'data of the state' and evolved into 'the collection of facts'.

Zu Statist ‘Staatsmann’ wird ... bald (Anfang 18. Jh.) in der Form in der Form Statistik f. für ‘Beschreibung eines Staates, eines Landes’ und (2. Hälfte 18. Jh.) die ‘Wissenschaft von der Erfassung, Erforschung und Beschreibung von Massenerscheinungen in Natur und Gesellschaft’ ... bezeichnet

(Quote from the online Etymologisches Wörterbuch des Deutschen)

The terms statistics, relating originally just to data of the state, now relates more generally to the methods that revolve about the sampling, observation, reporting and analyzing the quantitative data in any field.

Statistics is a broad field and can involve mathematical topics such as computing the probabilities of specific types of models of the sampling process and models of observation, but also topics like computer science (making computations, working with databases), social science (developing interview techniques and sampling methods), and visualization (what sort graphs are best suited to convey data).

---

The scenario that you describe, where the statistician computes a p-value and determines a significance level, seems to relate to a narrow part of statistics which is called 'hypothesis testing'. It is not so much the method of statistics as statistics involves more than just that. And in addition, it is also not belonging solely to statistics. Hypothesis testing is also a part of the philosophy of science. Statisticians relate more to designing the models for performing the computations in hypothesis testing, but it are scientists in a particular field that make a decision to use a hypothesis test. It is not the statistician that says that hypothesis testing is the way to go.

on what basis do we choose the significance level ?

These levels are chosen based on experience and to balance the probabilities between false positive and false negative cases. For instance, in particle physics they now use a $5\sigma$ confidence level and it evolved into that because they perform so many experiments that they would observe too many false positive discoveries if they did not use a strict level (see Origin of "5$\sigma$" threshold for accepting evidence in particle physics?).