Should I be considering the results from a study that just states percentages but didn't do any statistical analysis?

Question

I have a question regarding results from a study and it's probably a very basic question (but I only have a very very basic understanding of epidemiology). I'm looking at what characteristics have a significant association with an outcome from multiple studies, so I have generally looked at results that were found to be significantly associated with the outcome through statistical analysis (regression analysis mostly and some studies used odds ratio). However, I found a study that just states percentages and says there is an association. I'm a little bit wary of taking those results. I've looked in the methods but it does not say anything except that it conducted an analysis using StatSoft Statistica 13.1. So I'm not sure if I should be considering this?

Edit: The study also mentions the number of cases examined and provides n (%) for the study. In case anyone is wondering what the study looks like, here is a link: https://doi.org/10.3390/vaccines9050475

Edit: My main issue is that the study uses summary statistics and there is no statistical significance to the results, and that is why I was wondering if I should even be considering these results because of that.

Answer 1

It seems that the paper in question didn't report statistical significance of the results, but you can use the formula for the standard error of a binomial proportion estimate to help evaluate the results. From this page, for example, the standard error of a binomial proportion estimate $\hat p$ based on an overall sample size $n$ can be taken as $\sqrt{\hat p(1-\hat p)/n}$. With a sample size of 419, as reported by the paper, that would correspond to about 0.014 standard error in an estimated proportion of 0.1, to a maximum standard error of about 0.024 in an estimated proportion of 0.5.

There are several detailed approaches for how to use that relationship to evaluate signficance statistically; see this Wikipedia page for example.

Answer 2

By using summary based on their sample and not model based approach they failed the main rule of Causal Inference, "transferability". Their results cannot be transferred to the population and hence highly biased.

And exact details are mentioned by James Stanley below, which provides more details why results from non-model based approach should not be considered.

"(i.e. here your main concerns are that the estimates provided will effectively be crude/unadjusted results, and the proper consideration should take into account adjustment for confounding to get a robust causal estimate?)"

Answer 3

The authors of the paper acknowledge the lack of accuracy in their study and that's why they did not compute confidence intervals (statistical significance would not make sense because the research is not testing a hypothesis)

The study’s main limitation was the nonrandom volunteer sample of survey participants, which made it impossible to calculate a participation rate or confidence limits of the observed proportions.

Therefore the study is not a very strong measurement. This is also mentioned by the researchers who consider the search as 'a preliminary description of attitudes'

The study is limited by its nonrandom sample of HCWs but provides a preliminary description of attitudes towards SARS-CoV-2 vaccination.

---

Does it matter that they did not state significance?

However, I found a study that just states percentages and says there is an association. I'm a little bit wary of taking those results.

Even if they would have computed confidence intervals or provided some other calculation, then you should still be wary.

The output from SPSS* is overemphasized, but it is just a number from a model. Too many people look just at the p-value and regard it as a proof that the research is correct, a sort of validation label, and don't look further anymore.

But, the p-values can be wrongly calculated when the model is not appropriate. The sampling or the experiment could be biased. Systematic errors may have occurred. Etc.

The value of this study is also not as a scientific rigorous measurement of attitudes. It is a preliminary study to investigate attitudes. It also seems more like a research as they would do in businesses. Based on those results you can build new studies in the case that one would investigate it further and do actual research related to understanding the attitudes.

^{*Or whatever other software that researchers regard as a black box that they don't understand. But if they put in the numbers in the way that they learned during their university course, then they will get some magic number. And if it doesn't satisfy then possibly they can change the test by clicking some other buttons in the software or redo the test untill they have something they can publish.}