What statistical test do I use to check percentage differences?

Question

I am aware this is very basic, and I think I may be overcomplicating it but can someone please confirm what the best thing to do is?

I am looking at female representation within special advisers in UK Government, specifically to see whether women are reaching the highest bands of seniority (which is PB4 in the data).

Here is a summary of the data for 2022:

Female Representation Between Bands

However, I want to look at what % of the total cohort of women is in each band compared to men and then calculate if this is statistically significant?

% of cohort in each category

So for PB4, 3% of the total cohort of women make it to PB4 as opposed to 6% of all men, which is a difference of 100% - now I want to know what test can I do to test if that difference is statistically significant?

Thank you!

---

Before I refined my research Q to looking at %'s I tried a fishers exact (as my data violates Q-Q plots) and it came up with not-statistically significant.

But I am unsure if I did something wrong, because how can a difference of 100% not be significant?

Answer 1

A q-q plot would not be the correct method to examine the distributions here. These are grouped count data and while qq plots are designed for continuous variables. A Fisher exact test would be an acceptable test but I think it would not tell you whether there were statistical evidence of your main question regarding the decreasing proportion of women at higher levels of influence. It's not a test of trend. What you want is a test of trend.

My eyeball assessment is that these data are pretty weak for showing a significant trend because that is not the direction shown in the middle two bands where most of the counts lie and there's actually evidence in the other direction, and furthermore the numbers at the highest band are so small that no test is likely to find significance. 2 out of 7 versus 5 out of 7 is not strong evidence despite your impression that the contrast is clear. Now I'm going to retire briefly and apply some R code to the task of building a trend test.

Here's one sort of trend test appropriate for this question implemented in Poisson regression:

summary(glm(V1 ~ PB+offset(log(V1+V2)), data=cohort.df, fam="poisson"))
Call:
glm(formula = V1 ~ PB + offset(log(V1 + V2)), family = "poisson", 
    data = cohort.df)
Deviance Residuals: 
      1        2        3        4

-0.3462   0.4316  -0.3278   0.2194
Coefficients:
            Estimate Std. Error z value Pr(>|z|)

(Intercept) -0.84681    0.43475  -1.948   0.0514 .
PB          -0.04269    0.18527  -0.230   0.8177

Signif. codes:  0 ‘*’ 0.001 ‘’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
(Dispersion parameter for poisson family taken to be 1)
Null deviance: 0.51500  on 3  degrees of freedom

Residual deviance: 0.46172  on 2  degrees of freedom
AIC: 20.774
Number of Fisher Scoring iterations: 4

So there is an estimated negative coefficient for the ratio of women to band subtotals, but the strength of the association is very weak and chance-like. You could also have don a Cochran_armitage trend test and I'm guessing that you could probably find an R package that would do that named test. Theres a good vignette on handling count data in R by Zeileis, Kleiber, and Jackman: https://cran.r-project.org/web/packages/pscl/vignettes/countreg.pdf You could also consult: "R (and S-PLUS) Manual to Accompany Agresti’s Categorical Data Analysis (2002) 2 nd edition" by Laura A. Thompson, 2008©. I got my copy many years ago but it still appears available: https://www.stat.purdue.edu/~zhanghao/MAS/handout/R%20Manual%20to%20Agresti%E2%80%99s%20Categorical%20Data%20Analysis.pdf

Answer 2

I'm not exactly sure what your goal is, but if you want to know whether or not females make up 50% of the advisers in a particular band, I would suggest a binomial test binom.test(x,n,p).

So for the PB4 band it's binom.test(2,(2+5),0.5), so 2 'successes'(x) , 2+5=7 'trials'(n) and 0.5 'probability of success'(p), so assuming there are 50% women. The test gives a p value of about 0.45. So, although 5 is more than 100% more than 2, it's not significant, but that's not surprising, as you only have 7 'trials'.

P.S.: As you didn't give too many details about your approach with the Fisher exact test, I'm not sure what you did, but if you did something like:

cohort <- matrix(c(2,16,27,8,5,22,46,10),nrow=4)
fisher.test(cohort)

you wouldn't end up with a test that looks at whether 3% women in PB4 is less than 6% men, but rather whether the distribution of women and men on the four bands is different, which according to the test is not the case.