Simple multiple choice for test statistic and significance level

Question

A study aims to determine if a new tutoring program (method 1) is better than standard preparatory program (method 2) to prepare students for the SAT. It measures the proportion of students who score in the top quarter of test takers. Results: 53% of sampled students using the standard program (method 2) scored in the top quarter of test takers, while 45% of sampled students using the new program (method 1) scored in the top quarter of test takers. A hypothesis test will be performed using a 5% significance level.

1. Will the test statistic be positive or negative?
2. True or false: At a 5% significance level, the results are statistically significant.

I put negative for #1 and false for #2, but I got both of them wrong. Can someone explain how I solve this type of question in the future and, most important, avoid choosing incorrectly?

Answer 1

1) It depends on the test statistic! You can sensibly get positive or negative test statistics, but the two I'd most likely use (two sample proportions test of $\pi_2-\pi_1$ OR chi-square) would both be positive.

2) There's not enough information - we don't have sample sizes!

[Well, in fact we sort of can sometimes figure out a difference must be significant even without the sample sizes, just based on the counts that can give us the observed proportions:

i) What's the smallest sample size in which we could observe 45%? Assuming it's a rounded fraction, it looks like 5/11

ii) what's the smallest sample size in which we could observe 53%? That looks like 8/15 is the smallest possible.

So if it's significant at the lowest possible sample size, and the next few 45%-vs-53% sample-sizes up (to allow for the fact that the actual difference might get a little smaller and more than undo the gain in smaller standard errors), it will be significant at larger samples. But in this case it turns out it's not significant for 5/11 vs 8/15:

> prop.test(c(5,8),c(11,15))

2-sample test for equality of proportions with
continuity correction

data:  c(5, 8) out of c(11, 15) 
X-squared = 0, df = 1, p-value = 1
alternative hypothesis: two.sided 
95 percent confidence interval:
 -0.5452923  0.3877165 
sample estimates:
   prop 1    prop 2 
0.4545455 0.5333333 

So that's no help - since it's not significant at the smallest sample sizes, we still can't tell for sure. I assume you have sample size information elsewhere that you didn't notice.]