Help finding the right test

Question

I am running an experiment and now it is time to do some analysis, but I am having a hard time figuring out the right way to analyze my data. I have a number of questions, but first let me give you some context.

This is a behavioral experiment, where the setting is as follows:

i) There are 2 conditions, let's call them A and B.

ii) Each participant is randomly assigned to either A or B.

iii) Regardless of the condition, each participant goes through 8 trials.

iv) At each trial, we show a stimuli to the participant, and then record whether they exhibit a certain behavior. So we have a binary variable, call it Target_Behavior, which takes the value 1 is the participant exhibits this behavior, and 0 otherwise.

The first thing I want to study is whether there is a significant difference between the two conditions, regarding the frequency participants exhibit the target behavior. As you can see, for each participant we have 8 measurements, so it is a repeated measures setting, but I don't know how to choose the right test that takes this into account.

Furthermore, I don't know whether my data allows for any parametric assumptions, so I am only using non-parametric tests, which is detrimental to the power of my tests.

So far, I have been using a very simple (and probably wrong) solution. I am looking at each trial on its own. This way, for any trial we have only a single measurement for each participant, so, for example considering trial 1, I make a list containing the values of Target_Behavior for condition A, and the same thing for condition B. Finally, I do a chi-square test, comparing the two lists.

I am pretty sure this is suboptimal though, while it also poses an extra challenge. I am convinced there is a significant difference between conditions A and B, but the test I just described only gives significant results if there is a difference of about 30% in frequency between the two conditions. If they are at 20% it turns out to be not significant, however, this doesn't seem right to me, 20% is a huge discrepancy!

I have about 20 participants in each condition, if this is of any help.

I would really appreciate your help in figuring the right way to deal with this situation. I have been looking online for weeks, and still I haven't found a solution. I have some additional questions, but for now I think this is the most important one.

Thank you in advance!

Answer 1

Caveat: You don't state it explicitly but you imply that, for each participant, their trials are independent with the same probability of success. In this case, the number of successes is a binomial random variable with size = 8 and unknown probability p.

Update: In the comments you explain that the independence assumption holds but the success probability may change as participants become familiar with the experiment. In this case, the trials are binomial random variables with size = 1 and unknown probability p_t. You can still use binomial regression after updating the model to include a fixed effect t for the sequence of trials.

---

We can compare conditions A and B with a binomial regression. We assume that:

• Each participant goes through a fixed number of trials (not necessarily the same number of trials).
• The success probability of each participant is a function of their group assignment + random noise.

Under this model, conditions A and B have effects α and β, and the average success probabilities are a function of this effect. [Mathematically, effect = logit(probability).]

Finally, we test if there is a difference between conditions A and B with a t-test on the difference in effects, β - α.

The following code illustrates how to do this analysis in R.

library("broomExtra")
library("lme4")
set.seed(1234)
n <- 20
trials <- 8
alpha <- 0.5
beta <- 2
sigma <- 0.1 # std. deviation of participant random effect
id <- seq(1, 2 * n)
condition <- rep(c("A", "B"), each = n)
On average participants in group A have effect alpha
and participants in group B have effect beta
effect <- ifelse(condition == "A", alpha, beta)
Add random noise to each participant's effect.
effect <- effect + rnorm(2 * n, sd = sigma)
aggregate(effect, list(condition), mean)
#>   Group.1         x
#> 1       A 0.4749336
#> 2       B 1.9422930
The success probability is the inverse logit of the participant's effect.
prob <- plogis(effect)
Finally generate the outcome: number of success in a fixed number of trials.
successes <- rbinom(2 * n, size = trials, prob = prob)

Let's start with a simpler model that assumes participants in each group have exactly the same probability of success. This is not true in the simulation and it's also not a reasonable assumption in general.

model1 <- glm(
  cbind(successes, trials - successes) ~ condition,
  family = binomial(link = "logit")
)
tidy(model1, conf.int = TRUE, conf.level = 0.95)
#> # A tibble: 2 × 7
#>   term        estimate std.error statistic     p.value conf.low conf.high
#>   <chr>          <dbl>     <dbl>     <dbl>       <dbl>    <dbl>     <dbl>
#> 1 (Intercept)    0.674     0.167      4.03 0.0000548      0.352      1.01
#> 2 conditionB     1.67      0.326      5.12 0.000000299    1.06       2.34

It's more appropriate — and in the simulation definitely correct — that participants effects are not fixed but random, with mean equal to the condition effect.

model2 <- glmer(
  cbind(successes, trials - successes) ~ condition + (1 | id),
  family = binomial(link = "logit")
)
tidy(model2, conf.int = TRUE, conf.level = 0.95)
#> # A tibble: 3 × 9
#>   effect   group term   estimate std.error statistic  p.value conf.low conf.high
#>   <chr>    <chr> <chr>     <dbl>     <dbl>     <dbl>    <dbl>    <dbl>     <dbl>
#> 1 fixed    <NA>  (Inte…    0.704     0.200      3.51  4.42e-4    0.311      1.10
#> 2 fixed    <NA>  condi…    1.72      0.367      4.69  2.67e-6    1.00       2.44
#> 3 ran_pars id    sd__(…    0.440    NA         NA    NA         NA         NA

In the summary table, conditionB estimates the effect difference β - α between conditions A and B. Since this is a simulation, we know that the true difference is β - α = 2 - 0.5 = 1.5. Its estimate from the data is conditionB = 1.72 with a 95% confidence interval [1.00, 2.44]. The t-statistic for the null hypothesis that there is no difference between the conditions is 4.69 and the corresponding p-value is 2.7e-6. So there is strong evidence to reject the null, which of course doesn't hold in the simulation.

The effects α and β are logits (log odds) of the success probabilities; the difference β - α is the log odds ratio. It's easy to convert effects to probabilities with the inverse logit transformation.

# Pr{success in group A}, Pr{success in group B}, Pr{success in B} - Pr{success in A}
c(plogis(alpha), plogis(beta), plogis(beta) - plogis(alpha))
#> [1] 0.6224593 0.8807971 0.2583377
# estimate of Pr{success in B} - Pr{success in A}
plogis(1.72 + 0.704) - plogis(0.704)
#> [1] 0.2495652