Combining biased noisy measurements

Question

I'm looking for a way to combine noisy biased measurements to find confidence intervals. As an example, we have two people Tom & Mary that are each taking free throws on separate days, we have the following results from the two days:

Day	Player	makes	attempts
Day 1: Sunny	Mary	89	100
Day 1: Sunny	Tom	171	200
Day 2: Windy	Mary	119	200
Day 2: Windy	Tom	51	100
Day 3: Rainy	Mary	24	50
Day 3: Rainy	Tom	24	50

What kind of strategies are there to calculate which player is better and with what confidence?

I though about doing a gaussian approximation for each day and Combining uncertain measurements, but from the answers I've seen they all assume unbiased measurements, and as you can see, there is a bias based upon the weather.

---

Edit I have a strategy, but I am curious if it's way off base:

1. For each day, I calculate the pct diff between the two players by:
   • approximating the distributions with gaussians
   • subtracting the gaussians:
     • $\mu_{\Delta} = (\mu_{mary} -\mu_{tom})$
     • $\sigma^2_{\Delta} = (\sigma^2_{mary} +\sigma^2_{mary})$
   • normalizing to $\Delta\%$ of tom's distribution
     • $\mu_{\Delta\%} = \mu_{\Delta} / \mu_{tom}$
     • $\sigma^2_{\Delta\%} = \sigma^2_{\Delta} / \mu_{tom}^2$
   • Result:
     • Day 1: $\mu = 0.041$, $\sigma^2 = 0.0022$
     • Day 2: $\mu = 0.017$, $\sigma^2 = 0.0139$
     • Day 3: $\mu = 0.000$, $\sigma^2 = 0.0409$
2. I use the answer here https://stats.stackexchange.com/a/275520/3143 to calculate the mean of the deltas:
   • \begin{equation} \hat{x} = \frac{\Sigma_ix_i/\sigma^2_{x,i}}{\Sigma_i1/\sigma^2_{x,i}} \end{equation}
   • $$ \frac{1}{\sigma^2} = \sum_i \frac{1}{\sigma_{x,i}^2}$$
   • Result:
     • $\mu = 0.0557$, $\sigma^2 = 0.0018$
     • 95% confidence interval: -2.83% - 13.97%

---

Edit 2 As a check, I ran my calculations across the same data with identical days, once grouped and once split:

Day	Player	makes	attempts
Each Day	Mary	85	100
Each Day	Tom	75	100
Total	Mary	255	300
Total	Tom	225	300

And got the following results:

• By Day and aggregated:
  • $\mu = 0.133$, $\sigma^2 = 0.00186$
  • confidence interval: 4.89% - 21.77%
• Total Row:
  • $\mu = 0.133$, $\sigma^2 = 0.00185$
  • confidence interval: 4.87% - 21.79%

Answer 1

The terms you used are misleading. your "Noise" - is called sampling error, and basically any statistical method addresses sampling error.

There is no "Bias", but dependence on multiple independent variables. So you would look at (multiple) logistic regression for your problem. People talk of "controlling" for other variables (How exactly does one “control for other variables”?)

Think of studies of effect of smoking on chance of heart attack * controlling* for eg age/gender etc. ...

This is just a standard logistic regression problem with weather and player as factors.

So you would look at confidence intervals around the coefficient for player. (see eg https://sphweb.bumc.bu.edu/otlt/mph-modules/bs/bs704_multivariable/bs704_multivariable8.html)

That's the basic "mechanics". as others have said, quite what you mean by aggregate will change the details.

Answer 2

Since I haven't received any answers, I'm posting the strategy I ended up using as an answer here. I'll wait to accept this answer for a bit to see if anyone else has a different strategy or simulation which might show different results:

1. For each day, I calculate the pct diff between the two players by:
   • approximating the distributions with gaussians
   • subtracting the gaussians:
     • $\mu_{\Delta} = (\mu_{mary} -\mu_{tom})$
     • $\sigma^2_{\Delta} = (\sigma^2_{mary} +\sigma^2_{mary})$
   • normalizing to $\Delta\%$ of tom's distribution
     • $\mu_{\Delta\%} = \mu_{\Delta} / \mu_{tom}$
     • $\sigma^2_{\Delta\%} = \sigma^2_{\Delta} / \mu_{tom}^2$
   • Result:
     • Day 1: $\mu = 0.041$, $\sigma^2 = 0.0022$
     • Day 2: $\mu = 0.017$, $\sigma^2 = 0.0139$
     • Day 3: $\mu = 0.000$, $\sigma^2 = 0.0409$
2. I use the answer here https://stats.stackexchange.com/a/275520/3143 to calculate the mean of the deltas:
   • \begin{equation} \hat{x} = \frac{\Sigma_ix_i/\sigma^2_{x,i}}{\Sigma_i1/\sigma^2_{x,i}} \end{equation}
   • $$ \frac{1}{\sigma^2} = \sum_i \frac{1}{\sigma_{x,i}^2}$$
   • Result:
     • $\mu = 0.0557$, $\sigma^2 = 0.0018$
     • 95% confidence interval: -2.83% - 13.97%

I did some simplified simulations for deltas in the range of +/-16% which were close to the number I got from the above strategy.