TL;DR - Final formula is at the end. It would be great if someone can verify that I didn't make any mistakes.
If we exchange token x for token y, as per the constant product formula:
x * y = k
Let a be the amount of x we are exchanging to get b amount of y. Therefore:
(x + a) * (y - b) = k
The execution price of the trade, by definition, is just b / a.
If our target execution price is e, then b / a = e => b = ea
Therefore:
(x + a) * (y - ea) = k
But x * y is also equal to k, therefore:
x * y = (x + a) * (y - ea)
Now we can just rewrite the equation to get a in terms of the other variables.
x * y = x * (y - ea) + a * (y - ea)
xy = xy - eax + ay - ea^2
- eax + ay - ea^2 = 0
-ea^2 + a(y - ex) = 0
We know a is not zero, else the price would be undefined which is not possible. Because a is not zero, we can safely divide across by a:
-ea + y - ex = 0
Now a few more slight adjustments:
ea = y - ex
a = (y - ex) / e
a = (y / e) - x
So this is the final formula:
a = (y / e) - x
where a is the maximum amount we can trade to get an execution price of e or better, and x and y are the number of input and output tokens in the pool before the trade respectively.
TEST CASE - where 1 ETH is worth 2,000 USD
x = 100 ETH
y = 200,000 USD
e = 1,950
a = (y / e) - x
= (200,000 / 1,950) - 100
= 2.564 ETH