In continuous games, the probability distributions over the players' strategy spaces are infinite. How then is it even possible to then derive a mixed-strategy nash equilibrium?
One would have to show that the mixed-strategy of one player is the best possible mixed-strategy response to the strategy of another player. In the continuous case, how is this even possible? How do you maximize over "all possible probability distributions".
Can anybody provide a clue?
The "trick" is to identify an equilibrium action where you know (or can pin-down) the payoff. You can then impose indifference between this action's known payoff and the payoffs from other actions (which will depend on the mixing probabilities) to determine those equilibrium mixing probabilities.
A nice example can be found in a Varian's "A Model of Sales". Let me describe a simple version of that model before discussing what it teaches us about finding continuous mixed equilibria.
Set up
Two firms each set a price, $p_i$, for a homogeneous product they product at zero cost. There is a continuum of consumers, each willing to pay up to $v$ for the product.
A fraction $a$ of the consumers (call them "shoppers") will buy from whichever firm sets the lowest price (breaking ties randomly). A fraction $\frac{1-a}{2}$ of the consumers (call them "1's captives") always buy from firm $1$ (as long as $p_1<v$). The final $\frac{1-a}{2}$ of the consumers (call them "2's captives") always buy from firm $2$ (as long as $p_2<v$).
You should be able to quickly check that this game has no pure strategy equilibrium (think about why we can't have an equilibrium with $0<p_1<p_2$, $0<p_1=p_2$, or $0=p_1=p_2$).
Thus, we look for a mixed strategy equilibrium where prices are drawn from the distribution $F$. Let the support of $F$ (i.e., the range of prices over which firms mix) be $[\underline{p},\overline{p}]$.
No firm will ever choose $p_i>v$ because it then serves no consumers, but could earn a profit of $v(1-a)/2$ by serving its captives at a price of $v$. Thus, $\overline{p}\leq v$.
A firm's profit if it sets price $p$ is $$\frac{1-a}{2}p+ap[1-F(p)].$$ The first term is the profit from captives (who the firm serves for sure). The second term is the profit from shoppers, who are served only if the rival firm sets a higher prices (which happens with probability $1-F(p)$).
Finding the mixed-strategy equilibrium
We know that if the firm is to mix over prices $p$ and $\overline{p}$ then it must be indifferent: $$\frac{1-a}{2}p+ap[1-F(p)]=\frac{1-a}{2}\overline{p}+a\overline{p}[1-F(\overline{p})].$$ Notice that if we could eliminate the $\overline{p}$ terms from the right-hand side then we would have an equation that could be solved for the equilibrium strategy, $F$.
The next part is the key to being able to find the equilibrium, so I'll put it in a box:
Let's think about what happens when a firm sets $p_i=\overline{p}$. Because $\overline{p}$ is the highest price ever charged in equilibrium, we know that the firm never serves the shoppers at that price (i.e., $F(\overline{p})=1$). Moreover, since the firm serves only its captives who will buy at any price below $v$, it must be the case that $\overline{p}=v$.
We therefore know that:
Putting these facts together with the indifference condition, we have $$\frac{1-a}{2}p+ap[1-F(p)]=\frac{1-a}{2}v,$$ which implies $$F(p)=1-\frac{\frac{1-a}{2}(v-p)}{ap},$$ which is our mixed strategy equilibrium. The only part of the equilibrium we haven't solved is the value of $\underline{p}$. But this is easily found by solving $F(\underline{p})=0$ for $\underline{p}$ now that we know $F(\cdot)$.
Final remarks
The trick used in the above example is pretty useful more generally.
Unless the equilibrium strategy, $F$, has an atom, we know by definition that $F(\underline{p})=0$ and $F(\overline{p})=1$. Thus, if we can use economic reasoning to determine the value of either $\overline{p}$ or $\underline{p}$ (as in the box above) then we can eliminate all of the terms that depend on $\overline{p}$ (or $\underline{p}$) on the right hand side of our indifference equation and then solve for $F$.
I am not sure if you are looking for a very abstract, intuitive trick for finding mixed Nash equilibrium, or just an example of how to possibly find one.
I will assume the latter.
Suppose we have the following game:
$$ L \ \ \ \ \ \ |\ \ R \\ U (1,1) | (0,0) \\ D (0,0) | (1,1) $$
Clearly $(U,L)$ and $(D,R)$ are NE (they are pure NE). However, there is a mixed strategy NE too. How can we find this?
We do the following:
Suppose Player 1 mixes, going $U$ with probability $\alpha$, and $D$ with probability $1-\alpha$. (So $\alpha \in [0,1]$) That means Player 2 must have the same expected outcome whether they go $L$ or $R$.
So, we can express Player 2's strategy like this:
If they went $L$, their expected pay-off is:
$$1\times \alpha + 0 \times (1-\alpha) = \alpha$$
If they went $R$, their expected pay-off is:
$$0 \times \alpha + 1 \times (1-\alpha) = 1 -\alpha$$
Remember, their expected outcome from $L$ and $R$ must be equal, thus:
$$\alpha = 1-\alpha \Rightarrow 2\alpha = 1 \Rightarrow \alpha = \frac{1}{2}$$
What we just did is solve for Player 1's mixed strategy- going $U$ with probability $\frac{1}{2}$, and $D$ with probability $1-\frac{1}{2} = \frac{1}{2}$.
We can find Player 2's probabilities in a similar way. Note that the process to do this is different depending on the type of game- but their main "trick" here is noting in mixed strategy equilibrium, expected outcomes over the strategies must be equal.
