is it possible to obtain a closed-form solution w.r.t. ${P_j:\forall j}$ (or in terms of special functions) for the following equations:
$\alpha P_0=P_1$, $\alpha<1$
$\alpha P_j=P_{j+1}+P_{j+2}+\dots+P_{2j+1}$ for $j=1,2,....$
$\sum_{i=1}^\infty P_i=1$
$P_i\geq 0, \forall i$
OR let me put the very original equations below:
$\lambda P_0=\mu P_1$
$\lambda P_{j-1} + \mu (P_{2j}+P_{2j+1})=(\lambda+\mu)P_j, \forall j>0$
$\sum_{i=1}^\infty P_i=1$
$P_i\geq 0, \forall i$
If you set $\alpha=1$ and impose the condition that $P_{2i}=P_{2i+1}$, then you get the sequence $P_0,P_2,P_4,... = 1,1/2,0,1/4,-1/4,0,1/4,1/8,-3/8,-1/8,1/8,0,...$. Define $Q_{2i}=P_{2i}-P_{2i-2}$. The pattern of these differences is simpler: $Q_2,Q_4,Q_6,...=-1/2,-1/2,1/4,-1/2,1/4,-1/8,-1/2,1/4,...,(-1/2)^{b(i)}$ where $b(i)$ is the number of $1$s in the binary expansion of $i$. To prove this, one can check that $Q_{2k}=Q_{4k}=-2Q_{4k+2}$ using $P_{2k}=2P_{4k+2}$.
Since there are arbitrarily large numbers of low binary weight, the differences $Q_{2i}$ do not converge to $0$ so the terms $P_i$ do not converge to $0$, so the sum does not converge and can't be normalized to $1$. Nevertheless, for other values of $\alpha$ I expect that you can produce a convergent series this way involving the binary weight function $b$.
For $\alpha \ne 1$, we can still impose that $P_{2i}=P_{2i+1}$ for $i\gt 0$. It appears that there is the following formula for $Q_{2i}$ where $2i =2^j + \epsilon 2^{j-1} + k$, $\epsilon \in \{0,1\}, k \lt 2^{j-1}$,$2i \gt 0$:
$$Q_{2i} = 2^{-j} \alpha^2 (\alpha-3)(\alpha+1)^{j-2} \left( -\frac{\alpha-2}{\alpha-3} \right)^\epsilon \left(-\frac{\alpha}{\alpha+1} \right)^{b(k)}$$
This means $Q_{2i}$ can be written as a product over the binary digits of $2i$ where the first two digits are special, or we can express $Q_{2i}$ recursively.
$Q_4 = \frac{1}{4} \alpha^2 (\alpha-3)$
$Q_6 = -\frac{1}{4} \alpha^2 (\alpha-2)$
And then for $k \gt 1$,
$Q_{4k} = \frac{1+\alpha}{2} Q_{2k}$
$Q_{4k+2} = -\frac{\alpha}{2} Q_{2k}$
I expect that the proofs for the case $\alpha=1$ don't need to be modified significantly to prove these.
This question is a moving target, so this answer no longer satisfies all of the conditions. I think I'm done.
There is no solution to the problem in the first version of the OP.
Proof:
We have to consider two cases:
Case a) at least one of the $P_j$ is zero
The let $k$ be the smallest index for which $P_k = 0$.
Then from
$0 = \alpha P_k = P_{k+1} + ... + P_{2k+1}$
we have
$P_{k+1} = P_{k+2} = .. = P_{2k+1} = 0$
and, inductively, $P_j = 0$ if $j \ge k$.
On the other hand
$\alpha P_{k-1} = P_{k} + ... = 0$
and so on downwards so that all $P_j = 0$ which contradicts the normalization condition. Hence we can rule out case a)
Case b) all $P_j$ are positive
I shall show that there is no solution to the recursive equations with
(1) $P_j \gt 0, j = 1, 2, 3, ...$
First from
$\alpha P_1 = P_2 + P_3$
we conclude
$\alpha \gt 0$
Notice also the $P_0$ appears only in the relation
$\alpha P_0 = P_1$
which shows that
$P_0 \gt 0$ as well, but $P_0$ does not appear in the normalization. Therefore we consider it as a mere abrevíation for $P_1/ \alpha $.
Now we transform the recursive relation into a standard form, which we define here to be one in which an element with a specific index is defined in terms of elements with smaller indices.
Define
(2) $Q_i = P_{i+1}+P_{i+2}+..., i = 0,1,2,...$
As a sum over positive quantities we have
$Q_i \gt 0, i = 0, 1, 2, ...$
The inversion of (2) is
(3) $P_i = Q_{i-1} - Q_i , i = 1, 2, ... $
Now the equations become
$\alpha P_j = Q_j - Q_{2j+1}, j =1, 2, 3, ... $
Using (3) we get
$\alpha (Q_{j-1}-Q_j) = Q_j - Q_{2j+1}$
or
(4) $Q_{2j+1} = (1+\alpha ) Q_j - \alpha Q_{j-1}, j = 1, 2, ...$
This is now a recursive relation in standard form.
The inital values are
$Q_0 = P_1 + P_2 + ... = 1$
because of the normalization condition.
And
$Q_1 = 1 - P_1 = 1 - \alpha P_0$
can be considered as a free parameter in the interval (0,1).
Before we solve (4) we observe that it defines only the elements with an odd index.
Therefore we let
$Q_{2k} = C_k > 0, k = 1, 2, ...$
with arbitrary $C_k$ in the interval (0,1).
Performing now the first few steps of the solution to (4) the reader will find that
$P_{10} = - C_5 - \alpha (1+\alpha ) Q_1$
But this is a negative quantity, and the contradiction proves the statement.
