In RBJ's Audio EQ Cookbook, the analog prototype for a low shelf filter is given as such:
$$H(s)=A\cdot\frac{s^2+\left(\frac{\sqrt{A}}{Q}\right)\cdot s + A}{A\cdot s^2 + \left(\frac{\sqrt{A}}{Q}\right)\cdot s + 1}$$
Where does this prototype come from? From what I understand, the lowpass and highpass filters are standard Butterworth filters. But how was this particular transfer function derived? Is it some transformation of a Butterworth filter?
If you remove (for the time being) that leading factor $A$ as a constant gain factor:
$$H(s)=\frac{s^2+\left(\frac{\sqrt{A}}{Q}\right)s + A}{As^2 + \left(\frac{\sqrt{A}}{Q}\right)s + 1}$$
what you get then is a symmetric, but otherwise general shelf that could be equally described as "LowShelf" or "HighShelf". In dB, the gain at the low end is the negative of the dB gain at the high end. And, in log gain (dB) vs. log frequency (octaves or decades), the gain function always has odd symmetry about the "resonant frequency" (more accurately described as the "significant frequency" which is the shelf mid-point frequency, a little different from the biquad resonant frequency), which in the normalized case, is $\Omega_0=1$. At that shelf mid-point frequency this general low/high shelf has a gain of $1$ (or 0 dB).
$$\begin{align} \Big|H(s)\Big|^2 &= \left| \frac{s^2+\left(\frac{\sqrt{A}}{Q}\right)s + A}{As^2 + \left(\frac{\sqrt{A}}{Q}\right)s + 1} \right|^2 \\ \\ \Big|H(j\Omega)\Big|^2 &= \left| \frac{(j\Omega)^2+\left(\frac{\sqrt{A}}{Q}\right)(j\Omega) + A}{A(j\Omega)^2 + \left(\frac{\sqrt{A}}{Q}\right)(j\Omega) + 1} \right|^2 \\ \\ &= \left| \frac{A-\Omega^2+j\frac{\sqrt{A}}{Q}\Omega}{1-A\Omega^2+j\frac{\sqrt{A}}{Q}\Omega} \right|^2 \\ \\ &= \frac{\Big|A-\Omega^2+j\frac{\sqrt{A}}{Q}\Omega \Big|^2}{\Big|1-A\Omega^2+j\frac{\sqrt{A}}{Q}\Omega\Big|^2} \\ \\ &= \frac{(A-\Omega^2)^2+\left(\frac{\sqrt{A}}{Q}\Omega\right)^2}{(1-A\Omega^2)^2+\left(\frac{\sqrt{A}}{Q}\Omega\right)^2} \\ \\ &= \frac{ A^2-2A\Omega^2+\Omega^4 + \frac{A}{Q^2} \Omega^2}{1-2A\Omega^2+A^2\Omega^4 + \frac{A}{Q^2} \Omega^2} \\ \\ &= \frac{ A+\frac{1}{A}\Omega^4 - (2 - \frac{1}{Q^2}) \Omega^2}{\frac{1}{A}+A\Omega^4 - (2 -\frac{1}{Q^2}) \Omega^2} \\ \\ &= \frac{ \frac{A}{\Omega^2}+\frac{\Omega^2}{A} - (2 - \frac{1}{Q^2})}{\frac{1}{A\Omega^2}+A\Omega^2 - (2 -\frac{1}{Q^2})} \\ \end{align}$$
Remember that the "significant frequency" is set to one. If you express the normalized frequency in terms of log frequency, then
$$ \Omega \triangleq 2^p $$
where $p$ is the log frequency, in octaves, centered at the "significant frequency".
$$\begin{align} \Big|H(j2^p)\Big|^2 &= \frac{ \frac{A}{2^{2p}}+\frac{2^{2p}}{A} - (2 - \frac{1}{Q^2})}{\frac{1}{A2^{2p}}+A2^{2p} - (2 -\frac{1}{Q^2})} \\ \\ &= \frac{2^{2p-\log_2(A)} + 2^{-2p+\log_2(A)} - (2 - \frac{1}{Q^2})}{2^{2p+\log_2(A)} + 2^{-2p-\log_2(A)} - (2 - \frac{1}{Q^2})} \\ \\ &= \frac{2 \cosh\big(\log(2)(2p-\log_2(A))\big) - (2 - \frac{1}{Q^2})}{2 \cosh\big(\log(2)(2p+\log_2(A))\big) - (2 - \frac{1}{Q^2})} \\ \\ &= \frac{\cosh\big(2\log(2)p-\log(A)\big) - (1 - \frac{1}{2Q^2})}{\cosh\big(2\log(2)p+\log(A)\big) - (1 - \frac{1}{2Q^2})} \\ \end{align}$$
Now if we express this as log amplitude (let's say in dB) vs. log frequency (in octaves w.r.t. the "significant frequency"), this becomes
$$\begin{align} \tfrac{10}{\log(10)} \log \left(\Big|H(j2^p)\Big|^2 \right) &= \tfrac{10}{\log(10)} \log \left(\frac{\cosh\big(2\log(2)p-\log(A)\big) - (1-\frac{1}{2Q^2})}{\cosh\big(2\log(2)p+\log(A)\big) - (1-\frac{1}{2Q^2})}\right) \\ \\ &= \tfrac{10}{\log(10)} \log \left(\cosh\big(2\log(2)p-\log(A)\big) - (1-\tfrac{1}{2Q^2})\right) \\ & \qquad - \tfrac{10}{\log(10)} \log \left(\cosh\big(2\log(2)p+\log(A)\big) - (1-\tfrac{1}{2Q^2})\right) \\ \\ &= \tfrac{10}{\log(10)} \log \left(\cosh\big(\log(A)-2\log(2)p\big) - (1-\tfrac{1}{2Q^2})\right) \\ & \qquad - \tfrac{10}{\log(10)} \log \left(\cosh\big(\log(A)+2\log(2)p\big) - (1-\tfrac{1}{2Q^2})\right) \\ \end{align}$$
Since $\cosh(\cdot)$ is an even-symmetry function, I think you can show that the function above is an odd-symmetry function in $p$. Substitute $-p$ in for $p$ and you will see that the + and - terms swap.
The slope (in dB/octave) of this curve is
$$\begin{align} \frac{\mathrm{d}}{\mathrm{d}p} \tfrac{10}{\log(10)} \log \left(\Big|H(j2^p)\Big|^2 \right) &= \tfrac{10}{\log(10)} \frac{\mathrm{d}}{\mathrm{d}p} \log \left(\cosh\big(2\log(2)p-\log(A)\big) - (1-\tfrac{1}{2Q^2})\right) \\ & \qquad - \tfrac{10}{\log(10)} \frac{\mathrm{d}}{\mathrm{d}p} \log \left(\cosh\big(2\log(2)p+\log(A)\big) - (1-\tfrac{1}{2Q^2})\right) \\ \\ \\ &= \tfrac{10}{\log(10)} \frac{2\log(2)\sinh\big(2\log(2)p-\log(A)\big)}{ \cosh\big(2\log(2)p-\log(A)\big) - (1-\tfrac{1}{2Q^2})} \\ & \qquad - \tfrac{10}{\log(10)} \frac{2\log(2)\sinh\big(2\log(2)p+\log(A)\big)}{ \cosh\big(2\log(2)p+\log(A)\big) - (1-\tfrac{1}{2Q^2})} \\ \end{align}$$
When $p=0$ (at the midshelf) this shelf slope is:
$$\begin{align} \frac{\mathrm{d}}{\mathrm{d}p} \tfrac{10}{\log(10)} \log \left(\Big|H(j2^p)\Big|^2 \right) \Bigg|_{p=0} &= \tfrac{10}{\log(10)} \frac{2\log(2)\sinh(-\log(A))}{ \cosh(-\log(A)) - (1-\tfrac{1}{2Q^2})} \\ & \qquad - \tfrac{10}{\log(10)} \frac{2\log(2)\sinh(\log(A))}{\cosh(\log(A)) - (1-\tfrac{1}{2Q^2})} \\ \\ &= - \tfrac{10}{\log(10)} \frac{4\log(2)\sinh(\log(A))}{ \cosh(\log(A)) - (1-\tfrac{1}{2Q^2})} \\ \\ &= - \tfrac{10}{\log(10)} \frac{2\log(2) \left(A - \tfrac{1}{A} \right) }{ \tfrac{1}{2} \left(A + \tfrac{1}{A} \right) - (1-\tfrac{1}{2Q^2})} \\ \\ &= - \tfrac{40\log(2)}{\log(10)} \ \frac{A^2 - 1}{A^2 + 1 - 2A(1-\tfrac{1}{2Q^2})} \\ \end{align} $$
To find (or to determine the existences of) maxima or minima, we set the derivative to zero.
$$\begin{align} \frac{\mathrm{d}}{\mathrm{d}p} \tfrac{10}{\log(10)} \log \left(\Big|H(j2^p)\Big|^2 \right) &= \tfrac{10}{\log(10)} \frac{2\log(2)\sinh\big(2\log(2)p-\log(A)\big)}{ \cosh\big(2\log(2)p-\log(A)\big) - (1-\tfrac{1}{2Q^2})} \\ & \qquad - \tfrac{10}{\log(10)} \frac{2\log(2)\sinh\big(2\log(2)p+\log(A)\big)}{ \cosh\big(2\log(2)p+\log(A)\big) - (1-\tfrac{1}{2Q^2})} \\ \\ &= 0 \\ \end{align}$$
That means:
$$ \sinh\big(2\log(2)p-\log(A)\big) \left( \cosh\big(2\log(2)p+\log(A)\big) - (1-\tfrac{1}{2Q^2}) \right) \\ \qquad = \sinh\big(2\log(2)p+\log(A)\big) \left( \cosh\big(2\log(2)p-\log(A)\big) - (1-\tfrac{1}{2Q^2}) \right) $$
or
$$ \left( \frac{2^{2p}}{A} - \frac{A}{2^{2p}} \right) \left( A 2^{2p} + \frac{1}{A 2^{2p}} - 2 + \frac{1}{Q^2} \right) \\ \qquad \qquad = \left( A 2^{2p} - \frac{1}{A 2^{2p}} \right) \left( \frac{2^{2p}}{A} + \frac{A}{2^{2p}} - 2 + \frac{1}{Q^2} \right) $$
or
$$ 2^{4p} - A^2 + \frac{1}{A^2} - \frac{1}{2^{4p}} - \left( \frac{2^{2p}}{A} - \frac{A}{2^{2p}} \right)\left(2 - \frac{1}{Q^2} \right) \\ \qquad \qquad = 2^{4p} - \frac{1}{A^2} + A^2 - \frac{1}{2^{4p}} - \left( A 2^{2p} - \frac{1}{A 2^{2p}} \right) \left( 2 - \frac{1}{Q^2} \right) $$
or
$$\begin{align} \frac{1}{A^2} - A^2 &= \left( \frac{2^{2p}}{A} - \frac{A}{2^{2p}} - A 2^{2p} + \frac{1}{A 2^{2p}} \right)\left(1 - \frac{1}{2Q^2} \right) \\ \\ &= \left( 2^{2p}\left(\frac{1}{A} - A \right) - \frac{1}{2^{2p}}\left(A - \frac{1}{A}\right) \right)\left(1 - \frac{1}{2Q^2} \right) \\ \\ &= \left( 2^{2p}\left(\frac{1}{A} - A\right) + \frac{1}{2^{2p}}\left(\frac{1}{A} - A\right) \right)\left(1 - \frac{1}{2Q^2} \right) \\ \\ &= \left(\frac{1}{A} - A\right) \left( 2^{2p} + \frac{1}{2^{2p}} \right)\left(1 - \frac{1}{2Q^2} \right) \\ \end{align}$$
or
$$\begin{align} \frac{1}{A} + A &= \left( 2^{2p} + \frac{1}{2^{2p}} \right)\left(1 - \frac{1}{2Q^2} \right) \\ \\ \cosh\big(\log(A)\big) &= \cosh\big(2 \log(2) p\big) \left(1 - \frac{1}{2Q^2} \right) \\ \end{align}$$
Now the $\cosh(\cdot)$ on both sides must be positive, so this cannot have a solution for any finite $p$ while $Q \le \sqrt{\frac12}$. That means the shelf gain is monotonically increasing or monotonically decreasing.
But when $Q > \sqrt{\frac12}$, then the log frequency $p$ can be directly solved for in the equation above and will have two values $\pm p$ which means an equal number of octaves above and below the shelf midpoint frequency.
So we know that the steepest monotonic slope ($Q = \sqrt{\frac12}$) of the shelf at the shelf midpoint ($p=0$) is
$$ - \tfrac{40\log(2)}{\log(10)} \ \frac{A^2 - 1}{A^2 + 1} $$
and that the general $Q$ shelf slope is:
$$ - \tfrac{40\log(2)}{\log(10)} \ \frac{A^2 - 1}{A^2 + 1 - 2A(1-\tfrac{1}{2Q^2})} $$
The shelf slope parameter is defined simply as the ratio of the two slopes:
$$\begin{align} S & \triangleq \frac{\tfrac{40\log(2)}{\log(10)} \ \frac{A^2 - 1}{A^2 + 1 - 2A(1-\tfrac{1}{2Q^2})}}{\tfrac{40\log(2)}{\log(10)} \ \frac{A^2 - 1}{A^2 + 1}} \\ \\ & = \frac{A^2 + 1}{A^2 + 1 - 2A(1-\tfrac{1}{2Q^2})} \\ \\ & = \frac{1}{1 - \frac{2A}{A^2 + 1}(1-\tfrac{1}{2Q^2})} \\ \end{align}$$
I think that's how it comes out in the Cookbook, ain't it?
If you use a parameterization with a (pole or zero) frequency and a Q-factor for numerator and denominator of a biquadratic function you get the following general second-order transfer function
$$H(s)=G_{\infty}\frac{s^2+\frac{\omega_z}{Q_z}s+\omega_z^2}{s^2+\frac{\omega_p}{Q_p}s+\omega_p^2}\tag{1}$$
For a low shelving filter we want $H(\infty)=1$, i.e., $G_{\infty}=1$. The DC gain equals
$$H(0)=\frac{\omega_z^2}{\omega_p^2}=A^2\tag{2}$$
We're free to choose a frequency normalization, i.e., fix either $\omega_p$ or $\omega_z$ and choose the other according to our specified DC gain $A^2$. If we choose $\omega_z=\sqrt{A}$, we obtain $\omega_p=1/\sqrt{A}$. With this normalization, and with $G_{\infty}=1$, Eq. $(1)$ becomes
$$H(s)=\frac{s^2+\frac{\sqrt{A}}{Q_z}s+A}{s^2+\frac{1}{\sqrt{A}Q_p}s+\frac{1}{A}}\tag{3}$$
For symmetry reasons we would like the inverse transfer function $1/H(s)$ to equal the original transfer function with the inverse DC gain. I.e., replacing $A$ by $1/A$ in $(3)$ and equating it with $1/H(s)$ results in the final requirement $Q_z=Q_p=Q$. This yields the final transfer function
$$H(s)=\frac{s^2+\frac{\sqrt{A}}{Q}s+A}{s^2+\frac{1}{\sqrt{A}Q}s+\frac{1}{A}}=A\frac{s^2+\frac{\sqrt{A}}{Q}s+A}{As^2+\frac{\sqrt{A}}{Q}s+1}\tag{4}$$
Note that the chosen frequency normalization results in
$$|H(j\omega_0)|=A,\qquad\omega_0=1\tag{5}$$
i.e., the gain at $\omega_0=1$ is half the DC gain (in dB).
