The functional form of the relationship is not linear;
since $f(a) = \frac{a}{1+a}$, it looks somewhat similar to a logistic form (it's logistic in $\ln a$), which suggests you might be able to try a quasibinomial model with logit link... but because the argument to $f$ is $\alpha+\beta X$ that won't work - we can't take logs of the two terms separately.
That is, for a quasibinomial GLM you can fit a model of the form $E(\frac{Y}{M}) = \frac{\exp(\alpha+\beta x)}{1+\exp(\alpha+\beta x)}$, but your specification indicates you want it without the "$\exp$":
$E(\frac{Y}{M}) = \frac{\alpha+\beta x}{1+\alpha+\beta x}$
I don't think you can do that with a GLM with the response in that form.
However, I do see two possibilities:
Use nonlinear least squares on $E(\frac{Y}{M}) = \frac{\alpha+\beta x}{1+\alpha+\beta x}$
Let $m=\mu_Y(x)=E(Y|x)$. Since you can transform the relationship $m = \frac{\alpha+\beta x}{1+\alpha+\beta x}$ to $\frac{m}{1-m} = \alpha+\beta x$, you might consider whether a model of the form $E[\frac{y/M}{1\:-\:y/M}]=E[\frac{y}{M-y}] = \alpha+\beta x$ could be fitted as a linear regression -- or indeed as a (possibly quasi-) GLM, which could allow you to choose a variance function.
Whether you choose 1 or 2 or something else depends on how you think the error term behaves - you need to consider the question "how does the error term enter the relationship between $y$ and $x$ and how is its variance related either to $x$ or to the mean?". Further, if the you're interested in unbiased estimation on the scale of $y$ you may need to correct the predictions of the backtransformed model, say via a Taylor expansion.