I recently received the following question via email. I'll post an answer below, but I was interested to hear what others thought.
Would you call logistic regression a non-parametric test? My understanding is that simply labelling a test non-parametric because its data is not normally distributed, is insufficient. Its more to do with lack of assumptions. logistic regression does have assumptions.
Larry Wasserman defines a parametric model as a set of distributions "that can be parameterized by a finite number of parameters." (p.87) In contrast a nonparametric model is a set of distributions that cannot be paramterised by a finite number of parameters.
Thus, by that definition standard logistic regression is a parametric model. The logistic regression model is parametric because it has a finite set of parameters. Specifically, the parameters are the regression coefficients. These usually correspond to one for each predictor plus a constant. Logistic regression is a particular form of the generalised linear model. Specifically it involves using a logit link function to model binomially distributed data.
Interestingly, it is possible to perform a nonparametric logistic regression (e.g., Hastie, 1983). This might involve using splines or some form of non-parametric smoothing to model the effect of the predictors.
I'd say logistic regression isn't a test at all; however a logistic regression may then lead to no tests or several tests.
You're quite correct that labelling something nonparametric because it's not normal is insufficient. I'd call the exponential family explicitly parametric, so I'd usually regard logistic regression (and Poisson regression and Gamma regression and ...) as parametric, though there can be circumstances in which I might accept an argument that particular logistic regressions could be regarded as nonparametric (or at least in a vaguely hand-wavy sense, only quasi-"parametric").
Beware any confusion over the two senses in which a regression may be called nonparametric.
If I fit a Theil linear regression it is nonparametric in the sense that I have left the error distribution undefined (it corresponds to adjusting the regression slope until the Kendall correlation between residuals and $x$ is 0) ... but it is parametric in the sense that I have a fully specified relationship between $y$ and $x$ parameterized by the slope and intercept coefficients.
If on the other hand I fit a kernel polynomial regression (say a local linear regression), but with normal errors, that is also called nonparametric, but in this case it's the parameterization of the relationship between $y$ and $x$ that's nonparametric (at least potentially infinite-dimensional), not the error distribution.
Both senses are used, but when it comes to regression, the second kind is actually used more often.
It's also possible to be nonparametric in both senses, but harder (with sufficient data, I could, for example, fit a Theil locally-weighted linear regression).
The two senses aren't quite as distinct as it may seem at first, however, since if we consider the model as specifying the conditional distribution of the response, then the second sense is specifically about modelling the location (typically the mean) of that conditional distribution, while the first sense relates to the model for the shape of the conditional distribution model. They're both relating to aspects of the conditional distribution. Going back to that second sense, if the distribution is otherwise specified (up to a fixed finite number of parameters) it might be better described as semiparametric, but that's not the convention in this area.
In the case of GLMs, the second form of nonparametric multiple regression include GAMs; that second form is the sense in which Hastie is generally operating (and under which he's operating in that quote).
One helpful distinction that might add a little to the answers above: Andrew Ng gives a heuristic for what it means to be a non-parametric model in Lecture 1 from the course materials for Stanford's CS-229 course on machine learning.
There Ng says (pp. 14-15):
Locally weighted linear regression is the first example we’re seeing of a non-parametric algorithm. The (unweighted) linear regression algorithm that we saw earlier is known as a parametric learning algorithm, because it has a fixed, finite number of parameters (the $\theta_{i}$’s), which are fit to the data. Once we’ve fit the $\theta_{i}$’s and stored them away, we no longer need to keep the training data around to make future predictions. In contrast, to make predictions using locally weighted linear regression, we need to keep the entire training set around. The term “non-parametric” (roughly) refers to the fact that the amount of stuff we need to keep in order to represent the hypothesis $h$ grows linearly with the size of the training set.
I think this is a useful contrasting way to think about it because it infuses the notion of complexity directly. Non-parametric models are not inherently less-complex, because they may require keeping much more of the training data around. It just means that you're not reducing your use of the training data by compressing it down into a finitely parameterized calculation. For efficiency or unbiasedness or a host of other properties, you may want to parameterize. But there may be performance gains if you can afford to forgo parameterizing and keep lots of the data around.
Hastie and Tibshirani defines that linear regression is a parametric approach since it assumes a linear functional form of f(X). Non-parametric methods do not explicitly assume the form for f(X). This means that a non-parametric method will fit the model based on an estimate of f, calculated from the model. Logistic regression establishes that p(x) = Pr(Y=1|X=x) where the probability is calculated by the logistic function but the logistic boundary that separates such classes is not assumed, which confirms that LR is also non-parametric
I think logistic regression is a parametric technique.
This might be helpful, from Wolfowitz (1942) [Additive Partition Functions and a Class of Statistical Hypotheses The Annals of Mathematical Statistics, 1942, 13, 247-279]:
“the distribution functions [note: plural!!!] of the various stochastic variables which enter into their problems are assumed to be of known functional form, and the theories of estimation and of testing hypotheses are theories of estimation of and of testing hypotheses about, one or more parameters, finite in number, the knowledge of which would completely determine the various distribution functions involved. We shall refer to this situation for brevity as the parametric case, and denote the opposite situation, where the functional forms of the distributions are unknown’, as the non-parametric case.
Also, having heard this discussed rather a lot, I found this amusing by Noether (1984) [Nonparametrics: The Early Years-Impressions and Recollections The American Statistician, 1984, 38, 173-178]:
“The term nonparametric may have some historical significance and meaning for theoretical statisticians, but it only serves to confuse applied statisticians.”
