It seems like your question and comments come from the same place where the sentiment—"machine learning is just curve fitting"—comes from. The sentiment is true at a low level. But statistical modeling is a useful concept because it systematically tells you what curves you should be looking for. Statistical modeling is a language to turn assumptions about data into objectives that computers can solve using observations. Sure, these objectives are well known for tasks such as plain classification or plain regression. But new tasks, new variants of tasks, or new ways of mapping tasks to existing statistical models, come up all the time. And creative specifications of the statistical model happen to be fruitful for cutting-edge machine learning research. More on that later.
Basic idea
Before demonstrating the usefulness of statistical modeling, let's roughly define it. The other answers correctly tell you to abandon the overly-constrained definition of $Y = m(x) + \epsilon$. So what's a better, more generic definition? A statistical model is a random variable, representing output data, which is a function of input data; the model specifies a random process by which input data is transformed into output data. Under this definition, unlike $Y = m(x) + \epsilon$, you don't need to get hung up on explicitly defining the distribution of random errors. (Answers to this other CV question speak a bit more to that.) For example, here's a specification of the statistical model underlying logistic regression:
$$
m(x) = \beta_0 + \beta_1 x \\
f(x) = \sigma(m(x)) \\
Y_i \sim \text{Bernoulli}(f(x_i)) \text{ independent}.
$$
Two questions you may have:
If I believe the output data is a deterministic transformation of the input, why would I model it as if it's random?
How did specifying $Y_i \sim \dots$ make any progress on the problem of predicting new data? What's the need for this formalism?
Re the first question: a lot can be said. I'll leave it to you to read these sections of these ML textbooks (which are freely available online):
Now that we've established that a random model is realistic for the purposes of prediction, let's establish that $Y_i \sim \dots$ represented progress on the problem.
Go back to the logistic regression model above. To make new predictions, we just need to "fill in" the values of $\beta_0$ and $\beta_1$ using training data. A principled way to do that is to estimate them via maximum likelihood estimation (MLE). (Section 5.5.2 of Deep Learning1 summarizes some of the theoretical justification for MLE.) It's in expressing the likelihood function that the formalism $Y_i \sim \dots$ is required:
$$
\begin{align*}
\mathcal{L}(\theta \: | \: Y_1 = y_1, \dots, Y_n=y_n) &= \prod_{i=1}^{n} {\Pr}_\theta(Y_i = y_i) & \text{independence} \\
&= \prod_{i=1}^{n} {\Pr}_\theta(Y_i = 1)^{y_i} (1 - {\Pr}_\theta(Y_i = 1))^{(1 - y_i)} & \text{Bernoulli} \\
&= \dots & \text{take log, simplify.}
\end{align*}
$$
The theoretical simplicity and empirical effectiveness of this paradigm—
- model output as a random variable
- minimize negative log-likelihood plus regularization
—is at the core of the answer to your question. It's a mistake to take this idea for granted. As Georg M. Goerg put it in their comment:
once you put on your probability distribution hat , you will find that it is much easier (and much higher success rate) to find "natural" loss functions for your ml / decision making problems compared to going by "intuition" alone.
In that comment thread, you claim that one can arrive at the mean squared error objective by appealing to Euclidean distance. Fair enough. But what objective will you use when the output is the time until an event? What objective will you use when the output depends on previous outputs? What objective will you use when the output is a rate, or a category? I'm sure there are non-statistical-looking answers to each of these questions. And perhaps some of them result in a more predictive model! But isn't the statistical modeling approach just so easy? You translate your assumptions about how the data were generated into a probabilistic process, and then maximize the likelihood of this process. Such extensibility is a sign of a good framework.
Examples from deep learning
Perhaps that rough explanation wasn't convincing enough. You may be more convinced if it's demonstrated that many novel deep learning approaches explicitly and intentionally follow the paradigm:
- model output as a random variable
- minimize negative log-likelihood plus regularization.
GPT
Model: let $U$ be a random variable indicating the index of a token / its input ID.
$$
\mathbf{u}_i = (u_{i-k}, u_{i-k+1}, \dots, u_{i-1}) \\
\mathbf{h}_i = \text{Transformer}(\mathbf{u}_i) \\
f(\mathbf{u}_i) = \text{softmax}(W \mathbf{h}_i) \\
U_i \sim \text{Categorical}(f(\mathbf{u}_i)).
$$
Likelihood: see equation (1) of the first GPT paper3 for the likelihood objective, which comes from applying the probability chain rule.
Regularization: the $\text{Transformer}$ typically includes dropout, and the optimizer typically includes weight decay.
ChatGPT
The novelty of InstructGPT4 (a very close cousin of ChatGPT) is in performing Reinforcement Learning with Human Feedback (RLHF)5. An important piece of RLHF's reward modeling subproblem is formulating a loss function based on observations of preferences. Currently, most alignment work assumes preferences are binary.
Model: let $Z$ be a random variable indicating that a language model's sampled response $\mathbf{y}^{(w)}$ is preferred to an alternate sampled response $\mathbf{y}^{(l)}$ for a prompt $\mathbf{x}$. Apply the Bradley-Terry model of binary preference relations. (See section 2.2.3 of the RLHF paper.)
$$
\mathbf{h}(\mathbf{x}, \mathbf{y}) = \text{Transformer}((\mathbf{x}, \mathbf{y})) \\
r(\mathbf{x}, \mathbf{y}) = \mathbf{w}^T \mathbf{h}(\mathbf{x}, \mathbf{y}) \\
f(\mathbf{x}, \mathbf{y}^{(w)}, \mathbf{y}^{(l)}) = \sigma \big( r(\mathbf{x}, \mathbf{y}^{(w)}) - r(\mathbf{x}, \mathbf{y}^{(l)}) \big) \\
Z_i \sim \text{Bernoulli} \big( f(\mathbf{x}_i, \mathbf{y}^{(w)}_i, \mathbf{y}^{(l)}_i) \big) \text{ independent}.
$$
Likelihood: see equation 2 of this paper6 (for a cleaner expression which matches my notation).
Regularization: when training the language model according to the reward model, there's a penalty for deviating too far from its initial distribution.
As new forms of human feedback are experimented with, new loss functions will need to be formulated. Such a task is easier to tackle when you specify a statistical model for how that feedback came about.
Diffusion models
This one takes much more notation, so I'm just going to point you to the concise summary in Section 2 of this paper7. Observe that the specification of the statistical model is quite contrived and creative.
Text embeddings
The top 4 (at the time this post was written) state-of-the-art approaches for embedding text rely on the same sort of loss function described in this paper8.
Model: given a batch of $n$ texts, let $Z_i$ indicate which text in the batch $\mathbf{x}_i$ is similar to. (Below, $\tau$ is a hyperparameter often referred to as temperature.)
$$
\mathbf{h}(\mathbf{x}) = \text{Transformer}(\mathbf{x}) \\
s(\mathbf{x}, \mathbf{x}') = \frac{\mathbf{h}(\mathbf{x})^T \mathbf{h}(\mathbf{x}')}{||\mathbf{h}(\mathbf{x})||_2 ||\mathbf{h}(\mathbf{x}')||_2} \\
s^{(k)}_i = s(\mathbf{x}_i, \mathbf{x}_k) / \tau \\
\mathbf{p}_i = \text{softmax}(s^{(1)}_i, s^{(2)}_i, \dots, s^{(n)}_i) \\
Z_i \sim \text{Categorical}(\mathbf{p}_i).
$$
Likelihood: see, e.g., section 2.2 of the InstructOR paper9.
I wanted to highlight text embedding models because, at first glance, it seems like a purely geometric problem: all we need to do is push similar texts closer in embedding space and dissimilar texts farther. Indeed, many approaches to metric learning involve purely geometric loss functions, e.g., contrastive loss, triplet loss. If we are certain about the similarity and dissimilarities of texts in each batch, how could statistical modeling be warranted? It's hard to pinpoint a reason. I just find it relevant to your question that it is warranted, at least empirically. (Note: if the dissimilar texts are crudely randomly sampled from a pool of texts, then it's clear that modeling similarity as random is more realistic. See word2vec.10)
Object detection
I'm getting kind of tired so just read section 3 of this paper11.
Why doesn't {insert architecture paper} specify a statistical model?
There are many highly impactful ML papers which are focused purely on $m(x)$—the model's architecture. These papers develop computationally efficient ways to flow signal from the input $x$ to, ideally, arbitrary outputs. They don't need to specify statistical models because they seek to outperform other architectures on benchmark tasks, for which appropriate loss functions have already been designed.
Even if the paper includes a novel loss function, it's not always useful to verbosely specify a statistical model. More on that now.
Important disclaimer
(Treat the thoughts below as opinions.)
The statistical modeling paradigm—
- model output as a random variable
- minimize negative log-likelihood plus regularization
—is not a rule in ML, or anything really. Don't stress too much about not having a crystal-clear probabilistic interpretation of your loss function. If summing up a few different loss functions seems reasonable, then try it. If re-weighing terms in a loss function could counteract a problem you've observed, then try it. If it works, it works. The statistical modeling paradigm is useful because it's easy and fast to start with something principled, and then iterate on it or move on. There are bigger fish to fry when you're working on an ML problem. The majority of ML research is not focused on specifying statistical models.
And there are many examples in deep learning where it's not as useful to connect the loss function to a statistical model. A notable example is that training a Generative Adversarial Network is fruitfully thought of in game-theoretic terms.12
But for decades past, and probably some time to come, many of the big ideas in deep learning contain an instance of the high-level recipe: specify statistical model, maximize likelihood.
References
Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Murphy, Kevin P. Probabilistic machine learning: an introduction. MIT press, 2022.
Radford, Alec, et al. "Improving language understanding by generative pre-training." (2018).
Ouyang, Long, et al. "Training language models to follow instructions with human feedback." Advances in Neural Information Processing Systems 35 (2022): 27730-27744.
Christiano, Paul F., et al. "Deep reinforcement learning from human preferences." Advances in neural information processing systems 30 (2017).
Rafailov, Rafael, et al. "Direct preference optimization: Your language model is secretly a reward model." arXiv preprint arXiv:2305.18290 (2023).
Ho, Jonathan, Ajay Jain, and Pieter Abbeel. "Denoising diffusion probabilistic models." Advances in neural information processing systems 33 (2020): 6840-6851.
Oord, Aaron van den, Yazhe Li, and Oriol Vinyals. "Representation learning with contrastive predictive coding." arXiv preprint arXiv:1807.03748 (2018).
Su, Hongjin, et al. "One embedder, any task: Instruction-finetuned text embeddings." arXiv preprint arXiv:2212.09741 (2022).
Mikolov, Tomas, et al. "Distributed representations of words and phrases and their compositionality." Advances in neural information processing systems 26 (2013).
He, Yihui, et al. "Bounding box regression with uncertainty for accurate object detection." Proceedings of the ieee/cvf conference on computer vision and pattern recognition. 2019.
Goodfellow, Ian, et al. "Generative adversarial nets." Advances in neural information processing systems 27 (2014).
