18

I was reading about optimization for an ill-posed problem in computer vision and came across the explanation below about optimization on Wikipedia. What I don't understand is, why do they call this optimization "Energy minimization" in Computer Vision?

An optimization problem can be represented in the following way:

Given: a function $f: A \to R$ from some set $A$ to the real numbers

Sought: an element $x_0$ in $A$ such that $f(x_0) ≤ f(x)$ for all $x$ in $A$ ("minimization") or such that $f(x_0) ≥ f(x)$ for all $x$ in $A$ ("maximization").

Such a formulation is called an optimization problem or a mathematical programming problem (a term not directly related to computer programming, but still in use for example in linear programming – see History below). Many real-world and theoretical problems may be modeled in this general framework. Problems formulated using this technique in the fields of physics and computer vision may refer to the technique as energy minimization, speaking of the value of the function $f$ as representing the energy of the system being modeled.

Community
  • 1
iamprem
  • 283

2 Answers2

12

Energy-based models are a unified framework for representing many machine learning algorithms. They interpret inference as minimizing an energy function and learning as minimizing a loss functional.

The energy function is a function of the configuration of latent variables, and the configuration of inputs provided in an example. Inference typically means finding a low energy configuration, or sampling from the possible configuration so that the probability of choosing a given configuration is a Gibbs distribution.

The loss functional is a function of the model parameters given many examples. E.g., in a supervised learning problem, your loss is the total error at the targets. It's sometimes called a "functional" because it's a function of the (parametrized) function that constitutes the model.

Major paper:

Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, and F. J. Huang, “A tutorial on energy-based learning,” in Predicting Structured Data, MIT Press, 2006.

Also see:

LeCun, Y., & Huang, F. J. (2005). Loss Functions for Discriminative Training of Energy-Based Models. In Proceedings of the 10th International Workshop on Artificial Intelligence and Statistics (AIStats’05). Retrieved from http://yann.lecun.com/exdb/publis/pdf/lecun-huang-05.pdf

Ranzato, M., Boureau, Y.-L., Chopra, S., & LeCun, Y. (2007). A Unified Energy-Based Framework for Unsupervised Learning. Proc. Conference on AI and Statistics (AI-Stats). Retrieved from http://dblp.uni-trier.de/db/journals/jmlr/jmlrp2.html#RanzatoBCL07

Neil G
  • 15,219
  • 5
    Can you expand on what "They interpret inference as minimizing an energy function and learning as minimizing a loss function" means? How is an energy function different than a loss function? – Cliff AB Feb 07 '16 at 21:42
  • Could you please elaborate your answer – iamprem Feb 07 '16 at 21:54
  • @CliffAB Hopefully that's clearer? – Neil G Feb 07 '16 at 21:55
  • @NeilG: to be honest, I'm still slightly confused. To me, it sounds like the "energy function" is essentially the same thing as the likelihood function in statistics. Is that a reasonable interpretation or am I missing something more subtle? – Cliff AB Feb 07 '16 at 22:11
  • @CliffAB: The energy function can be the log-likelihood in which case the total exponentiated energy is one. However, that's not even necessary: Non-probabilistic energy-based models don't worry about this normalization, which can make learning them more efficient than probabilistic models. This is because it avoids evaluating expensive integrals over the configuration space. – Neil G Feb 07 '16 at 22:39
  • 1
    @NeilG Still hard to understand . The energy function is a function of the configuration of latent variables, and the configuration of inputs provided in an example. Inference typically means finding a low energy configuration... this sentence is confusing for me (not a native English speaker) . Could you give a real example(maybe iris classification? ) to show what part of a machine learning contain energy function or energy minimization . – Mithril Dec 07 '17 at 08:53
  • @Mithril did you read the paper I linked? – Neil G Dec 07 '17 at 10:58
2

In signal detection literature, the energy of a signal $x_t$ is defined as $$ E = \Sigma x_t^2 $$

When predicting some response y from some features x, a very common and simple way to proceed is to minimise the sum of the squared errors $$ SSE= \Sigma (y-\hat{y})^2 $$ where $\hat{y}$ is the fitted response. Notice the similarity? The SSE is energy. This energy is minimised by the fitted parameters.

Michael Heidelberg
  • 103
stan
  • 150
  • 1
    I think you're confusing the loss with the energy – Neil G Feb 07 '16 at 21:59
  • I am using the standard definition of energy from signal processing. Computer science / machine learning people do tend to redefine terms, I guess. I come from stats and signal processing background – stan Feb 07 '16 at 22:18
  • Your first formula is an energy function. The second formula is loss function since it's not a function of the configuration. – Neil G Feb 07 '16 at 22:22
  • @Neil I'm sure you are using the terminology correctly as defined in the papers you cited. It's just a different terminology from what I'm used to where SSE is energy – stan Feb 07 '16 at 22:28