News for hackers. Stuff that matters.
Re:Learning Concepts with Energy Functions by nerdponx on Wednesday November 07, 2018 @01:01PM (#18400863)
From the abstract of the article they linked:

Energy-Based Models (EBMs) capture dependencies between variables by associating a scalar energy to each configuration of the variables. Inference consists in clamping the value of observed variables and finding configurations of the remaining variables that minimize the energy. Learning consists in finding an energy function in which observed configurations of the variables are given lower energies than unobserved ones. The EBM approach provides a common theoretical framework for many learning models, including traditional discriminative and generative approaches, as well as graph-transformer networks, conditional random fields, maximum margin Markov networks, and several manifold learning methods.

Probabilistic models must be properly normalized, which sometimes requires evaluating intractable integrals over the space of all possible variable configurations. Since EBMs have no requirement for proper normalization, this problem is naturally circumvented. EBMs can be viewed as a form of non-probabilistic factor graphs, and they provide considerably more flexibility in the design of architectures and training criteria than probabilistic approaches.

Seems like a really interesting unification of the wide variety of techniques out there in statistics and machine learning, analogous to the "everything is a computation graph, as long as it's differentiable" revolution. I like it when this kind of thing has its day. Would be interesting to see how well it works non-robotics problems.

Re:Learning Concepts with Energy Functions by BucketSort on Wednesday November 07, 2018 @02:55PM (#18401845)
See Yann's tutorial on EBM: http://yann.lecun.com/exdb/publis/pdf/lecun-06.pdf

We actually ran into them when doing research in our startup. It is a really powerful perspective.

Re:Learning Concepts with Energy Functions by lucidrains on Wednesday November 07, 2018 @03:12PM (#18401980)
Thank you for this!
Re:Learning Concepts with Energy Functions by eli_gottlieb on Thursday November 08, 2018 @11:53AM (#18407652)
>Probabilistic models must be properly normalized, which sometimes requires evaluating intractable integrals over the space of all possible variable configurations. Since EBMs have no requirement for proper normalization, this problem is naturally circumvented. EBMs can be viewed as a form of non-probabilistic factor graphs, and they provide considerably more flexibility in the design of architectures and training criteria than probabilistic approaches.

Can someone explain to me what the major difference is between energy-based models and variational Bayes approximations, which throw out calculating the normalization constant and switch to maximizing the log joint probability of the data and model?