News for hackers. Stuff that matters.
Re:Continuous Bernoulli distribution of Variational autoenco... by sdenton4 on Saturday November 28, 2020 @02:03AM (#25235878)
Looks like this paper gives some background: https://arxiv.org/abs/1907.06845

I've dabbled in VAEs, but haven't read this paper. Anyone want to try an ELI5thYearGradStudent?

Re:Continuous Bernoulli distribution of Variational autoenco... by fxtentacle on Saturday November 28, 2020 @04:14AM (#25236434)
My reading is this:

When you use an VAE, you implicitly assume that your data never ever leaves the area between 0 and 1 and that it'll never ever touch the borders.

But if you have pictures of scanned-in handwritten digits, there's a chance that in reality, the line was blacker than the darkest color that your image file format can store. For example, if you did contrast enhancement after scanning, you likely cut off some extra dark or extra bright pixels.

A regular VAE cannot handle this situation that multiple brightness=0 (black) pixels actually had varying negative brightnesses which were clamped to 0 for technical reasons. A continuous Bernoulli can model 0 really meaning <=0. And 1 meaning >=1.

That's why with a continuous Bernoulli you get more saturated white and blacks than with just a Bernoulli. See the image in 5.1, for example. That's a good thing because it increases contrast, thus making your features better and dependent learning tasks easier.

Re:Continuous Bernoulli distribution of Variational autoenco... by aerodude on Saturday November 28, 2020 @08:08AM (#25237290)
Having a quick read, they're pointing out that the Bernoulli distribution is only supported for values of 0 and 1 (i.e. it's binary), whereas pixel values for a grayscale image are a decimal value in the interval [0, 1]. When you train a VAE, it's pretty standard to use a BCE loss, but this is wrong because the data isn't binary (i.e. it's not a Bernoulli distribution). They define a continuous analogue of a Bernoulli distribution that is supported in [0, 1], and use this as the loss function for training a VAE, which gives them reconstructions that are closer to the input.
Re:Continuous Bernoulli distribution of Variational autoenco... by sdenton4 on Saturday November 28, 2020 @05:07PM (#25240958)
Yeah, agree having skimmed the paper now. It seems kinda meh to me: it's obvious that improving the loss function is a good thing to do, and for any real application (ie, not greyscale mnist) you'd never use this binary value target.