News for hackers. Stuff that matters.
Re:The main trick in machine learning by ced on Tuesday December 10, 2013 @06:10PM (#6884565)
Model averaging can be quite useful when you're averaging over versions of the same model with different hyperparameters, e.g. the number of clusters in a mixture model.

Yeah, but in this case, there's a crucial difference: within the assumptions of a mixture model M, N=1, 2, ... clusters do make an exhaustive partition of the space, whereas if I compute a distribution for models M1 and M2, there is always M3, M4, ... lurking unexpressed and unaccounted for. In other words,

P(N=1|M) + P(N=2|M) + ... = 1

but

P(M1) + P(M2) << 1

Is the number of clusters even a hyperparameter? Wiki says that hyperparameters are parameters of the prior distribution. What do you think?

There are no comments on this comment yet.