News for hackers. Stuff that matters.
Re:Ideas in statistics that have powered AI by whimsicalism on Wednesday July 07, 2021 @04:43PM (#27765544)
> your MLE estimate will depend on your choice of variables if you don't pick a prior.

If you're doing MLE, then you don't have a prior (or rather, you have a uniform prior over the parameter(s) of interest).

Re:Ideas in statistics that have powered AI by contravariant on Wednesday July 07, 2021 @08:16PM (#27767222)
Yeah I suppose in the context of MLE it makes more sense to talk about your choice of variables. Which does matter unfortunately (which is kind of obvious when you note that you can convert any distribution into any other by chaging its coordinates [*]).

Using a prior gives you an 'out' by picking the Radon-Nikodym derivative w.r.t. that prior, since this definition of probability density is independent of your choice of variables. In most applications of MLE people implicitly use the (improper) uniform prior in which case you end up with the usual density. However this is usually done without justification, which is a bit dangerous.

[*]: For a rather extreme example consider that if X is exponentially distributed with mean 1 then so is -log(1 - e^-X) which you get by using the CDF, flipping the distribution and then using the inverse CDF, this transformation swaps 0 (the mode) with positive infinity.

Re:Ideas in statistics that have powered AI by radford-neal on Wednesday July 07, 2021 @09:26PM (#27767602)
No. Maximum likelihood estimation does not involve a prior over parameters, and the MLE does not depend on how the parameters are represented, nor does it depend on the way the variables are represented.

For example, the maximum likelihood estimate for a variance parameter is just the square of the maximum likelihood estimate for the corresponding standard deviation parameter, as you'd expect. And the maximum likelihood estimate for the parameters of a log-normal distribution for positive data are related in the obvious way to the maximum likelihood estimate for the parameters of a normal distribution for the logs of those data points.

Doing a 1-to-1 continuous transformation of the data causes the probability density (for any given parameter values) to be multiplied by a Jacobian factor, but this factor depends only on the data values, NOT on the probability density (determined by the parameters), so the MLE for the transformed data is the same as for the untransformed data.

Re:Ideas in statistics that have powered AI by whimsicalism on Wednesday July 07, 2021 @09:39PM (#27767677)
Agreed - great reply.