Re:A sane introduction to maximum likelihood estimation and ...
by conjectures on Thursday January 03, 2019 @06:45AM
(#18814429)
The post this is a reply to was correct, and this is not. E.g. a simple counter example is finding the autocorrelation parameter in an AR(1) model for an economic time series. Under your suggested definition of MLE this can't be done, which is simply not the case.
In fact, not approaching the more general case is liable to confuse learners as they may think that independence assumption is somehow baked into MAP/MLE, which it is not.
Re:A sane introduction to maximum likelihood estimation and ...
by nerdponx on Thursday January 03, 2019 @02:35PM
(#18818423)
I never suggested a definition of MLE. You need independence to use the "L = Π P(y | θ)" formulation, full stop.
Re:A sane introduction to maximum likelihood estimation and ...
by conjectures on Thursday January 03, 2019 @05:30PM
(#18819977)
Yes, you do need independence to assume the likelihood factorises. You do not need independence to find a MLE/MAP.
Re:A sane introduction to maximum likelihood estimation and ...
by stilley2 on Thursday January 03, 2019 @02:49PM
(#18818571)
True. But that form is a convenience, not a requirement.
Re:A sane introduction to maximum likelihood estimation and ...
by stilley2 on Thursday January 03, 2019 @12:11PM
(#18816767)
But that form is not required. A quick counter example. I'm trying to estimate a value from N measurements. The measurements experience Gaussian noise with some general covariance matrix K (i.e., they are not independent)
Therefore, y is a sample from N([1, 1, ..., 1]^T u, K).
The MLE is then ([1, 1, ..., 1]K^{-1}[1, 1, ..., 1]^T)^{-1} [1, 1, ..., 1] K^{-1} y. Or in words, multiply y by the inverse covariance matrix, sum the result, and divide by the sum of all the elements in the inverse covariance matrix. As a sanity check, when the measurements _are independent, this reduces to a weighted average, where the observations are weighted by their inverse variances.
Typically we write the likelihood function as
If you didn't have identically-distributed observations, the functional form of P would be different for each observation.And if you didn't have independent observations, then you're basically screwed in the general case. That expression for L is basically the definition of probabilistic independence: a finite set of random variables is mutually independent if and only if their joint probability function is equal to the product of the individual variables' probability functions.
If you have dependence between observations, you lose the ability to write L in that nice form. This is a non-negotiable consequence of basic probability theory.
The only way to do MAP estimation without iid observations is to know the joint distribution of your entire dataset, and be able to maximize that distribution with respect to θ given an arbitrary data set. This is possible but it's not quite the same thing as dumping your data into a GLM.