To show that
E[Y|X]
exists, essentially I did that.
Really, all need from X is its sigmaalgebra. That is, can do the argument conditioning on a sigmaalgrbra instead of a random variable X. For Y, I didn't include the details back to the Radon-Nikodym theorem, but may not even need that E[Y] is finite.
Sure don't need finite "second moments".
Doesn't ask for much, merely that Y be measurable, that is, a random variable. We get to f'get about Riemann integration of probability densities.
Full details, with some of the most elegant, powerful, and precise work in the history of civilization, are in the references I gave, to Rudin and Neveu.
E[ (Y - f(X)^2 ]
for real valued random variables X and Y. For Y it is enough that it has an expectation. For X, it need only be measurable, that is, a random variable.
I referenced the Radon-Nikodym theorem. Assuming that, the definition and basic properties of
E[Y|X]
are immediate.
Let's back up to something simpler: What real number a minimizes
E[ (Y - a)^2 ] = E[ Y^2 - 2aY + a^2 ] = E[Y^2] - E[2aY] + E[a^2] = E[Y^2] - 2aE[Y] + E[a^2] = E[Y^2] + E[Y]^2 - 2aE[Y] - E[Y]^2 + E[a^2] = E[Y^2] + E[Y]^2 - 2aE[Y] - E[Y]^2 + a^2 = E[Y^2] + (E[Y] - a)^2 - E[Y]^2
which is minimized by
a = E[Y]
So, if real random variable
X = a
that is, takes on only one value, then
E[ (Y - a)^2 ]
is minimized by
a = E[Y]
To improve on that case, suppose real random variable X is discrete, takes on finitely many values
a_i
for positive integer n and
i = 1, 2, ..., n
and suppose random variables
/ X = a_i
1_{X = a_i} = |
\ 0 otherwiseThen
X = sum_{i = 1}^n a_i 1_{X = a_i}
Let
b_i = E[Y 1_{X = a_i}]
Then by what we did above for
a = E[Y]
minimizing
E[ (Y - a)^2 ]
we have that
f(X) = sum_{i = 1}^n b_i 1_{X = a_i}
minimizes
E[(Y - f(X))^2]
And we would have
E[Y|X] = f(X)
Basically we are doing classic cross tabulation: We collect data on both X and Y. When X = a_i, we average the corresponding Y values and get b_i for our estimate of Y.
The Radon-Nikodym result, among other things, lets us do the same when X is not discrete, that is, is any random variable. We don't even need for X to have an expectation -- all we want is measurability so that from X we have a sigma algebra.
The Radon-Nikodym result is part of measure theory, with a novel von Neumann proof in Rudin. The connection with probability and conditional expectation is part of graduate probability as in Neveu.
My all-time favorite course in school was the graduate probability course I took from Neveu, etc.
I never wanted to be a college professor, but for some unusual reasons for some years I was. Now I'm happy that I'm no longer a college professor! I don't want to write a lecture in graduate probability here on Hacker News.
But you can see above that I made no use of moments.
This thread started with conditional probability is amazing. Then, easy enough, I mentioned that conditional probability is a special case of conditional expectation which is even more amazing.
In addition, conditional expectation is the core of both Markov processes and martingales. The martingale convergence theorem is astounding,
My real interest now is my Internet startup -- back to it.
In probability theory the Radon-Nikodym is used to justify existence of random variables. But it cannot be used to show that any variable has an expected value (ie has a moment, ie is integrable). The Cauchy distribution is a traditional counter example to such assertion.
> For Y it is enough that it has an expectation
That is false. Just look at your “simple example”. You arrive to E[Y^2]. An expectation is not enough. It should be clear intuitively because MSE is a minimization of the second moment of a certain rv, so you are bound to have at least some constraints for second-order relationships for the variables you’ve used to construct that certain rv.
I think you may need to re-read Rudin and Neveau and refresh your knowledge. (Though I can say nothing about Neveau, I studied using the textbook by Shao.)
E[Y^2]
always exists although it might be infinity.You mentioned moments -- as I recall some writing in statistics uses that work, but I never have.
Your description of the Radon-Nikodym result does not look right. Again, I won't want to write a lecture in graduate probability, but from memory the usual description of that result goes:
Given real valued, integrable function f and sigma algebra s, there exists real value function g measurable with respect to sigma algebra s so that g integrates like f over all sets in s. So, with real valued random variable Y in place of function f, we let
E[Y|s]
be the function g. If s is the sigma algebre generated by real random variable X, then we writeOf course that makes it possible. Note that it doesn’t say that f is simply measurable. By definition being integrable means E[f] < infinity. (And being square-integrable means E[f^2] < infinity, an even stronger condition).
Consider Y = Z/X where Z, X iid ~ N(0,1). Try to calculate E[Y|X], and try to minimize E[ (Y - f(X))^2 ]. Neither will make sense. I am sure both Rudin and Neveu were aware of this trivial result.
Irrelevant.
Again, once again, over again, yet again, for real valued random variable Y,
E[Y^2]
always exists.> By definition being integrable means E[|f|] < infinity.
Wrong.
Correcting you is a waste of my time.
For me that's hardly an intuitive notion. For example, in some econometrics textbooks you can meet the reversed argument: OLS is motivated as a way to estimate conditional expectation function. So one can ask, isn't OLS "just common, traditional, nearly universal least squares with all the powers, advantages, and attributes there unto pertaining" simply because it is a way to estimate conditional expectation? And now you are back to square one.
> you add an assumption not needed with my approach via the Radon-Nikodym result
Existence of a unique minimizer of E[(X-f(Y))^2] is a property of L2. You may carryover many properties to L1 by continuity but it isn't one of them. If you don't assume existence of second moments, it is non-trivial to show that min E[(X-f(Y))^2] makes sense if any. I think that you have to put specific constraints on X and Y, but I don't know which are the least restrictive.