News for hackers. Stuff that matters.
Re:The Unreasonable Effectiveness of Conditional Probabilities by graycat on Saturday February 25, 2023 @05:41PM (#34941193)
Moments and Hilbert space need have nothing to do with real valued function f of real variable x minimizing

E[ (Y - f(X)^2 ]

for real valued random variables X and Y. For Y it is enough that it has an expectation. For X, it need only be measurable, that is, a random variable.

I referenced the Radon-Nikodym theorem. Assuming that, the definition and basic properties of

E[Y|X]

are immediate.

Let's back up to something simpler: What real number a minimizes

E[ (Y - a)^2 ] = E[ Y^2 - 2aY + a^2 ] = E[Y^2] - E[2aY] + E[a^2] = E[Y^2] - 2aE[Y] + E[a^2] = E[Y^2] + E[Y]^2 - 2aE[Y] - E[Y]^2 + E[a^2] = E[Y^2] + E[Y]^2 - 2aE[Y] - E[Y]^2 + a^2 = E[Y^2] + (E[Y] - a)^2 - E[Y]^2

which is minimized by

a = E[Y]

So, if real random variable

X = a

that is, takes on only one value, then

E[ (Y - a)^2 ]

is minimized by

a = E[Y]

To improve on that case, suppose real random variable X is discrete, takes on finitely many values

a_i

for positive integer n and

i = 1, 2, ..., n

and suppose random variables

               / X = a_i
1_{X = a_i} = | \ 0 otherwise

Then

X = sum_{i = 1}^n a_i 1_{X = a_i}

Let

b_i = E[Y 1_{X = a_i}]

Then by what we did above for

a = E[Y]

minimizing

E[ (Y - a)^2 ]

we have that

f(X) = sum_{i = 1}^n b_i 1_{X = a_i}

minimizes

E[(Y - f(X))^2]

And we would have

E[Y|X] = f(X)

Basically we are doing classic cross tabulation: We collect data on both X and Y. When X = a_i, we average the corresponding Y values and get b_i for our estimate of Y.

The Radon-Nikodym result, among other things, lets us do the same when X is not discrete, that is, is any random variable. We don't even need for X to have an expectation -- all we want is measurability so that from X we have a sigma algebra.

The Radon-Nikodym result is part of measure theory, with a novel von Neumann proof in Rudin. The connection with probability and conditional expectation is part of graduate probability as in Neveu.

My all-time favorite course in school was the graduate probability course I took from Neveu, etc.

I never wanted to be a college professor, but for some unusual reasons for some years I was. Now I'm happy that I'm no longer a college professor! I don't want to write a lecture in graduate probability here on Hacker News.

But you can see above that I made no use of moments.

This thread started with conditional probability is amazing. Then, easy enough, I mentioned that conditional probability is a special case of conditional expectation which is even more amazing.

In addition, conditional expectation is the core of both Markov processes and martingales. The martingale convergence theorem is astounding,

My real interest now is my Internet startup -- back to it.

Re:The Unreasonable Effectiveness of Conditional Probabilities by LudwigNagasena on Saturday February 25, 2023 @07:19PM (#34941836)
I am not sure what you mean by moments, but in probability theory “the n-th moment” means E[X^n]; and you’ve clearly used them.

In probability theory the Radon-Nikodym is used to justify existence of random variables. But it cannot be used to show that any variable has an expected value (ie has a moment, ie is integrable). The Cauchy distribution is a traditional counter example to such assertion.

> For Y it is enough that it has an expectation

That is false. Just look at your “simple example”. You arrive to E[Y^2]. An expectation is not enough. It should be clear intuitively because MSE is a minimization of the second moment of a certain rv, so you are bound to have at least some constraints for second-order relationships for the variables you’ve used to construct that certain rv.

I think you may need to re-read Rudin and Neveau and refresh your knowledge. (Though I can say nothing about Neveau, I studied using the textbook by Shao.)

Re:The Unreasonable Effectiveness of Conditional Probabilities by graycat on Sunday February 26, 2023 @11:34AM (#34946804)
For any real valued random variable,

     E[Y^2]
always exists although it might be infinity.

You mentioned moments -- as I recall some writing in statistics uses that work, but I never have.

Your description of the Radon-Nikodym result does not look right. Again, I won't want to write a lecture in graduate probability, but from memory the usual description of that result goes:

Given real valued, integrable function f and sigma algebra s, there exists real value function g measurable with respect to sigma algebra s so that g integrates like f over all sets in s. So, with real valued random variable Y in place of function f, we let

     E[Y|s]
be the function g. If s is the sigma algebre generated by real random variable X, then we write
Re:The Unreasonable Effectiveness of Conditional Probabilities by LudwigNagasena on Sunday February 26, 2023 @04:03PM (#34949435)
> Given real valued, integrable function f

Of course that makes it possible. Note that it doesn’t say that f is simply measurable. By definition being integrable means E[f] < infinity. (And being square-integrable means E[f^2] < infinity, an even stronger condition).

Consider Y = Z/X where Z, X iid ~ N(0,1). Try to calculate E[Y|X], and try to minimize E[ (Y - f(X))^2 ]. Neither will make sense. I am sure both Rudin and Neveu were aware of this trivial result.

Re:The Unreasonable Effectiveness of Conditional Probabilities by graycat on Wednesday March 01, 2023 @01:03AM (#34979268)
> Of course that makes it possible. Note that it doesn’t say that f is simply measurable.

Irrelevant.

Again, once again, over again, yet again, for real valued random variable Y,

     E[Y^2] 
always exists.

> By definition being integrable means E[|f|] < infinity.

Wrong.

Correcting you is a waste of my time.

Re:The Unreasonable Effectiveness of Conditional Probabilities by LudwigNagasena on Wednesday March 01, 2023 @05:42AM (#34980854)
You are not correcting me, you are correcting Rudin :)