Let X, Y be two random variables with E[Y 2] < ∞. Show that E[Y |X]is the random variable which minimizes E[(Y − f (X))2|X], and then reduce that E[Y |X]minimizes E[(Y − f (X))2], that is the best estimator for Y given X is E[Y |X]

The problem is asking us to prove that the conditional expectation E[Y|X] is the best estimator for Y given X. This is a fundamental result in the theory of conditional expectation and is often referred to as the orthogonality principle.

Here are the steps to prove this:

1. Define the error term e = Y - f(X). We want to minimize the expected value of the square of this error term, E[e^2|X].

2. Expand the square to get E[e^2|X] = E[(Y - f(X))^2|X] = E[Y^2 - 2Yf(X) + f(X)^2|X].

3. Now, we know that E[Y^2|X] is just a constant (since Y^2 is a random variable and we're taking its expected value), so we can't minimize this term.

4. The term -2E[Yf(X)|X] can be maximized if we choose f(X) = E[Y|X], because E[Yf(X)|X] ≤ E[Y|X]E[f(X)|X] by Jensen's inequality (since the function xy is concave), and the equality holds if and only if f(X) = E[Y|X].

5. The term E[f(X)^2|X] is minimized when f(X) = E[Y|X], because for any random variable Z, E[Z^2] ≥ (E[Z])^2, with equality if and only if Z is a constant almost surely. Here, Z = f(X), so E[f(X)^2|X] ≥ (E[f(X)|X])^2 = (E[Y|X])^2, with equality if and only if f(X) = E[Y|X] almost surely.

6. Therefore, the function f(X) that minimizes E[(Y - f(X))^2|X] is f(X) = E[Y|X].

7. To show that E[Y|X] also minimizes E[(Y - f(X))^2] (without conditioning on X), note that E[(Y - f(X))^2] = E[E[(Y - f(X))^2|X]] by the law of iterated expectations. Since E[Y|X] minimizes E[(Y - f(X))^2|X] for each X, it also minimizes the expectation of this quantity.

So, we have shown that the conditional expectation E[Y|X] is the best estimator for Y given X in the sense that it minimizes the mean squared error E[(Y - f(X))^2].