Sheet 1

Exercise 1 (Properties of Conditional Expectation).

Let $X$ and $Y$ be two integrable random variables, $𝒢\subseteq ℱ$ $\sigma$-Algebras.

1. Prove $𝔼[𝔼[X|ℱ]]=𝔼[X]$ and $𝔼[1|ℱ]=1$ a.s.

   Solution.

   Using the second defining property of conditional expectation with the set $\mathrm{\Omega }$ we have

   $𝔼[𝔼[X|ℱ]]=𝔼[{\mathrm{𝟙}}_{\mathrm{\Omega }}𝔼[X|ℱ]]=𝔼[{\mathrm{𝟙}}_{\mathrm{\Omega }}X]=𝔼[X].$

   For the second claim we simply check the definition of conditional expectation for the constant function $1$. Since constants are measureable with regard to any $\sigma$-algebra, the first requirement is a given, the second one is trivial. ∎

2. Prove $𝔼[𝔼[X|ℱ]|𝒢]=𝔼[𝔼[X|𝒢]|ℱ]=𝔼[X|𝒢]$ a.s.

   Solution.

   Since $𝔼[X|𝒢]$ is by definition $𝒢$ measurable and we have $𝒢\subseteq ℱ$, it is also $ℱ$ measurable. We therefore have

   $𝔼[𝔼[X|𝒢]|ℱ]\stackrel{\text{linearity}}{=}𝔼[X|𝒢]𝔼[1|ℱ]=𝔼[X|𝒢].$

   To show that $𝔼[𝔼[X|ℱ]|𝒢]=𝔼[X|𝒢]$, we will show that $𝔼[X|𝒢]$ fulfills the defining requirements. $𝒢$ measurability is obvious, what is left to show is that for any $A\in 𝒢\subseteq ℱ$ we have

   $𝔼[{\mathrm{𝟙}}_A𝔼[X|ℱ]]$ $\stackrel{A\in ℱ}{=}𝔼[{\mathrm{𝟙}}_{A}X]\stackrel{A\in 𝒢}{=}𝔼[{\mathrm{𝟙}}_A𝔼[X|𝒢]].\mathit{∎}$

3. Assume that $X=Y$ almost surely. Prove that, almost surely, $𝔼[X|Y]=Y$ and $𝔼[Y|X]=X$.

   Solution.

   Due to symmetry it is sufficient to prove $𝔼[X|Y]=Y$. To show this we simply check the defining properties of $𝔼[X|Y]$. First, $Y$ is $\sigma (Y)$ measurable by definition, and second, we have for all $A\in \sigma (Y)$

   $𝔼[X{\mathrm{𝟙}}_A]$ $=𝔼[Y{\mathrm{𝟙}}_A].\mathit{∎}$

Exercise 2.

The following questions are independent.

1. Let $X$ and $Y$ be i.i.d. Bernoulli variables: $\mathbb{P}(X=1)=1-\mathbb{P}(X=0)=p$ for some $p\in (0,1)$. We set $Z≔{\mathrm{𝟏}}_{\{X+Y=0\}}$. Compute $𝔼[X|Z]$ and $𝔼[Y|Z]$. Are these random variables independent?

   Solution.

   First, due to $Z={\mathrm{𝟙}}_{\{X+Y=0\}}\in \{0,1\}$ we can write

   $𝔼[X|Z]$ $=𝔼[X|Z=0]\cdot {\mathrm{𝟙}}_{Z=0}+𝔼[X|Z=1]\cdot {\mathrm{𝟙}}_{Z=1}$

   $={\displaystyle \frac{𝔼[X\cdot {\mathrm{𝟙}}_{Z=0}]}{\mathbb{P}(Z=0)}}\cdot {\mathrm{𝟙}}_{Z=0}+\underset{=0}{\underbrace{{\displaystyle \frac{𝔼[X\cdot {\mathrm{𝟙}}_{Z=1}]}{\mathbb{P}(Z=1)}}}}\cdot {\mathrm{𝟙}}_{Z=1}$

   as $X=0$ for $Z=1$. So we only need to think about the first term. First, we have

   $P(Z=0)$ $=1-\mathbb{P}(Z=1)=1-\underset{{(1-p)}^2}{\underbrace{\mathbb{P}(X=0)\mathbb{P}(Z=0)}}$

   $=2p-p^2=p(2-p),$

   and due to $X\cdot {\mathrm{𝟙}}_{Z=0}=X$ we get

   $𝔼[X\cdot {\mathrm{𝟙}}_{Z=0}]=𝔼[X]=𝔼[{\mathrm{𝟙}}_{X=1}]=p.$

   Collecting all three results together, results in

   $𝔼[X|Z]$ $={\displaystyle \frac{1}{2-p}}\cdot {\mathrm{𝟙}}_{Z=0}.$

   Due to symmetry we get the same result for $𝔼[Y|Z]$ which implies $𝔼[X|Z]=𝔼[Y|Z]$ a.s. In particular they are not independent! ∎

2. Let $X$ be a square-integrable random variable and $𝒢\subset ℱ$ a sub-$\sigma$-algebra. We define the conditional variance

   $$\text{Var}(X|𝒢)≔𝔼[{(X-𝔼[X|𝒢])}^2|𝒢].$$

   Prove the following identity:

   $$\text{Var}(X)=𝔼[\text{Var}(X|𝒢)]+\text{Var}(𝔼[X|𝒢]).$$

   Solution.

   Recall that we have $\text{Var}(X)=𝔼[X^2]-𝔼{[X]}^2$, the same holds true for the conditional variance:

   $\text{Var}(X|𝒢)$ $=𝔼[X^2-2X𝔼[X|𝒢]+𝔼{[X|𝒢]}^2|𝒢]$

   $=𝔼[X^2|𝒢]-2𝔼[X𝔼[X|𝒢]|𝒢]+𝔼[𝔼{[X|𝒢]}^2|𝒢]$

   $=𝔼[X^2|𝒢]-𝔼{[X|𝒢]}^2.$ (1)

   With those identities we can directly prove our claim by calculation

   $𝔼\left[\text{Var}(X|𝒢)\right]+\text{Var}\left(𝔼[X|𝒢]\right)$ $=𝔼\left[\text{Var}(X|𝒢)\right]+𝔼\left[𝔼{[X|𝒢]}^2\right]-𝔼{\left[𝔼[X|𝒢]\right]}^2$

   $=𝔼\left[\text{Var}(X|𝒢)+𝔼{[X|𝒢]}^2\right]-𝔼{\left[𝔼[X|𝒢]\right]}^2$

   $\stackrel{(\mathit{\text{<a href="\#S0.E1" title="(1) ‣ item (ii) ‣ Exercise 2. ‣ Sheet 1" class="ltx\_ref ltx\_font\_italic" style="font-size:70\%;"><span class="ltx\_text ltx\_ref\_tag">1</span></a>}})}{=}𝔼\left[𝔼[X^2|𝒢]\right]-𝔼{\left[𝔼[X|𝒢]\right]}^2$

   $=𝔼[X^2]-𝔼{[X]}^2$

   $=\text{Var}(X)\mathit{∎}$

Exercise 3 (Factorization lemma).

(Important!) Let $X,Y:(\mathrm{\Omega },𝒜)\to (ℝ,ℬ(ℝ))$ be random variables. Show that, if $X$ is $(\sigma (Y),ℬ(ℝ))$-measurable, then there exists a measurable function $g:(ℝ,ℬ(ℝ))\to (ℝ,ℬ(ℝ))$, such that $X=g(Y)$.

Solution.

First we consider simple functions, i.e.

$X={\displaystyle \sum _{k=1}^n}{a}_k{\mathrm{𝟙}}_{A_k}$

for $a_k\ge 0$ and $A_k\in 𝒜$ for all $k=1,\mathrm{\dots },n$. Without loss of generality we can assume the $A_k$ to be disjoint and $a_k\ne a_n$ for $k\ne n$. This, together with the fact that $X$ is supposed to be $\sigma (Y)$ measurable, implies there exists some $B_k\in ℬ(ℝ)$ s.t. $A_k\in Y^{-1}(B_k)$ for all $k$, because we have

$A_k\stackrel{\text{disjoint}}{=}{X}^{-1}(\{a_k\})\in \sigma (Y)=\{Y^{-1}(B):B\in ℬ(ℝ)\}.$

Hence we have

$X={\displaystyle \sum _{k=1}^n}{a}_k\underset{{\mathrm{𝟙}}_{A_k}}{\underbrace{({\mathrm{𝟙}}_{B_k}\circ Y)}}=\left(\underset{≔g}{\underbrace{{\displaystyle \sum _{k=1}^n}{a}_k\cdot {\mathrm{𝟙}}_{B_k}}}\right)\circ Y,$

where $g$ is measurable.

Now we just assume $X\ge 0$ is $(\sigma (Y),ℬ(ℝ))$-measurable, which implies there exists a sequence ${(X_n)}_{n\in \mathbb{N}}$ of $(\sigma (Y),ℬ(ℝ))$-measurable simple functions with $X_n\uparrow X$. Taking intersections of the indicator sets of $X_{n+1}$ and $X_n$ implies that $\mathrm{\Delta }{X}_n=X_n-X_{n-1}$ is a simple function too, which can be written as $\mathrm{\Delta }{X}_n=g_n(Y)$. Since $X_n$ converges we have

$X=\underset{n\to \mathrm{\infty }}{lim}{X}_n=\underset{n\to \mathrm{\infty }}{lim}{\displaystyle \sum _{k=1}^n}\mathrm{\Delta }{X}_n={\displaystyle \sum _{k=1}^{\mathrm{\infty }}}{g}_k(Y)=\underset{=:g}{\underbrace{{\displaystyle \sum _{k=1}^{\mathrm{\infty }}}{g}_k}}\circ Y$

using $X_0=0$ without loss of generality. And $g$ is measurable as a limit of measurable functions. Note that $g$ is well defined as ${\sum }_{k=1}^n{g}_k$ with $g_k\ge 0$ is monotonously increasing although it might be infinite. This is why we used the telescoping trick. Splitting $X$ into $X=X^{+}-X^{-}$ yields the general case. ∎

Exercise 4 (Best Estimators).

Let $X$ be a real random variable.

1. Find the estimator $m$ minimizing $𝔼[{\mathrm{𝟙}}_{X\ne m}]$ when $X$ is discrete. What is the best estimator for a dice roll using this loss?

   Solution.

   The best estimator is the maximum likelihood estimator ${\max }_{m}P(X=m)$, as

   $\underset{m}{\min }𝔼[{\mathrm{𝟙}}_{X\ne m}]=\underset{m}{\min }\mathbb{P}(X\ne m)=\underset{m}{\min }[1-P(X=m)]=1-\underset{m}{\max }P(X=m).$

   So for a fair dice, any of its faces $\{1,\mathrm{\dots },6\}$ is a minimizer. ∎

We define the median of $X$ to be $𝕄[X]:=inf\{m\in ℝ:f(m)>0\}$, where

$f:\{\begin{array}{cc}ℝ\to [-1,1]\hfill & \\ m\mapsto \mathbb{P}(X\le m)-\mathbb{P}(X>m)\hfill & \end{array}$

1. (Unimportant) Prove that $\mathbb{P}(X\le 𝕄[X])\ge \frac{1}{2}$ and $\mathbb{P}(X\ge 𝕄[X])\ge \frac{1}{2}$, and show that for all $m>𝕄[X]$ we do not have $\mathbb{P}(X\le m)\ge \frac{1}{2}$.

   Hint.

   Calculate the limits ${\lim }_{m\mathrm{\uparrow }𝕄\mathit{}\mathrm{[}X\mathrm{]}}\mathit{}f\mathit{}\mathrm{(}m\mathrm{)}$ and ${\lim }_{m\mathrm{\downarrow }𝕄\mathit{}\mathrm{[}X\mathrm{]}}\mathit{}f\mathit{}\mathrm{(}m\mathrm{)}$ using the continuity of measures. Also note that $f\mathit{}\mathrm{(}m\mathrm{)}\mathrm{=}\mathrm{2}\mathit{}{F}_X\mathit{}\mathrm{(}m\mathrm{)}\mathrm{-}\mathrm{1}$, where $F_X$ is the cumulative distribution function of $X$.

   Solution.

   As $f$ is right-continuous, since $F_X(m)=\mathbb{P}(X\le m)$ is right-continuous, we have

   $2\mathbb{P}(X\le 𝕄[X])-1=f(𝕄[X])=\underset{m\downarrow 𝕄[X]}{lim}f(m)\ge 0$

   and therefore $\mathbb{P}(X\le 𝕄[X])\ge \frac{1}{2}$. For the second claim we need to use continuity of measures directly

   $=\underset{m\uparrow 𝕄[X]}{lim}\mathbb{P}(X\le m)-\mathbb{P}(X>m)$

   $=\underset{m\uparrow 𝕄[X]}{lim}f(m)\le 0$

   Therefore we have , which implies that $\mathbb{P}(X\ge 𝕄[X])\ge \frac{1}{2}$.

   Lastly, for any $m>𝕄[X]$, we can find a small $ϵ>0$ with $m-ϵ>𝕄[X]$. Therefore

   which implies and therefore . ∎

   Remark.

   This property is usually the defining property of the median. If this property does not define the median uniquely, $𝕄\mathit{}\mathrm{[}X\mathrm{]}$ is the largest median. A lower bound can be found similarly.

2. Prove that the median minimizes the $L^1$ error, i.e. $𝕄[X]=\arg {\min }_m𝔼[|X-m|]$. Whenever useful, you can assume continuity.

   Hint.

   Recall that for $X\mathrm{\ge }\mathrm{0}$ a.s., we have $𝔼\mathrm{[}X\mathrm{]}\mathrm{=}{\mathrm{\int }}_{\mathrm{0}}^{\mathrm{\infty }}\mathbb{P}\mathrm{(}X\mathrm{\ge }t\mathrm{)}dt\mathrm{=}{\mathrm{\int }}_{\mathrm{0}}^{\mathrm{\infty }}\mathbb{P}\mathrm{(}X\mathrm{>}t\mathrm{)}dt$. Use this fact to prove

   $𝔼[{(X-m)}_{+}]={\displaystyle {\int }_m^{\mathrm{\infty }}}\mathbb{P}(X>m)dt.$

   Solution.

   Recall that for $X\ge 0$ a.s., we have $𝔼[X]={\int }_0^{\mathrm{\infty }}\mathbb{P}(X\ge t)dt={\int }_0^{\mathrm{\infty }}\mathbb{P}(X>t)dt$. Applying this fact to ${(X-m)}_{+}$ we get

   $𝔼[{(X-m)}_{+}]$

   $={\displaystyle {\int }_m^{\mathrm{\infty }}}\mathbb{P}(X>s)ds$

   And similarly

   $𝔼[{(X-m)}_{-}]$

   $={\displaystyle {\int }_{-\mathrm{\infty }}^m}\mathbb{P}(s\ge X)ds$

   Put together we have

   $𝔼[|X-m|]$ $={\displaystyle {\int }_m^{\mathrm{\infty }}}\mathbb{P}(X>s)ds+{\displaystyle {\int }_{-\mathrm{\infty }}^m}\mathbb{P}(X\le s)ds.$

   Assuming $\mathbb{P}(X>s)$ and $\mathbb{P}(X\le x)$ are both continuous in $s$, we can use the fundamental theorem of calculus to obtain

   ${\displaystyle \frac{d}{dm}}𝔼[|X-m|]=\mathbb{P}(X\le m)-\mathbb{P}(X>m)=f(m).$

   Therefore we have

   $𝔼[|X-m|]-𝔼[|X-𝕄[X]|]={\displaystyle {\int }_{𝕄[X]}^m}f(t)𝑑t\ge 0$

   as $f$ is greater than zero for any $m\ge 𝕄[X]$ and less than zero for any $m\le 𝕄[X]$ and sorting the integral borders results in another sign flip. Notice that $f$ is strictly greater than zero for $m\ge 𝕄[X]$ which implies that $𝕄[X]$ is the largest median, i.e. all numbers greater than $𝕄[X]$ do not minimize $𝔼[|X-m|]$.

   As mentioned one can similarly define a lower bound, and all numbers in-between fulfill the defining property of medians of $\mathbb{P}(X\ge m)\ge \frac{1}{2}$ and $\mathbb{P}(X\le m)\ge \frac{1}{2}$. ∎

   Remark.

   In the non-continuous case, one still has right-continuity of $f$ and can prove right-differentiability of $𝔼\mathit{}\mathrm{[}\mathrm{|}X\mathrm{-}m\mathrm{|}\mathrm{]}$. This might be sufficient to prove it is monotonously increasing after $𝕄\mathit{}\mathrm{[}X\mathrm{]}$ and similarly monotonously decreasing before. But the proof is likely going to be complex or require a strong analysis foundation. Alternatively, a short and basic (but unintuitive) proof of the general case can be found here: https://math.stackexchange.com/a/2790390/445105.