Sheet 1

Exercise 1 (Convex Examples).

Prove the following functions are convex

1. affine linear functions, i.e. $f(x)=a^{T}x+c$ for $a\in {ℝ}^d$, $c\in ℝ$,

   Solution.

   We have

   $f(\lambda x+(1-\lambda )y)$ $=a^T(\lambda x+(1-\lambda )y)+\stackrel{1}{\overbrace{(\lambda +1-\lambda )}}c$

   = λ ⏟ (a^Tx+c) _f(x) +(1-λ) ⏟ (a^Ty+c) _f(y). ∎

2. norms, i.e. $x\mapsto \parallel x\parallel$,

   Solution.

   We have

   $$\parallel \lambda x+(1-\lambda )y\parallel \stackrel{\Delta }{\le }\parallel \lambda x\parallel +\parallel (1-\lambda )y\parallel \stackrel{\text{scaling}}{=}\lambda \parallel x\parallel +(1-\lambda )\parallel y\parallel .\mathit{∎}$$

3. sums of convex functions $f_k$, i.e. $f(x)={\sum }_{k=1}^n{f}_k(x)$,

   Solution.

   We have

   $$f(\lambda x+(1-\lambda y))=\sum _{k=1}^n\underset{\le \lambda f_k(x)+(1-\lambda )f_k(y)}{\underbrace{f_k(\lambda x+(1-\lambda )y)}}\le \lambda \underset{=f(x)}{\underbrace{\sum _{k=1}^n{f}_k(x)}}+(1-\lambda )\underset{=f(y)}{\underbrace{\sum _{k=1}^n{f}_k(y)}}.\mathit{∎}$$

4. $F(x):={sup}_{f\in ℱ}f(x)$ for a set of convex functions $ℱ$.

   Solution.

   We have

   $F(\lambda x+(1-\lambda )y)$ $\le \underset{f\in ℱ}{sup}\lambda f(x)+(1-\lambda )f(y)\le \underset{f\in ℱ}{sup}\lambda f(x)+\underset{g\in ℱ}{sup}(1-\lambda )g(y)$

   $=\lambda F(x)+(1-\lambda )F(y).\mathit{∎}$

   Exercise 2 (Finite Jensen).

   Let $\phi$ be convex, and ${\mathrm{\sum }}_{i\mathrm{=}\mathrm{1}}^n{\lambda }_i\mathrm{=}\mathrm{1}$ for ${\lambda }_i\mathrm{\ge }\mathrm{0}$. Prove

   $$\phi \left(\sum _{i=1}^n{\lambda }_i{x}_i\right)\le \sum _{i=1}^n{\lambda }_i\phi (x_i)$$

   and deduce ${\mathrm{(}\frac{\mathrm{1}}{n}\mathit{}{\mathrm{\sum }}_{i\mathrm{=}\mathrm{1}}^n{x}_i\mathrm{)}}^{\mathrm{2}}\mathrm{\le }\frac{\mathrm{1}}{n}\mathit{}{\mathrm{\sum }}_{i\mathrm{=}\mathrm{1}}^n{x}_i^{\mathrm{2}}$.

   Solution.

   For $n=2$ this is simply the definition of convexity. Assume it holds for $n$. Then the induction step is given by

   $\phi \left({\displaystyle \sum _{i=1}^{n+1}}{\lambda }_i{x}_i\right)$ $=\phi \left({\lambda }_{n+1}{x}_{n+1}+(1-{\lambda }_{n+1}){\displaystyle \sum _{i=1}^n}\underset{=:{\tilde{\lambda }}_i}{\underbrace{{\displaystyle \frac{{\lambda }_i}{1-{\lambda }_{n+1}}}}}{x}_i\right)$

   $\le {\lambda }_{n+1}\phi (x_{n+1})+(1-{\lambda }_{n+1}).\underset{\stackrel{\text{ind.}}{\le }{\sum }_{i=1}^n{\tilde{\lambda }}_i\phi (x_i)}{\underbrace{\phi \left({\displaystyle \sum _{i=1}^n}{\tilde{\lambda }}_i{x}_i\right)}}$

   $\le {\displaystyle \sum _{i=1}^{n+1}}{\lambda }_i\phi (x_i)$

   Where it is easy to check that ${\tilde{\lambda }}_i$ sums to one. For the second part select ${\lambda }_i=\frac{1}{n}$ and $\phi (x)=x^2$, which is convex because

   ${(\lambda x+(1-\lambda )y)}^2$ $=\underset{=\lambda (1-(1-\lambda ))}{\underbrace{{\lambda }^2}}{x}^2+\underset{=(1-\lambda )-\lambda (1-\lambda )}{\underbrace{{(1-\lambda )}^2}}{y}^2+2\lambda (1-\lambda )xy$

   $=\lambda x^2+(1-\lambda )y^2-\lambda (1-\lambda )\underset{={(x-y)}^2\ge 0}{\underbrace{(x^2+y^2-2xy)}}$

   $\le \lambda x^2+(1-\lambda )y^2.\mathit{∎}$

   Exercise 3 (Strict & Strong Convexity).

   Prove the following statements

   1. $\mu$-strong convexity implies strict convexity.

      Solution.

      Assuming $x\ne y$ we have

      But after moving the negative parts to the left we are left with a condition which is equivalent to strict convexity by Proposition A.1.8 in the lecture. ∎

   2. For twice differentiable $f$, the following are equivalent

      1. ${\nabla }^{2}f(x)⪰\mu 𝕀$

      2. $z^T{\nabla }^{2}f(x)z\ge \mu {\parallel z\parallel }^2$

      3. $f$ is $\mu$-strongly convex

      where $𝕀$ is the identity matrix and

      $$A⪰B:\iff A-B\mathit{\text{ is (weakly) positive definite}}.$$

      Solution.

      Let us get (iv)((b))(I)$\iff$(iv)((b))(II) out of the way first:

      $$z^T{\nabla }^{2}f(x)z-\mu {\parallel z\parallel }^2=⟨z,{\nabla }^{2}f(x)z⟩-⟨z,\mu 𝕀z⟩=z^T({\nabla }^{2}f(x)-\mu 𝕀)z$$

      If we have (iv)((b))(I), then the rightmost side is positive and thus we also have (iv)((b))(II). For the other direction we simply start from the left.

      For (iv)((b))(I)$\Rightarrow$(iv)((b))(III) let

      $$g(y):=f(y)-\frac{\mu }{2}{\parallel y\parallel }^2,$$

      because $f$ is $\mu$ strongly convex

      $g(y)$ $=f(y)-\frac{\mu }{2}{\parallel y\parallel }^2$

      ≥f(x) + ⟨∇f(x), y-x⟩+ μ 2 ∥y-x∥^2 - μ 2 ∥y∥^2

      $=\underset{=g(x)}{\underbrace{f(x)-\frac{\mu }{2}{\parallel x\parallel }^2}}+\frac{\mu }{2}{\parallel x\parallel }^2+⟨\underset{\nabla g(x)}{\underbrace{\nabla f(x)-\mu x}},y-x⟩+\mu ⟨x,y-x⟩-\mu ⟨x,y⟩+\frac{\mu }{2}{\parallel x\parallel }^2$

      $=g(x)+⟨\nabla g(x),y-x⟩$

      $g$ is still convex. So we know by the lecture that

      $${\nabla }^{2}g(x)={\nabla }^{2}f(x)-\mu 𝕀$$

      is positive definite.

      On the other hand for (iv)((b))(III)$\Rightarrow$(iv)((b))(I): if ${\nabla }^{2}f⪰\mu 𝕀$, we know that ${\nabla }^{2}g$ is positive definite and thus that $g$ is convex by the lecture. Reusing the equations above except for the inequality we obtain

      $f(x)+⟨\nabla f(x),y-x⟩+\frac{\mu }{2}{\parallel y-x\parallel }^2-\frac{\mu }{2}{\parallel y\parallel }^2$ $=g(x)+⟨\nabla g(x),y-x⟩$

      $=f(y)-\frac{\mu }{2}{\parallel y\parallel }^2$

      Adding $\frac{\mu }{2}{\parallel y\parallel }^2$ on both sides results in the definition of $\mu$-strong convexity of $f$. ∎

      Exercise 4 (Convexity and Minima).

      Prove the following statements

      1. If $f$ is convex, then every local minimum is also a global minimum.

         Solution.

         Let $x^{*}$ be a local minimum and assume it was not global. Then there exists $y$ such that

         Select $\lambda$ small enough such that $x^{*}+\lambda (y-x^{*})$ is still in the $ϵ$ neighborhood of $x^{*}$ where $x^{*}$ is a local minimum. Then obtain a contradiction

      2. If $f$ is strictly convex, then there exists at most one minimum.

         Solution.

         Let $x\ne y$ both be minima (which are global by (iv)((b))(I)). Then for some $\lambda \in (0,1)$

         which is a contradiction. ∎

      3. If $f$ convex and differentiable and $\nabla f(x^{*})=0$, then $x^{*}$ is a minimum.

         Solution.

         By Proposition A.1.8 convexity is equivalent to

         $$f(x)\ge f(x^{*})+⟨\underset{=0}{\underbrace{\nabla f(x^{*})}},x-x^{*}⟩=f(x^{*})\mathit{\hspace{1em}}\forall x.$$

         So $x^{*}$ is a (global) minimum. ∎

      Exercise 5 (Directional Minima).

      Let $f$ be some differentiable function. For every direction $d\mathrm{\in }{ℝ}^n$ define

      $$g_d(\alpha ):=f(x^{*}+\alpha d).$$

      Assume that for every $d$, $g_d$ is minimized by $\alpha \mathrm{=}\mathrm{0}$. Prove that

      1. We have necessarily $\nabla f(x^{*})=0$.

         Solution.

         If we select a standard basis vector as the direction, then

         $$0=g_{e_i}^{\prime }(\alpha ){|}_{\alpha =0}=⟨\nabla f(x^{*}+\alpha e_i),e_i⟩{|}_{\alpha =0}=⟨\nabla f(x^{*}),e_i⟩$$

         So the $i$-th entry of $\nabla f(x^{*})$ is zero. As $i$ was arbitrary, we are done. ∎

      2. $f(x^{*})$ is not necessarily a minimum of $f$.

         Hint.

         Let and define

         $$f(y,z):=(z-p{y}^2)(z-q{y}^2)$$

         consider $x^{\mathrm{*}}\mathrm{=}\mathrm{(}\mathrm{0}\mathrm{,}\mathrm{0}\mathrm{)}$ and prove that for .

         Solution.

         Let and define

         $$f(y,z):=(z-p{y}^2)(z-q{y}^2)$$

         then for we have for $y\ne 0$

         This means that $f(0,0)=0$ is not a minimum since we can get arbitrarily close to $x^{*}=(0,0)$ with $(y,m{y}^2)$ with smaller and smaller $y$.

         Let $d=(d_1,d_2)$ be an arbitrary direction. Then

         $g_d(\alpha )$ $=(\alpha d_2-p{(\alpha d_1)}^2)(\alpha d_2-q{(\alpha d_1)}^2)$

         $={\alpha }^2(d_2-p\alpha d_1^2)(d_2-q\alpha d_1^2)$

         $={\alpha }^2(d_2^2-\alpha d_2(p+q)d_1^2+{\alpha }^{2}pq{d}_1^2)$

         Thus $g^{\prime }(0)=0$ and $g^{\prime \prime }(0)=2d_2^2>0$ which implies $g$ has a minimum in zero. ∎

      Exercise 6 (Bregman Divergence).

      The Bregman Divergence $D_f^{\mathrm{(}B\mathrm{)}}$ of a continuously differentiable function $f\mathrm{:}{ℝ}^d\mathrm{\to }ℝ$ is defined as the error of the linear approximation and is related to $\mu$-strong convexity and Lipschitz continuous gradients as follows

      $$\frac{\mu }{2}{\parallel x-x_0\parallel }^2\stackrel{\begin{array}{c}\hfill \mu \mathit{\text{-strongly convex}}\hfill \\ \hfill \mathit{\text{(definition)}}\hfill \end{array}}{\le }\underset{=:D_f^{\left(B\right)}(x,x_0)}{\underbrace{f(x)-\stackrel{\mathit{\text{linear approximation}}}{\overbrace{f(x_0)-⟨\nabla f(x_0),x-x_0⟩}}}}\stackrel{\begin{array}{c}\hfill L\mathit{\text{-Lipschitz gradient}}\hfill \\ \hfill \mathit{\text{(Descent Lemma)}}\hfill \end{array}}{\le }\frac{L}{2}{\parallel x-x_0\parallel }^2.$$

      For $\mu \mathrm{=}\mathrm{0}$ this is simply the convexity condition by Prop. A.1.8. So non-negativity of the Bregman divergence implies convexity. The $L$-Lipschitz gradients provide us with an upper bound on the Bregman divergence on the other hand which immediately results in an upper bound on $f$

      $$f(x)=f(x_0)+⟨\nabla f(x_0),x-x_0⟩+\underset{\le \frac{L}{2}{\parallel x-x_0\parallel }^2}{\underbrace{D_f^{(B)}(x,x_0)}}.$$ (1)

      1. Prove for functions $f$ with $L$-Lipschitz gradients, we have for all $x_0$

         $$\underset{x}{\min }f(x)\le f(x_0)-\frac{1}{2L}{\parallel \nabla f(x_0)\parallel }^2.$$

         By minimizing the upper bound (1). What is the minimizer of the upper bound?

         Hint.

         First minimize over the direction $x\mathrm{-}{x}_{\mathrm{0}}$ subject to the length $\mathrm{\parallel }x\mathrm{-}{x}_{\mathrm{0}}\mathrm{\parallel }\mathrm{=}r$ being constant. Then minimize over $r$.

         Solution.

         We first solve the directional minimization problem

         $\underset{x:\parallel x-x_0\parallel =r}{argmin}f(x_0)+⟨\nabla f(x_0),x-x_0⟩+\frac{L}{2}{\parallel x-x_0\parallel }^2$ $=x_0+\underset{d:\parallel d\parallel =r}{argmin}⟨\nabla f(x_0),d⟩$

         $=x_0+\underset{d:\parallel d\parallel =r}{argmax}⟨-\nabla f(x_0),d⟩$

         $=x_0-{\displaystyle \frac{r\nabla f(x_0)}{\parallel \nabla f(x_0)\parallel }},$ (2)

         where the last equation is true because by Cauchy-Schwartz

         $$⟨-\nabla f(x_0),d⟩\stackrel{C.S.}{\le }\parallel \nabla f(x_0)\parallel r.$$

         So in summary we have

         $\underset{x}{\min }f(x)$ $\le \underset{r}{\min }\underset{\stackrel{(\mathit{\text{<a href="\#S0.E2" title="(2) ‣ item (iv)((b))(I) ‣ Exercise 6 (Bregman Divergence). ‣ item (iv)(b) ‣ Exercise 3 (Strict \& Strong Convexity). ‣ item (iv) ‣ Exercise 1 (Convex Examples). ‣ Sheet 1" class="ltx\_ref ltx\_font\_italic" style="font-size:70\%;"><span class="ltx\_text ltx\_ref\_tag">2</span></a>}})}{=}f(x_0)-r\parallel \nabla f(x_0)\parallel +\frac{L}{2}{r}^2}{\underbrace{\underset{x:\parallel x-x_0\parallel =r}{\min }f(x_0)+⟨\nabla f(x_0),x-x_0⟩+\frac{L}{2}{\parallel x-x_0\parallel }^2}}.$

         Minimizing over the length $r$ implies minimizing a convex parabola, so the first order condition is sufficient yielding

         $$r^{*}=\frac{\parallel \nabla f(x_0)\parallel }{L}.$$

         Reinserting $r^{*}$ into our upper bound yields the claim and we get the minimizer by inserting $r^{*}$ into (2) resulting in a gradient descent step

         $$x^{*}=x_0-\frac{1}{L}\nabla f(x_0).\mathit{∎}$$

      2. Prove for convex functions $f$ with $L$-Lipschitz gradients

         $$D_f^{(B)}(x,x_0)\ge \frac{1}{2L}{\parallel \nabla f(x)-\nabla f(x_0)\parallel }^2.$$

         Hint.

         Apply (iv)((b))(I) to

         $$\varphi (x):=D_f^{(B)}(x,x_0).$$

         Due to convexity you should already know the global minimum of $\varphi$.

         Solution.

         To apply (iv)((b))(I), we want to prove

         $$\varphi (x):=D_f^{(B)}(x,x_0)=f(x)-f(x_0)-⟨\nabla f(x_0),x-x_0⟩$$

         has $L$-Lipschitz gradient. But the gradient of $\varphi$ is given by

         $$\nabla \varphi (x)=\nabla f(x)-\nabla f(x_0),$$

         which is merely a shift of the gradient of $f$ and therefore still $L$-Lipschitz. Due to convexity we know the Bregman Divergence is greater than zero. So we have

         $$0=D_f^{(B)}(x_0,x_0)=\underset{y}{\min }\varphi (y)\stackrel{\text{<a href="\#S0.I1.i4.I1.i2.I4.i1" title="item (iv)((b))(I) ‣ Exercise 6 (Bregman Divergence). ‣ item (iv)(b) ‣ Exercise 3 (Strict \& Strong Convexity). ‣ item (iv) ‣ Exercise 1 (Convex Examples). ‣ Sheet 1" class="ltx\_ref"><span class="ltx\_text ltx\_ref\_tag">(iv)((b))(I)</span></a>}}{\le }\varphi (x)-\frac{1}{2L}{\parallel \nabla \varphi (x)\parallel }^2.$$

         Reordering and inserting the definition we obtain our claim

         $$\frac{1}{2L}{\parallel \nabla f(x)-\nabla f(x_0)\parallel }^2=\frac{1}{2L}{\parallel \nabla \varphi (x_0)\parallel }^2\le \varphi (x)=D_f^{(B)}(x,x_0).\mathit{∎}$$

      3. Prove for convex functions $f$ with $L$-Lipschitz gradients, we have for all $x,y$

         $$⟨\nabla f(x)-\nabla f(y),x-y⟩\ge \frac{1}{L}{\parallel \nabla f(x)-\nabla f(y)\parallel }^2.$$

         Hint.

         Use (iv)((b))(II) twice.

         Solution.

         We simply apply (iv)((b))(II) twice:

         $\frac{1}{L}{\parallel \nabla f(x)-\nabla f(y)\parallel }^2$≤D^(B)_f(y,x) + D^(B)_f(x,y)

         $=\bcancel{f(y)}-\cancel{f(x)}-⟨\nabla f(x),y-x⟩+\cancel{f(x)}-\bcancel{f(y)}-⟨\nabla f(y),x-y⟩$

         $=⟨\nabla f(x)-\nabla f(y),x-y⟩.\mathit{∎}$

      4. Prove for convex $f$ that the upper bound

         $$D_f^{(B)}(x,x_0)\le \frac{L}{2}{\parallel x-x_0\parallel }^2$$

         is sufficient for Lipschitz continuity of the gradient $\nabla f$.

         Hint.

         Argue why you can use (iv)((b))(III) without circular reasoning.

         Solution.

         We deduced this upper bound in (iv)((b))(I) and then never used the Lipschitz property again. In (iv)((b))(II) we only needed it to apply (iv)((b))(I) and in (iv)((b))(III) we only needed the property to apply (iv)((b))(II). So our premise is sufficient for (iv)((b))(III) without Lipschitz continuity of the gradient. So by (iv)((b))(III) and Cauchy-Schwarz we get

         $$\frac{1}{L}{\parallel \nabla f(x)-\nabla f(y)\parallel }^2\le ⟨\nabla f(x)-\nabla f(y),x-y⟩\stackrel{\text{C.S.}}{\le }\parallel \nabla f(x)-\nabla f(y)\parallel \parallel x-y\parallel .$$

         Multiplying this inequality by $L/\parallel \nabla f(x)-\nabla f(y)\parallel$ results in the claim. ∎

      5. In this last part we want to assume $f$ is $\mu$-strongly convex (and its gradient $L$-Lipschitz). Prove for all $x,y$

         $$⟨\nabla f(x)-\nabla f(y),x-y⟩\ge \frac{\mu }{L+\mu }{\parallel x-y\parallel }^2+\frac{1}{L+\mu }{\parallel \nabla f(x)-\nabla f(y)\parallel }^2.$$

         Hint.

         Note that strong convexity provides an alternative lower bound to (iv)((b))(II). Using this alternative lower bound we could directly obtain a modified version of (iv)((b))(III). But we are greedy. We want to use both lower bounds for an even tighter bound on the scalar product. We therefore want to use

         $$g_x(y):=f(y)-\frac{\mu }{2}{\parallel x-y\parallel }^2$$

         to break down the Bregman divergence of $f$, so we can have our cake and eat it:

         $$D_f^{(B)}(y,z)=\underset{\mathit{\text{use }}\mathit{\text{<a href="\#S0.I1.i4.I1.i2.I4.i2" title="item (iv)((b))(II) ‣ Exercise 6 (Bregman Divergence). ‣ item (iv)(b) ‣ Exercise 3 (Strict \& Strong Convexity). ‣ item (iv) ‣ Exercise 1 (Convex Examples). ‣ Sheet 1" class="ltx\_ref ltx\_font\_italic" style="font-size:70\%;"><span class="ltx\_text ltx\_ref\_tag">(iv)((b))(II)</span></a>}}}{\underbrace{D_{g_x}^{(B)}(y,z)}}+\underset{\mathit{\text{strong convexity}}}{\underbrace{\frac{\mu }{2}{\parallel y-z\parallel }^2}}.$$

         Remember to check convexity of $g_x$ and Lipschitz continuity of $\mathrm{\nabla }\mathit{}{g}_x$ before applying (iv)((b))(II). When applying (iv)((b))(II), the selection $z\mathrm{=}x$ might be helpful.

         Solution.

         We define

         $$g_x(y):=f(y)-\frac{\mu }{2}{\parallel x-y\parallel }^2.$$

         for which we have

         $$\nabla g_x(y)=\nabla f(y)-\mu (y-x).$$

         Therefore we have

         $D_{g_x}^{(B)}(y,z)$ $=g_x(y)-g_x(z)-⟨\nabla g_x(z),y-z⟩$

         $=f(y)-\frac{\mu }{2}{\parallel x-y\parallel }^2-f(z)+\frac{\mu }{2}{\parallel x-z\parallel }^2-⟨\nabla f(z)-\mu (z-x),y-z⟩$

         $=D_f^{(B)}(y,z)+\frac{\mu }{2}\underset{=-\left(-{\parallel x-z\parallel }^2+{\parallel x-z+z-y\parallel }^2-2⟨x-z,z-y⟩\right)}{\underbrace{\left[{\parallel x-z\parallel }^2-{\parallel x-y\parallel }^2+2⟨z-x,y-z⟩\right]}}$

         $=D_f^{(B)}(y,z)-\frac{\mu }{2}{\parallel z-y\parallel }^2.$

         Convexity is a given because $D_{g_x}^{(B)}\ge 0$ since $D_f^{(B)}(y,z)\ge \frac{\mu }{2}{\parallel y-z\parallel }^2$. Similarly we obtain $L-\mu$-Lipschitz continuity of $\nabla g_x(y)$ because

         $$D_{g_x}^{(B)}(y,z)=D_f^{(B)}(y,z)-\frac{\mu }{2}{\parallel z-y\parallel }^2\le \frac{L-\mu }{2}{\parallel z-y\parallel }^2.$$

         So we can apply (iv)((b))(II) to $D_{g_x}^{(B)}$ with $z=x$ to obtain

         $D_f^{(B)}(y,x)$ $=D_{g_x}^{(B)}(y,x)+\frac{\mu }{2}{\parallel x-y\parallel }^2$

         ≥ 1 2(L-μ) ∥∇g_x(y) - ∇g_x(x)∥^2 + μ 2 ∥x-y∥^2

         $=\frac{1}{2(L-\mu )}\underset{{\parallel (\nabla f(y)-\mu y)-(\nabla f(x)-\mu x)\parallel }^2}{\underbrace{{\parallel \nabla f(y)-\mu (y-x)-\nabla f(x)\parallel }^2}}+\frac{\mu }{2}{\parallel x-y\parallel }^2.$

         Which is symmetric in $x$ and $y$ so we can apply the same trick as in (iv)((b))(III) to get

         $⟨\nabla f(y)-\nabla f(x),y-x⟩$ $=D_f^{(B)}(y,x)+D_f^{(B)}(y,x)$

         $\ge \frac{1}{(L-\mu )}\underset{={\parallel \nabla f(y)-\nabla f(x)\parallel }^2-2\mu ⟨\nabla f(y)-\nabla f(x),y-x⟩+{\mu }^2{\parallel y-x\parallel }^2}{\underbrace{{\parallel \nabla f(y)-\mu (y-x)-\nabla f(x)\parallel }^2}}+\mu {\parallel x-y\parallel }^2.$

         Collecting the scalar product on the left results in

         $$\underset{=\frac{L+\mu }{L-\mu }}{\underbrace{\left(1+\frac{2\mu }{L-\mu }\right)}}⟨\nabla f(y)-\nabla f(x),y-x⟩\ge \frac{1}{(L-\mu )}{\parallel \nabla f(y)-\nabla f(x)\parallel }^2+\left(\frac{{\mu }^2}{L-\mu }+\mu \right){\parallel x-y\parallel }^2.$$

         Multiplying both sides by $\frac{L-\mu }{L+\mu }$ we finally get

         $$⟨\nabla f(y)-\nabla f(x),y-x⟩\ge \frac{1}{(L+\mu )}{\parallel \nabla f(y)-\nabla f(x)\parallel }^2+\underset{=\frac{\mu L}{L+\mu }}{\underbrace{\left(\frac{{\mu }^2+\mu (L-\mu )}{L+\mu }\right)}}{\parallel x-y\parallel }^2.\mathit{∎}$$