Commit 39a9de7a by nanored

### Début du DM 1

parent 838b9134
dm1.tex 0 → 100644
 \documentclass[11pt]{article} \usepackage[english]{babel} \usepackage[utf8]{inputenc} \usepackage[T1]{fontenc} \usepackage[left=2.7cm,right=2.7cm,top=3cm,bottom=3cm]{geometry} \usepackage{amsmath,amssymb,amsfonts} \usepackage{kpfonts} \usepackage{tikz} \usepackage{bbm} \title{ \noindent\rule{\linewidth}{0.4pt} \huge Convex Optimization --- DM 1 \noindent\rule{\linewidth}{1pt} } \newcommand{\R}{\mathbb{R}} \newcommand{\trans}{\mathsf{T}} \newcommand{\dist}{\text{\textbf{dist}}} \author{Yoann Coudert-\,-Osmont} \begin{document} \maketitle \section*{Exercise 2.12} \paragraph{(a)} For $x \in \R^n$ we have $\alpha \leqslant a^\trans x \leqslant \beta \Leftrightarrow \left\{ \begin{array}{lll} - a^\trans x & \leqslant & - \alpha \\ a^\trans x & \leqslant & \beta \end{array} \right.$. \\ Then a slab: $$\left\{ x \in \R^n ~|~ \alpha \leqslant a^\trans x \leqslant \beta \right\} = \left\{ x \in \R^n ~|~ - a^\trans x \leqslant - \alpha \right\} \cap \left\{ x \in \R^n ~|~ a^\trans x \leqslant - \beta \right\}$$ is the intersection of two halfspaces which are convex sets. Then \fbox{a slab is convex}. \paragraph{(b)} We denote by $e_i$ the vector with a one at position $i$ and zeros everywhere else. We then have : $$\left\{ x \in \R^n ~|~ \alpha_i \leqslant x_i \leqslant \beta_i, \; i = 1, ..., n \right\} = \bigcap_{i = 1}^n \left\{ x \in \R^n ~|~ \alpha_i \leqslant e_i^\trans x \leqslant \beta_i \right\}$$ Then a rectangle is the intersection of $n$ slabs. Because slabs are convex, \fbox{a rectangle is convex}. \paragraph{(c)} A wedge can be written as the intersection of two halfspaces : $$\left\{ x \in \R^n ~|~ a_1^\trans x \leqslant b_1, \; a_2^\trans x \leqslant b_2 \right\} = \left\{ x \in \R^n ~|~ a_1^\trans x \leqslant b_1 \right\} \cap \left\{ x \in \R^n ~|~ a_2^\trans x \leqslant b_2 \right\}$$ Because halfspaces are convex, \fbox{a wedge is convex}. \paragraph{(d)} We rewrite our set as an intersection of many sets: $$T = \left\{ x ~|~ \| x - x_0 \|_2 \leqslant \| x - y \|_2 \text{ for all } y \in S \right\} = \bigcap_{y \in S} \left\{ x ~|~ \| x - x_0 \|_2 \leqslant \| x - y \|_2 \right\}$$ Then we rewrite the condition on $x$ : $$\begin{array}{lll} \| x - x_0 \|_2 \leqslant \| x - y \|_2 & \Leftrightarrow & \| x - x_0 \|_2^2 \leqslant \| x - y \|_2^2 \\[2mm] & \Leftrightarrow & x^\trans x - 2 x^\trans x_0 + x_0^\trans x_0 \leqslant x^\trans x - 2 x^\trans y + y^\trans y \\[1mm] & \Leftrightarrow & 2 x^\trans (y - x_0) \leqslant y^\trans y - x_0^\trans x_0 \\[2mm] & \Leftrightarrow & (y - x_0)^\trans x \leqslant \frac{1}{2} \left( \| y \|_2^2 - \| x_0 \|_2^2 \right) \end{array}$$ We then have : $$T = \bigcap_{y \in S} \left\{ x ~|~ (y - x_0)^\trans x \leqslant \frac{1}{2} \left( \| y \|_2^2 - \| x_0 \|_2^2 \right) \right\}$$ So $T$ is the intersection of many halfspaces which are convex sets. Then \fbox{$T$ is convex}. \paragraph{(e)} We consider the set: $$U = \left\{ x ~|~ \dist(x, S) \leqslant \dist(x, T) \right\}$$ We will show that $U$ is not necessarily convex by taking the example with $n = 1$, $S = \{-1, 1\}$ and $T = \R \setminus S$. \\ For $x \in \R$. Because $T$ is dense in $\R$ we have $\dist(x, T) = 0$. Then $x$ is in $U$ if and only if $\dist(x, S)~=~0 \Leftrightarrow x \in S$. Thus $U = S$. $S$ is clearly not convex. So \fbox{$U$ is not convex}. \paragraph{(f)} We rewrite our set as an intersection of many sets: $$T = \left\{ x ~|~ x + S_2 \subseteq S_1 \right\} = \bigcap_{y \in S_2} \left\{ x ~|~ x + y \in S_1 \right\} = \bigcap_{y \in S_2} \left\{ z - y ~|~ z \in S_1 \right\} = \bigcap_{y \in S_2} f_y(S_1)$$ With $S_1 \subseteq \R^n$ a convex set and $f_y(z) = z - y$ an affine function. We know that the image of a convex set under an affine function is convex. So $f_y(S_1)$ is a convex set for all $y \in S_2$. Thus $T$ is the intersection of many convex sets. So \fbox{$T$ is convex}. \paragraph{(e)} We rewrite the condition on $x$: $$\begin{array}{lll} \| x - a \|_2 \leqslant \theta \| x - b \|_2 & \Leftrightarrow & \| x - a \|_2^2 \leqslant \theta^2 \| x - b \|_2^2 \\[2mm] & \Leftrightarrow & x^\trans x - 2 x^\trans x_0 + x_0^\trans x_0 \leqslant \theta^2 x^\trans x - 2 \theta^2 x^\trans y + \theta^2 y^\trans y \\[1mm] & \Leftrightarrow & (1 - \theta^2) x^\trans x + 2 x^\trans (\theta^2 b - a) \leqslant \theta^2 b^\trans b - a^\trans a \\[2mm] & \Leftrightarrow & (1 - \theta^2) \| x \|_2^2 + ( \theta^2 b - a)^\trans x \leqslant \frac{1}{2} \left( \| \theta b \|_2^2 - \| a \|_2^2 \right) \end{array}$$ We then consider the function $f(x) = (1 - \theta^2) \| x \|_2^2 + ( \theta^2 b - a)^\trans x$. It is supposed that $\theta < 1$ then $(1 - \theta^2) > 0$. Thus $f$ is the sum of an affine function and a squared norm multiplied by a positive scalar. Because $f$ is a sum of convex function, $f$ is convex. We consider $\alpha = \frac{1}{2} \left( \| \theta b \|_2^2 - \| a \|_2^2 \right)$. Then the set: $$S = \left\{ x ~|~ \| x - a \|_2 \leqslant \theta \| x - b \|_2 \right\} = \left\{ x ~|~ f(x) \leqslant \alpha \right\}$$ is the $\alpha$-sublevel set of $f$. So \fbox{$S$ is convex}. \section*{Exercise 3.21} \paragraph{(a)} We know that all norms on $\R^n$ are convex. Furthermore we also know that composition with affine function preserve convexity. So the function: $$f_i(x) = \left\| A^{(i)} x - b^{(i)} \right\|$$ is convex. Finally we know that pointwise maximum of convex functions is convex. So: \begin{center} \fbox{$\displaystyle f : x \mapsto \max_{i = 1, ..., k} f_i(x) \;$ is convex} \end{center} \paragraph{(b)} We denote by $R$ a subset of $\{1, ..., n\}$ of size $r$. Then we denote by $f_R$ the function: $$f_R(x) = \sum_{i \in R} |x_i|$$ If we denote by $e_R$ the vector which have ones at every positions in $R$ and zeros everywhere else, we obtain the following equality : $f_R(x) = \| e_R^\trans x \|_1$. We know that norms are convex and that composition with affine functions preserve convexity. So $f_R$ is convex. \\ For $S$ the set of indices of the $r$ largest components of $|x|$ we have $f_S(x) = f(x)$. Furthermore for every $R$ subset of $\{1, ..., n\}$ of size $r$ we have $f_R(x) \leqslant f(x)$. So: $$f(x) = \max_{R \subseteq \{1, ..., n\} \atop |R| = r} f_R(x)$$ Then $f$ is a pointwise maximum of convex functions. So \fbox{$f$ is convex}. \section*{Exercise 3.32} \paragraph{(a)} Let $x < y$ and $0 \leqslant \theta \leqslant 1$. Then we let $z = \theta x + (1 - \theta) y$. \\ To show that $fg$ is convex, we need to show that: $$f(z)g(z) \leqslant \theta f(x)g(x) + (1 - \theta) f(y)g(y)$$ Because $g$ is positive and $f$ convex, we have : $$f(z) g(z) \leqslant \left[ \theta f(x) + (1 - \theta) f(y) \right] g(z)$$ Because $f$ is positive and $g$ convex, we now have : \begin{array}{lll} f(z) g(z) & \leqslant & \left[ \theta f(x) + (1 - \theta) f(y) \right] \left[ \theta g(x) + (1 - \theta) g(y) \right] \\ & \leqslant & \theta^2 f(x)g(x) + \theta (1 - \theta) \left[ f(x)g(y) + f(y)g(x) \right] + (1 - \theta)^2 f(y)g(y) \end{array} \label{beg} Then we remarks : \left( f(x) - f(y) \right) \left( g(y) - g(x) \right) \leqslant 0 \Leftrightarrow f(x)g(y) + f(y)g(x) \leqslant f(x)g(x) + f(y)g(y) \label{ineq} We now consider the two cases : \begin{itemize} \item If $f$ and $g$ are non-decreasing. Then $f(x) - f(y) \leqslant 0$ and $g(y) - g(x) \geqslant 0$ so from the equivalence \eqref{ineq}, we have : $$f(x)g(y) + f(y)g(x) \leqslant f(x)g(x) + f(y)g(y)$$ \item If $f$ and $g$ are non-increasing. Then $f(x) - f(y) \geqslant 0$ and $g(y) - g(x) \leqslant 0$ so from the equivalence \eqref{ineq}, we have : $$f(x)g(y) + f(y)g(x) \leqslant f(x)g(x) + f(y)g(y)$$ \end{itemize} In both cases we obtain the same conclusion that we can use in \eqref{beg} to obtain: $$\begin{array}{lll} f(z) g(z) & \leqslant & \theta^2 f(x)g(x) + \theta (1 - \theta) \left[ f(x)g(x) + f(y)g(y) \right] + (1 - \theta)^2 f(y)g(y) \\ & \leqslant & \left[ \theta^2 + \theta (1 - \theta) \right] f(x)g(x) + \left[ (1 - \theta)^2 + \theta (1 - \theta) \right] f(y)g(y) \\ & \leqslant & \theta f(x)g(x) + (1 - \theta) f(y)g(y) \end{array}$$ So we have that \fbox{$fg$ is convex}. \paragraph{(b)} Let $x < y$ and $0 \leqslant \theta \leqslant 1$. Then we let $z = \theta x + (1 - \theta) y$. \\ To show that $fg$ is convex, we need to show that: $$f(z)g(z) \geqslant \theta f(x)g(x) + (1 - \theta) f(y)g(y)$$ Because $g$ is positive and $f$ concave, we have : $$f(z) g(z) \geqslant \left[ \theta f(x) + (1 - \theta) f(y) \right] g(z)$$ Because $f$ is positive and $g$ concave, we now have : \begin{array}{lll} f(z) g(z) & \geqslant & \left[ \theta f(x) + (1 - \theta) f(y) \right] \left[ \theta g(x) + (1 - \theta) g(y) \right] \\ & \geqslant & \theta^2 f(x)g(x) + \theta (1 - \theta) \left[ f(x)g(y) + f(y)g(x) \right] + (1 - \theta)^2 f(y)g(y) \end{array} \label{beg2} Then we remarks : \left( f(x) - f(y) \right) \left( g(y) - g(x) \right) \geqslant 0 \Leftrightarrow f(x)g(y) + f(y)g(x) \geqslant f(x)g(x) + f(y)g(y) \label{ineq2} We now consider the two cases : \begin{itemize} \item If $f$ is non-increasing and $g$ is non-decreasing. Then $f(x) - f(y) \geqslant 0$ and $g(y) - g(x) \geqslant 0$ so from the equivalence \eqref{ineq2}, we have : $$f(x)g(y) + f(y)g(x) \geqslant f(x)g(x) + f(y)g(y)$$ \item If $f$ is non-decreasing and $g$ is non-increasing. Then $f(x) - f(y) \leqslant 0$ and $g(y) - g(x) \leqslant 0$ so from the equivalence \eqref{ineq2}, we have : $$f(x)g(y) + f(y)g(x) \geqslant f(x)g(x) + f(y)g(y)$$ \end{itemize} In both cases we obtain the same conclusion that we can use in \eqref{beg2} to obtain: $$\begin{array}{lll} f(z) g(z) & \geqslant & \theta^2 f(x)g(x) + \theta (1 - \theta) \left[ f(x)g(x) + f(y)g(y) \right] + (1 - \theta)^2 f(y)g(y) \\ & \geqslant & \left[ \theta^2 + \theta (1 - \theta) \right] f(x)g(x) + \left[ (1 - \theta)^2 + \theta (1 - \theta) \right] f(y)g(y) \\ & \geqslant & \theta f(x)g(x) + (1 - \theta) f(y)g(y) \end{array}$$ So we have that \fbox{$fg$ is concave}. \paragraph{(c)} We start from the inequality : $$0 \leqslant \left( g(x) - g(y) \right)^2 = g(x)^2 + g(y)^2 - 2 g(x) g(y)$$ Let $0 \leqslant \theta \leqslant 1$. We then multiply by $\theta (1 - \theta) \geqslant 0$ : $$0 \leqslant \theta (1 - \theta) \left( g(x)^2 + g(y)^2 - 2 g(x) g(y) \right) = \theta (1 - \theta) \left( g(x)^2 + g(y)^2 \right) + \left( \theta^2 + (1 - \theta)^2 - 1 \right) g(x) g(y)$$ We add $g(x)g(y)$ to this inequality : $$\begin{array}{lll} g(x) g(y) & \leqslant & \theta (1 - \theta) \left( g(x)^2 + g(y)^2 \right) + \left( \theta^2 + (1 - \theta)^2 \right) g(x) g(y) \\ & \leqslant & \left[ \theta g(x) + (1 - \theta) g(y) \right] \left[ \theta g(y) + (1 - \theta) g(x) \right] \end{array}$$ Because $g$ is positive we can divide by $g(x)g(y) \left[ \theta g(x) + (1 - \theta) g(y) \right]$. This gives the following interesting result : \dfrac{1}{\theta g(x) + (1 - \theta) g(y)} \leqslant \dfrac{\theta}{g(x)} + \dfrac{1 - \theta}{g(y)} \label{inv} Now, let $z = \theta x + (1 - \theta) y$. Because $g$ is concave, we get : $$g(z) \geqslant \theta g(x) + (1 - \theta) g(y)$$ Because $g$ is positive the two sides of the inequality are non zero. We can take the inverse of each sides and inverse the inequality to obtain : $$\dfrac{1}{g(z)} \leqslant \dfrac{1}{g(x) + (1 - \theta) g(y)}$$ By using the inequality \eqref{inv} we finally get : $$\dfrac{1}{g(z)} \leqslant \dfrac{\theta}{g(x)} + \dfrac{1 - \theta}{g(y)}$$ Then the function $1/g$ is convex, positive and non-decreasing. Indeed because $g$ is non-increasing, its inverse is non-decreasing. Then $f$ and $1/g$ satisfy the conditions of the question \textbf{(a)}. So their product is convex. This means that \fbox{$f/g$ is convex}. \section*{Exercise 3.36} \paragraph{(a)} We have : $$f^*(y) = \max_{x \in \R^n} g(x, y) \qquad \text{where } g(x, y) = y^\trans x - \max_{i = 1, ..., n} x_i$$ In the case $n = 1$ we have $g(x, y) = (y - 1) x$. Then : \begin{center} \fbox{$\displaystyle n = 1 \quad \Rightarrow \quad f^*(y) = \left\{ \begin{array}{lrr} 0 & \text{if} & y = 1 \\ + \infty & \text{if} & y \neq 1 \end{array} \right.$} \end{center} \vspace{5mm} Now we consider $n > 1$. If there exists $i \in \{1, ..., n\}$ such that $y_i < 0$ then for $\lambda > 0$ we have : $$g(- \lambda e_i, y) = - \lambda y_i \xrightarrow[\lambda \rightarrow + \infty]{} +\infty$$ Where $e_i$ is the vector with a one at position $i$ and zeros everywhere else. Then if $y$ has a negative component $f^*(y) = +\infty$. \\ We now look at the case $y \in \R_+^n$. We denote by $\mathbbm{1}_n$ the vector with ones everywhere. We also denote by $m(x)$, the maximum $\max_{i = 1, ..., n} x_i$. Then $m \left( m(x) \mathbbm{1}_n \right) = m(x)$ and by positivity of $y$ we have $y^\trans x \leqslant y^\trans \left( m(x) \mathbbm{1}_n \right)$. Thus : $$g(x, y) \leqslant g \left( m(x) \mathbbm{1}_n, y \right)$$ This result allow us to reduce our maximum on $\R_n$ to a maximum on $\R$ : $$f^*(y) = \max_{x \in \R} \; g(x \mathbbm{1}_n, y) = \max_{x \in \R} \; x \left( \| y \|_1 - 1 \right)$$ We then obtain : \begin{center} \fbox{$\displaystyle f^*(y) = \left\{ \begin{array}{ll} 0 & \text{if } \; y \in \R_+^n, \; \| y \|_1 = 1 \\ + \infty & \text{else} \end{array} \right.$} \end{center} \end{document} \ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!