BoydC3pt1

---

Basic properties and examples (of convex functions)

Definition (of convex functions)

Def 1.1. \(f:\mathbb{R}^n\to\mathbb{R}\) is convex if \(\mathbf{dom}~f\) is a convex set and

\[f(\theta x+(1-\theta)y)\leq \theta f(x)+(1-\theta)f(y) \]

for all \(x,y\in \mathbf{dom}~f\), \(\theta\in\left[0,1\right]\).

Def 1.2. \(f\) is concave if \(−f\) is convex.

Def 1.3. \(f:\mathbb{R}^n\to\mathbb{R}\) is strictly convex if \(\mathbf{dom}~f\) is a convex set and

\[f(\theta x+(1-\theta)y)< \theta f(x)+(1-\theta)f(y) \]

for all \(x,y\in \mathbf{dom}~f\), \(x\neq y\), \(\theta\in\left(0,1\right)\).

Thm 1.1. \(f:\mathbb{R}^n\to\mathbb{R}\) is convex iff \(g:\mathbb{R}\to \mathbb{R}\),

\[g(t)=f(x+tv),\quad \mathbf{dom}~g=\{x+tv~|~x+tv\in\mathbf{dom}~f\} \]

is convex for \(x\in \mathbf{dom}~f\), \(v\in\mathbb{R}^n\).

Extended-value extensions

Def 1.4. If \(f\) is convex, extended-value extension \(\tilde{f}:\mathbb{R}^n\to \mathbb{R}\cup\{\infty\}\) :

\[\tilde{f}=\begin{cases} f(x)\quad & x\in\mathbf{dom}~f\\ \infty & x\notin \mathbf{dom}~f \end{cases} \]

and for \(0<\theta<1\), \(\forall~x\in\mathbb{R}^n\),

\[\tilde{f}(\theta x+(1-\theta)y)< \theta \tilde{f}(x)+(1-\theta)\tilde{f}(y) \]

First-order conditions

Thm 1.2. 1st-order condition: Suppose \(f\) is differentiable, \(f\) is convex iff \(\mathbf{dom}~f\) is convex and

\[f(y)\geq f(x)+\nabla f(x)^T(y-x) \]

If \(f\) is strictly convex, then \(f(y)> f(x)+\nabla f(x)^T(y-x)\)

If \(f\) is concave, then \(f(y) \leq f(x)+\nabla f(x)^T(y-x)\)

proof.

\(1^\circ\) convex \(\Rightarrow\) inequality.

convex \(\overset{\text{def 1.1}}{\Rightarrow}\) \(\theta f(y)\geq f(\theta y+(1-\theta)x)-(1-\theta)f(x)\) \(\Rightarrow\) \(f(y)\geq f(x)+\frac{f(x+\theta (y-x))-f(x)}{\theta}\) \(\Rightarrow\) \(f(y)\geq f(x)+\lim_{\theta\to 0}\frac{f(x+\theta (y-x))-f(x)}{\theta}\) \(\Rightarrow\) \(f(y)\geq f(x)+\nabla f(x)^T(y-x)\).

\(2^\circ\) inequality \(\Rightarrow\) convex.

Soln 1.

\[\begin{align} f(x) &\geq f(\theta x+(1-\theta)y)-(1-\theta)\nabla f(\theta x+(1-\theta)y)^T(y-x),\\ f(y) &\geq f(\theta x+(1-\theta)y)+\theta\nabla f(\theta x+(1-\theta)y)^T(y-x). \end{align} \]

From \(\theta (1)+(1-\theta)(2)\),

\[\begin{align*} \theta f(x) + (1-\theta) f(y)\geq f(\theta x+(1-\theta)y) \end{align*} \]

\(i.e.\) \(f\) is convex.

Soln 2. utilize the monotonicity of \(\nabla f(x)\).

\[\begin{aligned} f(y)&\geq f(x)+\nabla f(x)^T(y-x)\\ f(x)&\geq f(y)-\nabla f(y)^T(y-x) \end{aligned} \]

\(\Rightarrow\) \((\nabla f(y)-\nabla f(x))^T(y-x)\geq 0\).

let

\[g(\theta) := \theta f(x) + (1-\theta) f(y) - f(\theta x+(1-\theta)y) \]

Hence

\[g^{\prime}(\theta) = f(x)-f(y)+\nabla f(\theta x+(1-\theta)y)^T(y-x) \]

\(\forall~\theta_1,\theta_2\in\left[0,1\right],\theta_1\neq\theta_2\), \(\tilde{x}:=\theta_1 x+(1-\theta_1)y\), \(\tilde{y}:=\theta_2 x+(1-\theta_2)y\)

\[\begin{align*} &(g^{\prime}(\theta_1)-g^{\prime}(\theta_2))(\theta_1-\theta_2)\\ =& (\nabla f(\tilde{x})-\nabla f(\tilde{y}))^T(\theta_1-\theta_2)(y-x)\\ =& -(\nabla f(\tilde{x})-\nabla f(\tilde{y}))^T(\tilde{x}-\tilde{y})\\ \leq& 0 \end{align*} \]

So \(g^{\prime}(\theta)\) is monotonically decreasing.

Furthermore, \(g(0)=g(1)\) \(\Rightarrow\) \(\exists~\xi\in\left(0,1\right)\) \(s.t.\) \(g^{\prime}(\xi)=0\). Hence \(\forall~\theta\in \left[0,\xi \right]\), \(g^{\prime}(\theta)\geq 0\), so \(g(\theta)\geq g(0)=0\);\(\forall~\theta\in \left[\xi,1 \right]\), \(g^{\prime}(\theta)\leq 0\), so \(g(\theta)\geq g(1)=0\), which means \(\forall~\theta\in\left[0,1 \right]\), \(g(\theta)\geq 0\) \(i.e.\) \(f\) is convex.

Second-order conditions

Thm 1.3. 2nd-order conditions: For twice differentiable \(f\) with convex domain, \(f\) is convex iff its Hessian is positive semidefinite:

\[\nabla^2 f(x) \succeq 0 \]

\(f\) is strictly convex iff \(\nabla^2 f(x) \succ 0\).

Examples of (convex/concave functions)

• log-sum-exp: \(f(x)=\log \sum_{k=1}^n\exp (x_k)\) is convex
• geometric mean: \(f(x)=(\Pi_{k=1}^n x_k)^{\frac{1}{n}}\) on \(\mathbb{R}_{++}^n\) is concave.
• Log-determinant: \(f(X) = \log \det X\) is concave.

Sublevel sets

Def 1.5. \(\alpha\)-sublevel set of \(f:\mathbb{R}^n\to\mathbb{R}\) :

\[C_{\alpha}=\{x\in\mathbf{dom}~|~f(x)\leq\alpha\} \]

sublevel sets of convex functions are convex .

Converse is false. For example, \(f(x) =-e^{x}\) is not convex on \(\mathbb{R}\), but all its sublevel sets are convex.

If \(f\) is concave, then its \(\alpha\)-superlevel set, given by \(\{x \in \mathbf{dom}~f ~|~ f(x) \geq α\}\), is a convex set

Epigraph

Def 1.6. epigraph of \(f:\mathbb{R}^n\to\mathbb{R}\) :

\[\mathbf{epi}~f=\{ (x,t)\in\mathbb{R}^{n+1}~|~x\in\mathbf{dom}~f,~f(x)\leq t \} \]

\(f\) is convex iff \(\mathbf{epi}~f\) is a convex set.

A function is concave iff its hypograph : \(\mathbf{hypo}~f=\{(x,t)~|~t\leq f(x)\}\) is a convex set.

Jensen’s inequality and extensions

Thm 1.4. If \(f\) is convex, \(x_i\in\mathbf{dom}~f\), \(\theta_i\geq 0\), \(i\in\left[ k\right]\) and \(\sum_{i=1}^k \theta_i=1\), then

\[f(\sum_{i=1}^k\theta_i x_i)\leq\sum_{i=1}^k\theta_i f(x_i) \]

Furthermore, if \(p(x)\geq 0\) on \(S\subseteq\mathbf{dom}~f\), \(\int_S p(x)=1\), then

\[f(\int_S p(x)x~dx)\leq \int_S p(x) f(x)~dx \]

If \(x\) is a random variable, then

\[f(\mathbb{E}(x))\leq\mathbb{E}(f(x)) \]

---

Next part: BoydC3pt2