【笔记】凸优化 Convex Optimization

Differentiation

Def. Gradient
\(f:{\cal X}\sube\mathbb{R} ^N\to \mathbb{R}\) is differentiable. Then the gradient of \(f\) at \({\bf x}\in\cal{X}\), denoted by \(\nabla f({\bf x})\), is defined by

\[\nabla f({\bf x}) = \begin{bmatrix} \frac{\partial f}{\partial {\bf x} _1}({\bf x}) \\ \vdots \\ \frac{\partial f}{\partial {\bf x} _N}({\bf x}) \end{bmatrix} \]

Def. Hessian
\(f:{\cal X}\sube\mathbb{R} ^N\to \mathbb{R}\) is twice differentiable. Then the Hessian of \(f\) at \({\bf x}\in\cal{X}\), denoted by \(\nabla ^2 f({\bf x})\), is defined by

\[\nabla ^2 f({\bf x}) = \begin{bmatrix} \frac{\partial ^2 f}{\partial {\bf x} _1^2}({\bf x}) & \cdots & \frac{\partial ^2 f}{\partial {\bf x} _1, {\bf x} _N}({\bf x}) \\ \vdots & \ddots & \vdots \\ \frac{\partial ^2 f}{\partial {\bf x} _N, {\bf x} _1}({\bf x}) & \cdots & \frac{\partial ^2 f}{\partial {\bf x} _N^2}({\bf x}) \end{bmatrix} \]

Th. Fermat's theorem
\(f:{\cal X}\sube\mathbb{R} ^N\to \mathbb{R}\) is twice differentiable. If \(f\) admits a local extrememum at \({\bf x} ^ *\), then \(\nabla f(\bf x ^ *)=0\)

Convexity

Def. Convex set
A set \({\cal X}\sube \mathbb{R} ^N\) is said to be convex, if for any \({\bf x, y}\in \cal X\), the segment \(\bf [x, y]\) lies in \(\cal X\), that is \(\{\alpha{\bf x} + (1-\alpha){\bf y}:0\le\alpha\le 1 \}\sube\cal X\)

Th. Operations that preserve convexity

• \({\cal C} _i\) is convex for all \(i\in I\), then \(\bigcap _{i\in I} {\cal C} _i\) is also convex
• \({\cal C} _1, {\cal C} _2\) is convex, then \({\cal C _1+C _2}=\{x _1+x _2:x _1\in {\cal C} _1, x _2\in {\cal X} _2 \}\) is also convex
• \({\cal C} _1, {\cal C} _2\) is convex, then \({\cal C _1\times C _2}=\{(x _1, x _2):x _1\in {\cal C} _1, x _2\in {\cal X} _2 \}\) is also convex
• Any projection of a convex set is also convex

Def. Convex hull
The Convex hull \(\text{conv}(\cal X)\) of set \({\cal X}\sube \mathbb{R} ^N\), is the minimal convex set containing \(\cal X\), that is

\[\text{conv}({\cal X})=\left\{ \sum _{i=1}^m \alpha _i {\bf x} _i:\forall ({\bf x} _1,\dotsb, {\bf x} _m)\in {\cal X}, \alpha _i\ge 0, \sum _{i=1} ^m \alpha _i = 1 \right\} \]

Def. Epigraph
The epigraph of \(f:{\cal X}\to \mathbb{R}\), denoted by \(\text{Epi } f\), is defined by \(\{(x,y):x\in {\cal X}, y\ge f(x)\}\)

Def. Convex function
Convex set \(\cal X\). A function \(f:{\cal X}\to \mathbb{R}\) is said to be convex, iff \(\text{Epi }f\) is convex, or equivalently, for all \({\bf x, y}\in {\cal X},\alpha \in [0,1]\)

\[f(\alpha{\bf x} + (1-\alpha){\bf y})\le \alpha f({\bf x}) + (1-\alpha) f({\bf y}) \]

Moreover, \(f\) is said to be strictly convex if the inequality is strict when \(\bf x\ne y\) and \(\alpha\in (0,1)\). \(f\) is said to be (strictly) concave if \(-f\) is (strictly) convex.

Th. Convex function characterized by first-order differential
\(f:{\cal X}\sube\mathbb{R} ^N\to \mathbb{R}\) is differentiable. Then \(f\) is convex iff \(\text{dom}(f)\) is convex, and

\[\forall {\bf x, y}\in \text{dom}(f),\quad f({\bf y})-f({\bf x})\ge \nabla f({\bf x})\cdot({\bf y-x}) \]

交换 \(\bf x, y\)，得到 \(f({\bf x})-f({\bf y})\ge \nabla f({\bf y})\cdot({\bf x-y})\)，相加得：

\[\lang\nabla f({\bf x}) - \nabla f({\bf y}),{\bf x-y}\rang\ge 0 \]

其含义为 “梯度单调且内积大等于零”，这也是凸性的等价条件之一

Th. Convex function characterized by second-order differential
\(f:{\cal X}\sube\mathbb{R} ^N\to \mathbb{R}\) is twice differentiable. Then \(f\) is convex iff \(\text{dom}(f)\) is convex, and its Hessian is positive semidefinite (半正定)

\[\forall {\bf x}\in \text{dom}(f),\quad \nabla ^2 f({\bf x})\succeq 0 \]

• 对称阵为半正定，若其所有特征值非负；\(A\succeq B\) 等价于 \(A-B\) 为半正定
• If \(f\) is scalar (eg. \(x\mapsto x ^2\)), then \(f\) is convex iff \(\forall x\in \text{dom}(f), f''(x)\ge 0\)
• For example
  • Linear functions is both convex and concave
  • Any norm \(\Vert\cdot\Vert\) over convex set \(\cal X\) is a convex function
    \(\Vert\alpha{\bf x}+(1-\alpha){\bf y} \Vert\le \Vert\alpha{\bf x}\Vert+\Vert(1-\alpha){\bf y} \Vert\le \alpha\Vert{\bf x}\Vert+(1-\alpha)\Vert{\bf y}\Vert\)
• Using composition rules to prove convexity

Th. Composition of convex/concave functions
Assume \(h:\mathbb{R}\to\mathbb{R}\) and \(g:\mathbb{R} ^N\to\mathbb{R}\) are twice differentiable. Define \(f({\bf x})=h(g({\bf x})), \forall {\bf x}\in \mathbb{R} ^N\), then

• \(h\) is convex & non-decreasing, \(g\) is convex \(\implies\) \(f\) is convex
• \(h\) is convex & non-increasing, \(g\) is concave \(\implies\) \(f\) is convex
• \(h\) is concave & non-decreasing, \(g\) is concave \(\implies\) \(f\) is concave
• \(h\) is concave & non-increasing, \(g\) is convex \(\implies\) \(f\) is concave

Proof: It holds for \(N=1\), which suffices to prove convexity (concavity) along all lines that intersect the domain.
Example: \(g\) could be any norm \(\Vert\cdot\Vert\)

Th. Pointwise maximum of convex functions
\(f _i\) is a convex function defined over convex set \(\cal C\) for all \(i\in I\), then \(f(x)=\sup _{i\in I}f _i(x), x\in \cal C\) is a convex function.
Proof: \(\text{Epi } f = \bigcap _{i\in I} \text{Epi } f _i\) is convex

• \(f({\bf x})=\max _{i\in I}{\bf w} _i\cdot {\bf x}+b _i\) over a convex set, is a convex function
• The maximum eigenvalue \(\lambda _{\max}({\bf M})\) over the set of symmetric matrices, is a convex function, since \(\lambda _{\max}({\bf M})=\sup _{\Vert\bf x\Vert _2\le 1}{\bf x}'{\bf Mx}\) is supremum of linear functions \({\bf M}\mapsto{\bf x}'{\bf Mx}\)
  More generally, let \(\lambda _{k}(\bf M)\) denote the top \(k\) eigenvalues, then \({\bf M}\mapsto \sum _{i=1} ^{k}\lambda _{i}(\bf M)\) and \({\bf M}\mapsto \sum _{i=n-k+1} ^{n}\lambda _{i}({\bf M})=-\sum _{i=1} ^{k}\lambda _i(-\bf M)\) are both convex function

Th. Partial infimum
Convex function \(f\) defined over convex set \(\cal C\sube X\times Y\), and conves set \(\cal B\sube Y\). Then \({\cal A}=\{x\in {\cal X}:\exist y\in{\cal B}, (x, y)\in {\cal C} \}\) is convex set if non-empty, and \(g(x)=\inf _{y\in\cal B} f(x, y)\) for all \(x\in \cal A\) is convex function.
For example, the distance to convex set \(\cal B\), \(d(x)=\inf _{y\in \cal B}\Vert x-y\Vert\) is convex function

Th. Jensen's inequality
Let r.v. \(X\) in convex set \({\cal C}\sube \mathbb{R} ^N\), and convex function \(f\) defined over \(\cal C\). Then, \(\mathbb{E}[X]\in {\cal C}, \mathbb{E}[f(X)]\) is finite, and

\[f(\mathbb{E}[X])\le \mathbb{E}[f(X)] \]

Sketch of proof: extending \(f(\sum \alpha x)\le\sum \alpha f(x)\) and \(\sum \alpha=1\) that can be interpreted as probabilities, to arbitraty contributions.

Smoothness, strong convexity

参考 [https://zhuanlan.zhihu.com/p/619288199]
考虑二阶导的 lipschitz 连续性

Def. \(\beta\)-smooth
称函数 \(f\) 是 \(\beta\)-smooth 的，若

\[\forall{\bf x, y}\in\text{dom}(f),\quad \Vert\nabla f({\bf x}) - \nabla f({\bf y})\Vert\le \beta\Vert {\bf x-y}\Vert \]

等价于如下命题均成立：

• \(\frac{\beta}{2}\Vert {\bf x}\Vert ^2 - f({\bf x})\) 是凸函数
• \(\forall{\bf x, y}\in\text{dom}(f),\quad f({\bf y})\le f({\bf x}) + \nabla f({\bf x}) ^\top ({\bf y-x}) + \frac{\beta}{2} \Vert{\bf y-x}\Vert ^2\)
• \(\nabla ^2 f({\bf x})\preceq \beta I\)

证明/说明
证明 \(g({\bf x})=\frac{\beta}{2}\Vert {\bf x}\Vert ^2 - f({\bf x})\) 为凸，可以考虑 \(\lang\nabla g({\bf x}) - \nabla g({\bf y}),{\bf x-y}\rang\ge 0\)，应用柯西不等式可证
感性理解之，\(f\) 的起伏 “拗不过” \(\frac{\beta}{2}\Vert {\bf x}\Vert ^2\) 的凸性，即 \(f\) 起伏不够大，也就是比较平滑 smooth
证明第二、三条，代入 \(g({\bf y})-g({\bf x})\ge \nabla g({\bf x})\cdot({\bf y-x})\) 和 \(\nabla ^2 g({\bf x})\succeq 0\) 即可；它的几何含义见下

Def. \(\alpha\)-strongly convex
称函数 \(f\) 是 \(\alpha\)-strongly convex 的，若

• \(\forall{\bf x, y}\in\text{dom}(f),\quad \Vert\nabla f({\bf x}) - \nabla f({\bf y})\Vert\ge \alpha\Vert {\bf x-y}\Vert\)
• \(f({\bf x}) - \frac{\alpha}{2}\Vert {\bf x}\Vert ^2\) 是凸函数
• \(\forall{\bf x, y}\in\text{dom}(f),\quad f({\bf y})\ge f({\bf x}) + \nabla f({\bf x}) ^\top ({\bf y-x}) + \frac{\alpha}{2} \Vert{\bf y-x}\Vert ^2\)
• \(\nabla ^2 f({\bf x})\succeq \alpha I\)

Def. \(\gamma\)-well-conditioned
称函数 \(f\) 是 \(\gamma\)-well-conditioned 的，若其同时是 \(\alpha\)-strongly convex 和 \(\beta\)-smooth 的；定义 \(f\) 的 condition number 为 \(\gamma=\alpha /\beta\le 1\)

Th. Linear combination of two convex functions

• 考虑两个凸函数的加和，有：
  • 若 \(f\) 为 \(\alpha _1\)-strongly convex，\(g\) 为 \(\alpha _2\)-strongly convex，则 \(f+g\) 为 \((\alpha _1+\alpha _2)\)-strongly convex
  • 若 \(f\) 为 \(\beta _1\)-smooth，\(g\) 为 \(\beta _2\)-smooth，则 \(f+g\) 为 \((\beta _1 + \beta _2)\)-smooth
• 考虑凸函数的数乘 \(k>0\)，有：
  • 若 \(f\) 为 \(\alpha\)-strongly convex，则 \(kf\) 为 \((k\alpha)\)-strongly convex
  • 若 \(f\) 为 \(\beta\)-smooth，则 \(kf\) 为 \((k\beta)\)-smooth

证明，利用凸函数满足 \(\lang\nabla f({\bf x}) - \nabla f({\bf y}),{\bf x-y}\rang\ge 0\) 和 \(\frac{\beta}{2}\Vert {\bf x}\Vert ^2 - f({\bf x}), f({\bf x}) - \frac{\alpha}{2}\Vert {\bf x}\Vert ^2\) 的凸性即可

Projections onto convex sets

之后的算法会涉及向凸集投影的概念；定义 \(\bf y\) 向凸集 \(\cal K\) 的投影，为

\[\prod _{\cal K}({\bf y})\triangleq \underset{{\bf x}\in\cal K}{\text{arg min }} \Vert {\bf x-y}\Vert \]

可以证明投影总是唯一的；投影还具有一个很重要的性质：
Th. Pythagorean theorem 勾股定理
凸集 \(\cal K\sube \mathbb{R} ^d,{\bf y}\in\mathbb{R} ^d,{\bf x}=\prod _{\cal K}({\bf y})\)，则任意 \(\bf z\in\cal K\)，\(\bf \Vert y-z\Vert\ge \Vert x-z\Vert\)
即，对凸集内的任一点，其到投影点的距离不大于其到被投影点的距离

Constrained optimization 带约束优化

Def. Constrained optimization problem
\({\cal X}\sube \mathbb{R} ^N,\ f, g _i:{\cal X}\to\mathbb{R}, i\in [m]\)，则带约束优化问题（也称为 primal problem）的形式为

\[\min _{\bf x\in\cal X} f({\bf x}) \\ subject\ to: g _i({\bf x})\le 0, \forall i\in [m] \]

记 \(\inf _{\bf x\in\cal X} f({\bf x})=p ^ *\)；注意到目前我们没有假设任何的 convexity；对于 \(g=0\) 的约束我们可以用 \(g\le 0, -g\le 0\) 来刻画

Dual problem and saddle point

解决这类问题，可以先引入拉格朗日函数 Lagrange function，将约束以非正项引入；然后转化成对偶问题

Def. Lagrange function
为带约束优化问题定义拉格朗日函数，为

\[\forall {\bf x}\in {\cal X},\forall{\boldsymbol{\alpha}\ge 0},\quad {\cal L}({\bf x}, {\boldsymbol{\alpha}})=f({\bf x}) + \sum _{i=1}^{m} \alpha _i g _i({\bf x}) \]

其中 \({\boldsymbol{\alpha}}=(\alpha _1,\dotsb, \alpha _m)'\) 称为对偶变量 dual variable
对于约束 \(g=0\)，其系数 \(\alpha=\alpha _+ - \alpha _-\) 不需要非负（但是下文给出定理时，要求 \(g,-g\) 同时为凸，从而 \(g\) 得是仿射函数 affine，即形如 \({\bf w\cdot x+b}\)）
注意到 \(p ^ * = \inf _{\bf x} \sup _{\boldsymbol{\alpha}}{\cal L}({\bf x}, {\boldsymbol{\alpha}})\)，因为当 \(\bf x\) 不满足约束时 \(\sup _{\boldsymbol{\alpha}}\) 可以取到无穷大，从而刻画了约束
有趣的来了，我们能构造一个 concave function，称为对偶函数 Dual function

Def. Dual function
为带约束优化问题定义对偶函数，为

\[\forall{\boldsymbol{\alpha}}\ge 0,\quad F({\boldsymbol{\alpha}})=\inf _{\bf x\in\cal X} {\cal L}({\bf x},{\boldsymbol{\alpha}})=\inf _{\bf x\in\cal X} \left(f({\bf x}) + \sum _{i=1}^{m} \alpha _i g _i({\bf x})\right) \]

它是 concave 的，因为 \(\cal L\) 是关于 \(\boldsymbol{\alpha}\) 的线性函数，且 pointwise infimum 保持了 concavity
同时注意到对任意 \({\boldsymbol{\alpha}}\)，\(F({\boldsymbol{\alpha}})\le \inf _{\bf x\in\cal X}f({\bf x})=p ^ *\)
定义对偶问题

Def. Dual problem
为带约束优化问题定义对偶问题，为

\[\max _{\boldsymbol{\alpha}} F({\boldsymbol{\alpha}}) \\ subject\ to:{\boldsymbol{\alpha}}\ge 0 \]

对偶问题是凸优化问题，即求 concave 函数的最大值，记其为 \(d ^ *\)；由上文可知 \(d ^ *\le p ^ *\)，也就是：

\[d ^ * = \sup _{\boldsymbol{\alpha}} \inf _{\bf x} {\cal L}({\bf x}, {\boldsymbol{\alpha}})\le \inf _{\bf x} \sup _{\boldsymbol{\alpha}}{\cal L}({\bf x}, {\boldsymbol{\alpha}}) = p ^ * \]

称为弱对偶 weak duality，取等情况称为强对偶 strong duality
接下来会给出：

当凸优化问题满足约束规范性条件 constraint qualification (Slater's contidion) 时（此为充分条件），有 \(d ^ *=p ^ *\)，且该解的充要条件为拉格朗日函数的鞍点 saddle point

Def. Constraint qualification (Slater's condition)
假设集合 \(\cal X\) 的内点非空 \(\text{int}({\cal X})\ne \empty\)：

• 定义 strong constraint qualification (Slater's condition) 为

\[\exist \bar{{\bf x}}\in\text{int}({\cal X}):\forall i\in [m], g _i(\bar{{\bf x}})< 0 \]

• 定义 weak constraint qualification (weak Slater's condition) 为

\[\exist \bar{{\bf x}}\in\text{int}({\cal X}):\forall i\in [m], (g _i(\bar{{\bf x}})< 0)\vee (g _i(\bar{{\bf x}})=0\wedge g _i \text{ affine}) \]

（这个条件是在说明解存在吗？）
基于 Slater's condition，叙述拉格朗日函数的鞍点 saddle point 是带约束优化问题的解的充要条件

Th. Saddle point - sufficient condition
带约束优化问题，如果其拉格朗日函数存在鞍点 saddle point \(({\bf x} ^ *, \boldsymbol{\alpha} ^ *)\)，即

\[\forall {\bf x}\in {\cal X},\forall{\boldsymbol{\alpha}\ge 0},\quad {\cal L}({\bf x} ^ *, {\boldsymbol{\alpha}})\le {\cal L}({\bf x} ^ *, {\boldsymbol{\alpha}} ^ *)\le {\cal L}({\bf x}, {\boldsymbol{\alpha}} ^ *) \]

则 \({\bf x} ^ *\) 是该问题的解，\(f({\bf x} ^ *)=\inf f({\bf x})\)

Th. Saddle point - necessary condition
假设 \(f, g _i, i\in [m]\) 为 convex function：

• 若满足 Slater's condition，则带约束优化问题的解 \(\bf x ^ *\) 满足存在 \(\boldsymbol{\alpha} ^ *\ge 0\) 使得 \(({\bf x} ^ *, \boldsymbol{\alpha} ^ *)\) 是拉格朗日函数的鞍点
• 若满足 weak Slater's condition 且 \(f, g _i\) 可导，则带约束优化问题的解 \(\bf x ^ *\) 满足存在 \(\boldsymbol{\alpha} ^ *\ge 0\) 使得 \(({\bf x} ^ *, \boldsymbol{\alpha} ^ *)\) 是拉格朗日函数的鞍点

由于书本上没提供必要性的证明，且充分性证明不难但是不够漂亮，所以就不抄了，只给出我自己的思路（虽然可能有缺陷，下图也只是示意）：

回到最初的不等式：

\[d ^ * = \sup _{\boldsymbol{\alpha}} \inf _{\bf x} {\cal L}({\bf x}, {\boldsymbol{\alpha}})\le\inf _{\bf x} \sup _{\boldsymbol{\alpha}}{\cal L}({\bf x}, {\boldsymbol{\alpha}}) = p ^ * \]

定义 \({\bf x} ^ *\) 取到 \(p ^ *=\sup _{\boldsymbol{\alpha}}{\cal L}({\bf x} ^ *, {\boldsymbol{\alpha}})\le \sup _{\boldsymbol{\alpha}}{\cal L}({\bf x}, {\boldsymbol{\alpha}}),\ \forall {\bf x}\)（可能有多个）
定义 \({\boldsymbol{\alpha}} ^ *\) 取到 \(d ^ *=\inf _{\bf x} {\cal L}({\bf x}, {\boldsymbol{\alpha} ^ *})\ge \inf _{\bf x} {\cal L}({\bf x}, {\boldsymbol{\alpha}}),\ \forall {\boldsymbol{\alpha}}\)（可能有多个）
（鞍点）唯一性和函数凸性有关（虽然感觉不严格凸的话可以是一片“平”的区域），留给之后再说吧
试证明：

\(\cal L\) 存在鞍点、存在一组 \(({\bf x} ^ *, {\boldsymbol{\alpha}} ^ *)\) 是鞍点、\(p ^ * = d ^ *\)、\(p ^ * = d ^ * ={\cal L}({\bf x} ^ *, {\boldsymbol{\alpha}} ^ *)\) 四者等价

分别记为命题 \(A,B,C,D\)，显然有 \(B\to A, D\to C\)

证明 \(A\to B, B\to CD\)：
若存在鞍点（思路见图右上），记鞍点为 \(({\bf x}' , {\boldsymbol{\alpha}}')\)，则 \({\cal L}({\bf x}', {\boldsymbol{\alpha}} ^ *){\color{green}\le} {\cal L}({\bf x}', {\boldsymbol{\alpha}}')=\inf _{\bf x}{\cal L}({\bf x}, {\boldsymbol{\alpha}}'){\color{green}\le} \inf _{\bf x} {\cal L}({\bf x}, {\boldsymbol{\alpha} ^ *})\)，观察不等式两头则取等；进而有 \(\inf _{\bf x}{\cal L}({\bf x}, {\boldsymbol{\alpha}} ^ *)\le {\cal L}({\bf x}', {\boldsymbol{\alpha}} ^ *)= \inf _{\bf x}{\cal L}({\bf x}, {\boldsymbol{\alpha}}')\)，根据 \(\boldsymbol{\alpha} ^ *\) 定义该不等式取等，于是可以令 \(\boldsymbol{\alpha} ^ *\leftarrow\boldsymbol{\alpha}'\)；接下来对 \(\bf x ^ *,x'\) 同理
此时 \(({\bf x} ^ * , {\boldsymbol{\alpha}} ^ *)\) 是鞍点，即 \(\forall {\bf x},\forall {\boldsymbol{\alpha}},\ {\cal L}({\bf x} ^ *, {\boldsymbol{\alpha}})\le {\cal L}({\bf x} ^ *, {\boldsymbol{\alpha}} ^ *)\le {\cal L}({\bf x}, {\boldsymbol{\alpha}} ^ *)\)，则有 \(p ^ * = \sup _{\boldsymbol{\alpha}}{\cal L}({\bf x} ^ *, {\boldsymbol{\alpha}}) = {\cal L}({\bf x} ^ *, {\boldsymbol{\alpha}} ^ *) = \inf _{\bf x} {\cal L}({\bf x}, {\boldsymbol{\alpha} ^ *}) = d ^ *\)

证明 \(C\to BD\)：
若 \(p ^ * =d ^ *\)，即 \(\sup _{\boldsymbol{\alpha}}{\cal L}({\bf x} ^ *, {\boldsymbol{\alpha}})=\inf _{\bf x} {\cal L}({\bf x}, {\boldsymbol{\alpha} ^ *})\)，且 \(\sup _{\boldsymbol{\alpha}}{\cal L}({\bf x} ^ *, {\boldsymbol{\alpha}})\ge {\cal L}({\bf x} ^ *, {\boldsymbol{\alpha} ^ *})\ge \inf _{\bf x} {\cal L}({\bf x}, {\boldsymbol{\alpha} ^ *})\)；故三者取等，故 \(\forall {\bf x},\forall {\boldsymbol{\alpha}},\ {\cal L}({\bf x} ^ *, {\boldsymbol{\alpha}})\le \sup _{\boldsymbol{\alpha}}{\cal L}({\bf x} ^ *, {\boldsymbol{\alpha}})={\cal L}({\bf x} ^ *, {\boldsymbol{\alpha} ^ *})=\inf _{\bf x} {\cal L}({\bf x}, {\boldsymbol{\alpha} ^ *})\le {\cal L}({\bf x}, {\boldsymbol{\alpha} ^ *})\)，从而 \(({\bf x} ^ * , {\boldsymbol{\alpha}} ^ *)\) 是鞍点

综上得证。爽！

KKT conditions

Lagrangian version

若带约束优化问题满足 convexity，我们就可以用一个定理解决：KKT

Th. Karush-Kuhn-Tucker's theorem
假设 \(f, g _i:{\cal X}\to\mathbb{R},\forall i\in[m]\)，为 convex and differentiable，且满足 Slater's condition；则带约束优化问题

\[\min _{\bf x\in\cal X, {\bf g}({\bf x})\le {\bf 0}} f({\bf x}) \]

其拉格朗日函数为 \({\cal L}({\bf x}, {\boldsymbol{\alpha}})=f({\bf x}) + {\boldsymbol{\alpha}}\cdot {\bf g}({\bf x}), {\boldsymbol{\alpha}}\ge 0\)
则 \(\bar{\bf x}\) 是该问题的解，当且仅当存在 \(\bar{\boldsymbol{\alpha}}\ge 0\)，满足：

\[\begin{aligned} \nabla _{\bf x} {\cal L}(\bar{\bf x}, \bar{\boldsymbol{\alpha}}) &= \nabla _{\bf x} f(\bar{\bf x}) + \bar{\boldsymbol{\alpha}}\cdot \nabla _{\bf x} {\bf g}(\bar{\bf x}) = 0 \\ \nabla _{\boldsymbol{\alpha}} {\cal L}(\bar{\bf x}, \bar{\boldsymbol{\alpha}})&={\bf g}(\bar{\bf x})\le 0 \\ \bar{\boldsymbol{\alpha}}\cdot {\bf g}(\bar{\bf x}) &= 0 \end{aligned}\quad ;\text{KKT conditions} \]

其中后两条称为互补条件 complementarity conditions，即对任意 \(i\in[m], \bar{\boldsymbol{\alpha}} _i\ge 0,g _i(\bar{\bf x})\le 0\)，且满足 \(\bar{\boldsymbol{\alpha}} _i g _i(\bar{\bf x})=0\)

充要性证明：

必要性，\(\bar{\bf x}\) 为解，则存在 \(\bar{\boldsymbol{\alpha}}\) 使得 \((\bar{\bf x}, \bar{\boldsymbol{\alpha}})\) 为鞍点，从而得到 KKT 条件：第一条即鞍点定义，第二、三条：

\[\begin{aligned} \forall {{\boldsymbol{\alpha}}},{\cal L}(\bar{\bf x}, {\boldsymbol{\alpha}})\le{\cal L}(\bar{\bf x}, \bar{\boldsymbol{\alpha}})&\implies{\boldsymbol{\alpha}}\cdot {\bf g}(\bar{\bf x})\le \bar{\boldsymbol{\alpha}}\cdot {\bf g}(\bar{\bf x})\\{\boldsymbol{\alpha}}\to +\infty&\implies {\bf g}(\bar{\bf x})\le 0 \\ {\boldsymbol{\alpha}}\to 0 &\implies \bar{\boldsymbol{\alpha}}\cdot{\bf g}(\bar{\bf x})=0 \end{aligned} \]

充分性，满足 KKT 条件，则对于满足 \(\bf g({\bf x})\le 0\) 的 \(\bf x\)：

\[\begin{aligned} f({\bf x}) - f(\bar{\bf x}) &\ge \nabla _{\bf x} f(\bar{\bf x})\cdot ({\bf x-\bar{x}}) & ;\text{convexity of $f$} \\ &= -\bar{\boldsymbol{\alpha}}\cdot \nabla _{\bf x} {\bf g}(\bar{\bf x})\cdot ({\bf x-\bar{x}}) &; \text{first cond}\\ &\ge -\bar{\boldsymbol{\alpha}}\cdot ({\bf g}({\bf x})-{\bf g}(\bar{\bf x})) &; \text{convexity of $g$}\\ &= -\bar{\boldsymbol{\alpha}}\cdot {\bf g}({\bf x}) \ge 0 &;\text{third cond} \end{aligned} \]

Gradient descent version

我们还能用另一种思路阐述 KKT 定理，并且也是另一种求解带约束优化问题的方法：梯度下降
由于 \(\bf g\) 均为凸函数，显然 \({\cal K}=\{\bf x:g(x)\le 0\}\) 为凸集；因此约束其实就是把取值限定在凸集 \(\cal K\) 上，于是有：

Th. Karush-Kuhn-Tucker's theorem, gradient descent version
假设 \(f\) 为 convex and differentiable，\(\cal K\) 为凸集；则带约束优化问题

\[\min _{\bf x\in\cal K} f({\bf x}) \]

则 \(\bf x ^ *\) 是该问题的解，当且仅当

\[\forall {\bf y}\in{\cal K},\quad -\nabla f({\bf x ^ *}) ^\top ({\bf y-x ^ *}) \le 0 \]

其思想为，负梯度方向为 \(f\) 函数值下降的方向，若其与 \(\bf y-x ^ *\) 有相同方向的分量（内积大于零），则可以沿该方向移动，则 \(\bf x ^ *\) 不会是最优点
这个定理为我们的梯度下降算法提供了基础

Gradient descent

Unconstrained case

无约束凸优化问题的梯度下降 GD 算法
记 \(\nabla _t=\nabla f({\bf x} _t),h _t=f({\bf x} _t)-f({\bf x} ^ *),d _t = \Vert{\bf x} _t-{\bf x} ^ * \Vert\)

\[\begin{aligned} \hline & \underline{\text{Algorithm. Gradient descent}\qquad\qquad\qquad\qquad\qquad} \\ & \text{Input $T, {\bf x} _0, \{\eta _t\}$} \\ & \text{for $t=0,\dotsb, T-1$ do} \\ & \qquad \text{${\bf x} _{t+1} = {\bf x} _t - \eta _t \nabla _t$} \\ & \text{end for} \\ & \text{return $\bar{\bf x}=\text{argmin} _{{\bf x} _t}\{ f({\bf x} _t) \}$} \\ \hline \end{aligned} \]

其中合理地选择 \(\eta _t\)，决定了算法的效率；取 Polyak stepsize：\(\eta _t=\frac{h _t}{\Vert \nabla _t\Vert ^2}\)，有：

Th. Bound for GD with Polyak stepsize
假设 \(\Vert \nabla _t\Vert\le G\)，则

\[f(\bar{\bf x}) - f({\bf x} ^ *)=\min _{0\le t\le T}\{h _t\}\le \min \left\{ \frac{G d _0}{\sqrt{T}},\frac{2\beta d _0 ^2}{T}, \frac{3G ^2}{\alpha T},\beta d _0 ^2(1-\frac{\gamma}{4}) ^T \right\} \]

该定理的证明建立在 \(d _{t+1} ^2\le d _t ^2 - {h _t ^2}/{\Vert \nabla _t\Vert ^2}\) 上，这就感觉很扯淡了，因为用绝对值不等式放缩出来总是反的...
证明算法的效率时，可以用一些 “potential 势差” 函数来刻画，比如刻画势能 \(h _t = f({\bf x} _t)-f({\bf x} ^ *)\) 的降低，考虑势能差 \(h _{t+1}-h _t\)、梯度的范数 \(\Vert\nabla _t\Vert\)；比如到最优点的距离 \(d _t = \Vert {\bf x-x} ^ *\Vert\) 的降低，考虑 \(d _{t+1}-d _t\)；
先给出引理，代入 smooth 和 strong convexity 可以证明；这些式子方便我们后续进行放缩

\[\frac{\alpha}{2}d _t ^2\le h _t\le \frac{\beta}{2}d _t ^2\\ \frac{1}{2\beta}\Vert \nabla _t \Vert ^2\le h _t\le \frac{1}{2\alpha}\Vert \nabla _t \Vert ^2 \]

对于更新式 \({\bf x} _{t+1} = {\bf x} _t - \eta _t \nabla _t\)，如果考虑 \(d _{t+1}-d _t\)，那就两边同减 \(\bf x ^ *\) 并取范数，用三角不等式放缩，但是符号和上文的假设是反的，到此就证不下去了；如果假定 \(d _{t+1} ^2\le d _t ^2 - {h _t ^2}/{\Vert \nabla _t\Vert ^2}\)，那就能证明上面的 bound（注意 \(f(\bar{\bf x})-f({\bf x} ^ * )\le \frac{1}{T}\sum _t h _t\) 可以不等式放缩）
不过，我们可以从 \(h _{t+1}-h _t\) 出发，有：

\[\begin{aligned}h _{t+1}-h _t &= f({\bf x} _{t+1})-f({\bf x} _{t})\\ &\le \nabla f({\bf x} _t) ^\top({\bf x} _{t+1} - {\bf x} _{t})+\frac{\beta}{2}\Vert {\bf x} _{t+1}-{\bf x} _{t}\Vert ^2 \\ &= -\eta _t\Vert \nabla _t\Vert ^2+\frac{\beta}{2}\eta _t ^2\Vert \nabla _t\Vert ^2\\ &=-\frac{1}{2\beta}\Vert \nabla _t \Vert ^2\qquad ;\text{令 }\eta _t=\frac{1}{\beta} \\ &\le -\frac{\alpha}{\beta}h _t=-\gamma h _t\end{aligned} \]

于是 \(h _{T}\le (1-\gamma) h _{T-1}\le (1-\gamma) ^T h _0\le e ^{-\gamma T} h _0\)，这倒是个很不错的收敛保证，因此有

Th. Bound for GD, unconstrained case
假设 \(f\) 是 \(\gamma\)-well-conditioned，令 \(\eta _t=\frac{1}{\beta}\)，则

\[h _{T}\le e ^{-\gamma T} h _0 \]

Constrained case

带约束优化的梯度下降，只需要每次移动后，投影到凸集

\[\begin{aligned} \hline & \underline{\text{Algorithm. Basic gradient descent}\qquad\qquad\qquad\qquad\qquad} \\ & \text{Input $T, {\bf x} _0\in{\cal K}, \{\eta _t\}$} \\ & \text{for $t=0,\dotsb, T-1$ do} \\ & \qquad {\bf x} _{t+1} = \Pi _{\cal K}({\bf x} _t - \eta _t \nabla _t) \\ & \text{end for} \\ & \text{return $\bar{\bf x}=\text{argmin} _{{\bf x} _t}\{ f({\bf x} _t) \}$} \\ \hline \end{aligned} \]

它有个类似的上限：
Th. Bound for GD, constrained case
假设 \(f\) 是 \(\gamma\)-well-conditioned，令 \(\eta _t=\frac{1}{\beta}\)，则

\[h _{T}\le e ^{-\gamma T/4} h _0 \]

证明
首先是投影的定义

\[\begin{aligned} {\bf x} _{t+1} &= \Pi _{\cal K}({\bf x} _t - \eta _t \nabla _t) \\ &= \text{argmin} _{\bf x} \Vert {\bf x} - {\bf x} _t + \eta _t \nabla _t \Vert \\ &= \text{argmin} _{\bf x}( \nabla _t ^\top ({\bf x - x} _t) + \frac{1}{2\eta _t}\Vert {\bf x - x} _t\Vert ^2 ) \end{aligned} \]

根据如下式子，可以令 \(\eta _t=\frac{1}{\beta}\)；从而有：

\[\begin{aligned} h _{t+1}-h _t &= f ({\bf x} _{t+1}) - f ({\bf x} _t) \\ &\le \nabla _t ^\top ({\bf x} _{t+1} - {\bf x} _t) + \frac{\beta}{2}\Vert {\bf x} _{t+1} - {\bf x} _t\Vert ^2 \\ &= \text{min} _{\bf x} (\nabla _t ^\top ({\bf x} - {\bf x} _t) + \frac{\beta}{2}\Vert {\bf x} - {\bf x} _t\Vert ^2) \end{aligned} \]

为了摘掉 \(\min\)，我们可以代入某个 \(\bf x\)，代入哪个呢？可以考虑两点的连线，即 \((1-\mu){\bf x} _t + \mu {\bf x} ^ *\)：

\[\begin{aligned} h _{t+1}-h _t &\le \text{min} _{\bf x\in [{\bf x} _t, {\bf x} ^ * ]} (\nabla _t ^\top ({\bf x} - {\bf x} _t) + \frac{\beta}{2}\Vert {\bf x} - {\bf x} _t\Vert ^2) \\ &\le \mu \nabla _t ^\top ({\bf x} ^ * - {\bf x} _t) +\mu ^2 \frac{\beta}{2}\Vert {\bf x} ^ * - {\bf x} _t \Vert ^2 \\ &\le -\mu h _t + \mu ^2 \frac{\beta-\alpha}{2}\Vert {\bf x} ^ * - {\bf x} _t \Vert ^2\quad; \alpha\text{-strong convex}\\ &\le -\mu h _t + \mu ^2\frac{\beta - \alpha}{\alpha}h _t \quad;\text{Lemma} \end{aligned} \]

对 \(\mu\in[0,1]\) 取极值，得到

\[h _{t+1}\le h _t(1-\frac{\alpha}{4(\beta-\alpha)})\le h _t(1-\frac{\gamma}{4})\le h _t e ^{-\gamma/4} \]

从而得证

GD: Reductions to non-smooth and non-strongly convex functions

现在来考虑梯度下降对不一定 smooth、或不一定 strong convex 的凸函数时该怎么分析；下文提到的 reduction 方法可以导出近似最优的收敛速度，而且很简单、很普适

Case 1. reduction to smooth, non-strongly convex functions

考虑仅有 \(\beta\)-smooth 情况；不过实际上，凸函数都是 \(0\)-strongly convex
做法为，加一个适当的 strongly-convex 函数，将原函数扳成更加 strongly-convex 的

\[\begin{aligned} \hline & \underline{\text{Algorithm. Gradient descent, reduction to $\beta$-smooth functions}\qquad\qquad\qquad} \\ & \text{Input $f,T, {\bf x} _0\in{\cal K}$, parameter $\tilde{\alpha}$} \\ & \text{Let $g({\bf x})=f({\bf x}) + \frac{\tilde{\alpha}}{2}\Vert {\bf x-x} _0\Vert ^2$} \\ & \text{Apply GD on $g,T,\{\eta _t=\frac{1}{\beta}\},{\bf x} _0$} \\ \hline \end{aligned} \]

取 \(\tilde{\alpha}=\frac{\beta\log T}{D ^2 T}\) 时，这个算法的效率为 \(h _T=O(\frac{\beta \log T}{T})\)；对 GD 多加处理可以做到 \(O(\beta/T)\)

由于 \(f\) 是 \(\beta\)-smooth 和 \(0\)-strongly convex，加上一个 \(\tilde{\alpha}\)-smooth、\(\tilde{\alpha}\)-strongly convex 的 \(\Vert {\bf x- x} _0\Vert ^2\)，由上述提到的凸函数求和的定理，\(g\) 为 \((\beta+\tilde{\alpha})\)-smooth 和 \(\tilde{\alpha}\)-strongly convex
因此，对于 \(f\)，有：

\[\begin{aligned}h _t &= f({\bf x} _t) - f({\bf x} ^ * ) \\ &= g({\bf x} _t) - g({\bf x} ^ * ) + \frac{\tilde{\alpha}}{2}(\Vert {\bf x} ^ * - {\bf x} _0\Vert ^ 2 - \Vert {\bf x} _t - {\bf x} _0\Vert ^2 ) \\ &\le h _0 ^g \exp ^{-\tilde{\alpha} t/4(\tilde{\alpha}+\beta)} + \tilde{\alpha}D ^2\qquad;\text{$D$ is diameter of bounded $\cal K$} \\ &=O(\frac{\beta \log t}{t}) \qquad;\text{choosing $\tilde{\alpha}=\frac{\beta\log t}{D ^2 t}$, ignore some constants} \end{aligned} \]

Case 2. reduction to strongly convex, non-smooth functions

考虑仅有 \(\alpha\)-strongly convex 的情况，考虑将其改造得更平缓的方法——平滑操作
最简单的平滑操作就是邻域取平均，记 \(f\) 平滑后的函数为 \(\hat{f}_{\delta}:\mathbb{R} ^d\to\mathbb{R}\)，记 \(\mathbb{B}=\{{\bf v}:\Vert \bf v\Vert\le 1\}\)，取半径为 \(\delta\) 的球域做平均，用期望表示为：

\[\hat{f}_{\delta}({\bf x}) = \mathbb{E} _{{\bf v}\sim U(\mathbb{B})}[f({\bf x+\delta v})] \]

这种平滑方法具有如下性质，假设 \(f\) 是 \(G\)-Lipschitz 连续：

• 若 \(f\) 是 \(\alpha\)-strongly convex，则 \(\tilde{f} _\delta\) 也是 \(\alpha\)-strongly convex
• \(\hat{f} _\delta\) 是 \((dG/\delta)\)-smooth
• 任意 \(\bf x\in\cal K\)，\(|\hat{f} _\delta({\bf x}) - f({\bf x})|\le \delta G\)

证明：
第一条利用凸函数的线性组合，对于 \(\hat{f}_{\delta}({\bf x})=\int _{\bf v}\Pr[{\bf v}]f({\bf x+\delta v}){\rm d}{\bf v}\)，由于任意 \(\bf v\)，函数 \(f({\bf x+\delta v})\) 都是 \(\alpha\)-strongly convex 的，因此考察强凸性时，直接提出得到 \(\alpha\int _{\bf v}\Pr[{\bf v}]{\rm d}{\bf v}=\alpha\)；同时可见即使不是均匀分布，依然是这个结论
第二条利用斯托克斯公式，由于是均匀分布，可以将其转化为球面 \(\mathbb{S}=\{{\bf v:\Vert v\Vert}=1\}\) 上的积分

\[\mathbb{E} _{\bf v\sim\mathbb{S}}[f({\bf x+\delta v}){\bf v}]=\frac{\delta}{d}\nabla \hat{f} _\delta({\bf x}) \]

再利用 \(\Vert \nabla f({\bf x})-\nabla f({\bf y})\Vert \le \Vert{\bf x-y}\Vert\) 可证明 smoothness，其步骤和第三条的证明类似
第三条的证明为：

\[\begin{aligned}|\hat{f} _\delta({\bf x}) - f({\bf x})| &= \Big|\mathbb{E} _{{\bf v}\sim U(\mathbb{B})}[f({\bf x+\delta v})]-f({\bf x})\Big| \\ &\le \mathbb{E} _{{\bf v}\sim U(\mathbb{B})}\Big[|f({\bf x+\delta v})-f({\bf x})|\Big]&;\text{Jensen} \\ &\le \mathbb{E} _{{\bf v}\sim U(\mathbb{B})}\Big[ G\Vert\delta {\bf v}\Vert \Big]&;\text{Lipschitz} \\ &\le G\delta \end{aligned}\]

从而算法为：

\[\begin{aligned} \hline & \underline{\text{Algorithm. Gradient descent, reduction to non-smooth functions}\qquad\qquad\qquad} \\ & \text{Input $f,T, {\bf x} _0\in{\cal K}$, parameter $\delta$} \\ & \text{Let $\hat{f}_{\delta}({\bf x}) = \mathbb{E} _{{\bf v}\sim U(\mathbb{B})}[f({\bf x+\delta v})]$} \\ & \text{Apply GD on $g,T,\{\eta _t=\delta\},{\bf x} _0$} \\ \hline \end{aligned} \]

取 \(\delta=\frac{dG}{\alpha}\frac{\log t}{t}\) 时，该算法的效率为 \(h _T=O(\frac{G ^2 d\log t}{\alpha t})\)

暂不考虑如何计算 \(\hat{f} _\delta\) 的梯度，后面会给出估计方法
首先 \(\hat{f} _{\delta}\) 是 \(\frac{\alpha\delta}{dG}\)-well-conditioned

\[\begin{aligned}h _t &=f({\bf x} _t)-f({\bf x} ^ * ) \\ &\le \hat{f} _\delta({\bf x} _t) - \hat{f} _\delta({\bf x} ^ * ) + 2\delta G &;\text{for }|\hat{f} _\delta({\bf x}) - f({\bf x})|\le \delta G \\ &\le h _0 e ^{-\frac{\alpha\delta t}{4dG}} + 2\delta G \\ &=O(\frac{d G ^2\log t}{\alpha t}) &;\delta=\frac{dG}{\alpha}\frac{\log t}{t}\end{aligned} \]

另外，如果在原函数 \(f\) 直接做 GD 的话，依然有收敛保证，但是我们需要取序列的加权和：
令 \(\eta _t=\frac{2}{\alpha(t+1)}\)，得到迭代序列 \({\bf x} _1,\dotsb, {\bf x} _t\)，则

\[f\left( \frac{1}{t}\sum _{k=1} ^t\frac{2k}{t+1}{\bf x} _k \right) - f({\bf x} ^ * )\le \frac{2 G ^2}{\alpha(t+1)} \]

证明略

Case 3. reduction to general convex functions (non-smooth, non-strongly convex)

如果同时使用上述两个方法，会得到一个 \(\tilde{O}(d/\sqrt{t})\) 的方法，不过它得依赖 \(d\)
在 OCO 问题中会给出一个 \(O(1/\sqrt{t})\) 更一般算法

Fenchel duality

凸优化问题，对 \(f\) 不可导或者有无穷大的值的情况进行分析