BoydC5pt3

---

Previous part: BoydC5pt2

---

Optimality conditions

Certificate of suboptimality and stopping criteria

Since \(f_0(x)-p^*\leq f_0(x)-g(\lambda,\nu)\)

\[f_0(x^{(k)})-g(\lambda^{(k)},v^{(k)})\leq \varepsilon_{\text{abs}} \]

can be used as nonheuristic stopping criteria. A similar condition can be used to guarantee a given relative accuracy \(\varepsilon_{\text{rel}}\).

Complementary slackness

Thm 5.1. assume strong duality holds, \(x^*\) is primal optimal, \((\lambda^*,\nu^*)\) is dual optimal

\[\begin{aligned} f_0(x^*) = g(\lambda^*, \nu^*) &= \inf_x \left( f_0(x) + \sum_{i=1}^m \lambda_i^* f_i(x) + \sum_{i=1}^p \nu_i^* h_i(x) \right) \\ &\leq f_0(x^*) + \sum_{i=1}^m \lambda_i^* f_i(x^*) + \sum_{i=1}^p \nu_i^* h_i(x^*) \\ &\leq f_0(x^*) \end{aligned} \]

hence, the two inequalities hold with equality, \(x^*\) minimizes \(L(x,\lambda^*,\nu^*)\), \(\lambda_i f_i(x^*)=0\) (complementary slackness).

KKT optimality conditions

Thm 5.2. the following 4 conditions are called KKT conditions (differentiable \(f_i,h_i\)), \(\exists~\lambda,\nu\) \(s.t.\)

1. primal constraints: \(f_i(x)\leq 0\), \(h_i(x)=0\)
2. dual constraints: \(\lambda\succeq 0\)
3. complementary slackness: \(\lambda_i f_i(x^*)=0\)
4. gradient of Lagrangian w.r.t. \(x\) :

\[\nabla f_0(x) + \sum_{i=1}^{m} \lambda_i \nabla f_i(x) + \sum_{i=1}^{p} \nu_i \nabla h_i(x) = 0 \]

For a convex problem which satisfies Slater's condition, \(\exists~\lambda^*\geq 0,\nu^*\) satisfy KKT \(\Leftrightarrow\) \(x^*\) optimal .

proof.

KKT \(\Rightarrow\) optimal. \(x^*\) is feasible and

\[\begin{aligned} f(x)-f(x^*) &\geq \nabla f(x^*)^T(x-x^*)\\ &=-(\sum_{i=1}^{m} \lambda_i \nabla f_i(x^*) + \sum_{i=1}^{p} \nu_i \nabla h_i(x^*))^T(x-x^*)\\ &\geq -\sum_{i=1}^{m} \lambda_i f_i(x)\geq 0\\ L(x^*,\lambda^*,\nu^*) &=\inf_x L(x,\lambda^*,\nu^*)= f(x^*) =p^*=d^*\\ &=g(\lambda^*,\nu^*)\geq g(\lambda,\nu) \end{aligned} \]

\(\lambda^*,\nu^*\) are also optimal.

\(x^*\) optimal \(\Rightarrow\) KKT.

\(x^*\) optimal \(\Rightarrow\) \(\exist~\lambda^*\geq 0\) \(s.t.\) \(g(\lambda^*,\nu^*)=f(x^*)\) \(=\) \(L(x^*,\lambda^*,\nu^*)\) \(=\) \(p^*=d^*\)

\[\begin{aligned} \nabla_x L(x^*,\lambda^*,\nu^*)=0 &\Rightarrow ~\text{4th cond}\\ x^*~\text{feasible} &\Rightarrow \text{primal constraints}\\ \text{Slater's cond} + \text{thm 5.1} &\Rightarrow \text{complementary slackness}\\ \end{aligned} \]

Perturbation and sensitivity analysis

The perturbed problem

Def 6.1. For unperturbed problem

\[\begin{aligned} \min. \quad& f_0(x) \\ \text{s.t.} \quad& f_i(x) \leq 0, & \quad i = 1, \ldots, m \\ & h_i(x) = 0, & \quad i = 1, \ldots, p \end{aligned} \]

perturbed problem is

\[\begin{aligned} \min. \quad& f_0(x) \\ \text{s.t.} \quad& f_i(x) \leq u_i, & \quad i = 1, \ldots, m \\ & h_i(x) = v_i, & \quad i = 1, \ldots, p \end{aligned} \]

and its dual

\[\begin{aligned} \max. \quad& g(\lambda, \nu) - u^T \lambda - v^T \nu \\ \text{s.t.} \quad& \lambda \succeq 0 \end{aligned} \]

Def 6.2. define \(p^*(u,v)\) as the optimal value of the perturbed problem :

\[p^*(u,v) = \inf_x \{f_0(x)~|~\mathbf{f}(x)\preceq u,~\mathbf{h}(x)= v\} \]

A global inequality

Thm 6.1. Assume that strong duality holds, and that the dual optimum is attained. Let \((\lambda^*,\nu^*)\) be optimal for the dual of the unperturbed problem, then

\[p^*(u,v)\geq g(\lambda^*, \nu^*) - u^T \lambda^* - v^T \nu^* = p^*(0,0) - u^T \lambda^* - v^T \nu^* \]

Local sensitivity analysis

Thm 6.2.

\[\begin{aligned} \lambda_i &=-\frac{\partial p^*(u,v)}{\partial u_i}|_{u=0,v=0}\\ \nu_i &=-\frac{\partial p^*(u,v)}{\partial v_i}|_{u=0,v=0}\\ \end{aligned} \]

proof

\[\begin{aligned} \frac{\partial p^*(0,0)}{\partial u_i} = \lim_{t \searrow 0} \frac{p^*(te_i,0) - p^*(0,0)}{t} \geq -\lambda_i^*\\ \frac{\partial p^*(0, 0)}{\partial u_i} = \lim_{t \nearrow 0} \frac{p^*(t e_i, 0) - p^*(0, 0)}{t} \leq -\lambda_i^* \end{aligned} \]

---

Next part: BoydC5pt4