BoydC5pt1

---

The Lagrange dual function

The Lagrangian

Def 1.1. standard form problem (not necessarily convex)

\[\begin{align*} \min_x\quad &f_0(x) \\ s.t.\quad &f_i(x)\leq 0,\quad i\in\left[ m \right]\\ & h_i(x) = 0, \quad i\in\left[ n \right] \end{align*} \]

where \(x\in\mathbb{R}^n\), domain \(\mathcal{D}\), optimal value \(p^*\).

Rmk. In fact, since \(h_i(x)=0\) \(\Leftrightarrow\) \(h_i(x)\leq 0\) \(\land\) \(-h_i(x)\leq 0\), the standard form problem can be simplified as

\[\begin{align*} \min_x\quad &f_0(x) \\ s.t.\quad &f_i(x)\leq 0,\quad i\in\left[ m+2n \right]\\ \end{align*} \]

where \(f_{m+2i-1}=h_i\) and \(f_{m+2i}=-h_i\)

Define 1.2. Lagrangian \(L:\mathbb{R}^n\times \mathbb{R}^m \times \mathbb{R}^p \to \mathbb{R}\), with \(\mathbf{dom}~L\) \(=\) \(\mathcal{D}\times \mathbb{R}^m \times \mathbb{R}^p\),

\[L(x,\lambda,\nu) = f_0(x)+\sum_{i=1}^m\lambda_i f_i(x)+\sum_{i=1}^p \nu_i h_i(x) \]

\(\lambda_i\), \(\nu_i\) are called Lagrange multiplier.

From rmk of def 1.1, Lagrangian can be written as

\[\tilde{L}(x,\tilde{\lambda}) = f_0(x)+\sum_{i=1}^m\lambda_i f_i(x)+\sum_{i=1}^p (\lambda_{m+2i-1}-\lambda_{m+2i}) h_i(x) \]

For \(\lambda\succeq 0\), if \(x\) does not satisfy the constraints, \(\sup_{\lambda\succeq 0,v} L(x,\lambda,\nu)=\infty\); if \(x\) satisfies the constraints, \(\sup_{\lambda\succeq 0,v} L(x,\lambda,\nu)=f_0(x)\). Thus

\[p^* = \inf_x\sup_{\lambda\succeq 0,v} L(x,\lambda,\nu) = \inf_x\sup_{\tilde{\lambda}\succeq 0} \tilde{L}(x,\tilde{\lambda}) \]

The Lagrange dual function

Def 1.3. Lagrange dual function: \(g:\mathbb{R}^m\times\mathbb{R}^p\to\mathbb{R}\),

\[g(\lambda,\nu) = \inf_{x\in\mathcal{D}}L(x,\lambda,\nu) \]

\(g\) is concave, and can be \(-\infty\).

Lower bounds on optimal value

Thm 1.1. lower bound property:

\[g(\lambda,\nu)\leq p^*,\quad \lambda\succeq 0 \]

Examples

Ex 1.1. Two-way partitioning:

\[\begin{align*} \min \quad & x^TWx\\ s.t. \quad & x_i^2=1, \quad i\in\left[n\right] \end{align*} \]

its dual function

\[\begin{aligned} g(\nu)&=\inf_x x^TWx+\sum_{i=1}^n \nu_i(x_i^2-1)\\ &=\inf_x x^T(W+\mathbf{diag}(v))x-\mathbf{1}^T\nu\\ &=\begin{cases} -\mathbf{1}^T\nu\quad& W+\mathbf{diag}(\nu)\succeq 0\\ -\infty & \text{otherwise} \end{cases} \end{aligned} \]

The Lagrange dual function and conjugate functions

consider an optimization problem with linear inequality and equality constraints:

\[\begin{align*} \min \quad & f_0(x)\\ s.t. \quad & Ax\preceq b\\ & Cx =d \end{align*} \]

its dual function:

\[\begin{align*} g(\lambda,\nu) &= \inf_x f_0(x)+\lambda^T(Ax-b)+v^T(Cx-d)\\ &=-\sup_x((-\lambda^TA-v^TC)x-f_0(x))-b^T\lambda-d^T\nu\\ &=-f_0^*(-A^T\lambda-C^Tv)-b^T\lambda-d^T\nu \end{align*} \]

Ex 1.2. Equality constrained norm minimization:

\[\begin{aligned} \min \quad &\|x\|\\ s.t.\quad & Ax=b \end{aligned} \]

dual function

\[\begin{align*} g(\nu) &= \inf_x \|x\| + \nu^T(b-Ax)\\ &=b^T\nu-\sup_x (\nu^TAx-\|x\|)\\ &=b^T\nu- (\|A^Tx\|)^*\\ &\overset{(i)}{=} \begin{cases} b^T\nu \quad& \|A^Tx\|_*\leq 1\\ -\infty &\|A^Tx\|_*>1 \end{cases} \end{align*} \]

where \(\|v\|_*=\sup_{\|u\|\leq 1}u^Tv\) is the dual norm of \(\|\cdot\|\). \((i)\) holds for ex 3.1 in BoydC3.

Ex 1.3. Entropy maximization:

\[\begin{align*} \min \quad & f_0(x)=\sum_{i=1}^n x_i\log x_i\\ s.t. \quad & Ax\preceq b\\ & 1^Tx=1 \end{align*} \]

its dual function

\[\begin{align*} g(\lambda,\nu) &=-f^*_0(-A^T\lambda-v\mathbf{1})-b^T\lambda-v\\ &\overset{(i)}{=}-b^T\lambda-v-\sum_{i=1}^n e^{-a_i^T\lambda-v-1} \end{align*} \]

where \(a_i\) is the \(i\)-th colum of \(A\). \((i)\) holds fo ex 3.2 in BoydC3.

The Lagrange dual problem

Def 2.1. Lagrange dual problem:

\[\begin{aligned} \max \quad& g(\lambda,\nu)\\ s.t. \quad& \lambda\succeq 0 \end{aligned} \]

where optimal value denoted \(d^*\). \(\lambda,\nu\in\mathbf{dom}~g\),

\[\mathbf{dom}~g=\{(\lambda,\nu)~|~g(\lambda,\nu)> -\infty\} \]

Making dual constraints explicit

we can form an equivalent problem, in which these equality constraints are given explicitly as constraints. The following examples demonstrate this idea.

Ex 2.1. LP :

\[\begin{align*} \min \quad& c^Tx\\ s.t. \quad& Ax=b\\ & x\succeq 0 \end{align*} \]

and

\[g(\lambda,\nu)=\begin{cases} -b^T\nu\quad& A^T\nu-\lambda+c=0\\ -\infty \quad& \text{otherwise} \end{cases} \]

form an equivalent problem by making these equality constraints \((g(\lambda,\nu)> -\infty)\) explicit:

\[\begin{aligned} \max \quad& -b^T\nu\\ s.t. \quad& A^T\nu-\lambda+c=0\\ &\lambda\succeq 0 \end{aligned}\quad \Rightarrow \quad \begin{aligned} \max \quad& -b^T\nu\\ s.t. \quad& A^T\nu+c\succeq 0\\ \end{aligned} \]

Weak duality

Def 2.2. weak duality: \(d^*\leq p^*\)

• always holds (for convex and nonconvex problems)

Strong duality and Slater’s constraint qualification

Def 2.3. strong duality: \(d^* = p^*\)

• does not hold in general
• (usually, but not always) holds for convex problems
• conditions that guarantee strong duality in convex problems are called constraint qualifications (\(e.g.\) Slater’s condition)

Thm 2.1. Slater’s constraint qualifification:

strong duality holds for a convex problem

\[\begin{align*} \min\quad&f_0(x)\\ s.t.\quad&f_i(x)\leq 0,\quad i\in\left[m\right] &Ax=b \end{align*} \]

if it is strictly feasible, \(i.e.\),

\[\exists ~x\in \mathbf{int}~\mathcal{D}:\quad f_i(x)<0 \quad i\in\left[m\right],\quad Ax=b \]

• also guarantees that the dual optimum is attained (if \(p^*> -\infty\))
• can be sharpened: \(e.g.\) can replace \(\mathbf{int}~\mathcal{D}\) with \(\mathbf{reint}~\mathcal{D}\). If \(f_k\) is affine, \(f_k(x)\leq 0\) is enough (linear inequalities do not need to hold with strict inequality).

Ex 2.2. Inequality form LP (\(\exists~x~s.t.~Ax\prec b\)), Quadratic program (always) \(p^*=d^*\).

nonconvex example

\[\begin{align*} \min \quad &x^T Px\\ s.t. \quad &Ax\preceq b \end{align*} \]

---

Next part: BoydC5pt2