BoydC5pt1
The Lagrange dual function
The Lagrangian
Def 1.1. standard form problem (not necessarily convex)
where \(x\in\mathbb{R}^n\), domain \(\mathcal{D}\), optimal value \(p^*\).
Rmk. In fact, since \(h_i(x)=0\) \(\Leftrightarrow\) \(h_i(x)\leq 0\) \(\land\) \(-h_i(x)\leq 0\), the standard form problem can be simplified as
\[\begin{align*} \min_x\quad &f_0(x) \\ s.t.\quad &f_i(x)\leq 0,\quad i\in\left[ m+2n \right]\\ \end{align*} \]where \(f_{m+2i-1}=h_i\) and \(f_{m+2i}=-h_i\)
Define 1.2. Lagrangian \(L:\mathbb{R}^n\times \mathbb{R}^m \times \mathbb{R}^p \to \mathbb{R}\), with \(\mathbf{dom}~L\) \(=\) \(\mathcal{D}\times \mathbb{R}^m \times \mathbb{R}^p\),
\(\lambda_i\), \(\nu_i\) are called Lagrange multiplier.
From rmk of def 1.1, Lagrangian can be written as
\[\tilde{L}(x,\tilde{\lambda}) = f_0(x)+\sum_{i=1}^m\lambda_i f_i(x)+\sum_{i=1}^p (\lambda_{m+2i-1}-\lambda_{m+2i}) h_i(x) \]
For \(\lambda\succeq 0\), if \(x\) does not satisfy the constraints, \(\sup_{\lambda\succeq 0,v} L(x,\lambda,\nu)=\infty\); if \(x\) satisfies the constraints, \(\sup_{\lambda\succeq 0,v} L(x,\lambda,\nu)=f_0(x)\). Thus
\[p^* = \inf_x\sup_{\lambda\succeq 0,v} L(x,\lambda,\nu) = \inf_x\sup_{\tilde{\lambda}\succeq 0} \tilde{L}(x,\tilde{\lambda}) \]
The Lagrange dual function
Def 1.3. Lagrange dual function: \(g:\mathbb{R}^m\times\mathbb{R}^p\to\mathbb{R}\),
\(g\) is concave, and can be \(-\infty\).
Lower bounds on optimal value
Thm 1.1. lower bound property:
Examples
Ex 1.1. Two-way partitioning:
its dual function
The Lagrange dual function and conjugate functions
consider an optimization problem with linear inequality and equality constraints:
its dual function:
Ex 1.2. Equality constrained norm minimization:
dual function
where \(\|v\|_*=\sup_{\|u\|\leq 1}u^Tv\) is the dual norm of \(\|\cdot\|\). \((i)\) holds for ex 3.1 in BoydC3.
Ex 1.3. Entropy maximization:
its dual function
where \(a_i\) is the \(i\)-th colum of \(A\). \((i)\) holds fo ex 3.2 in BoydC3.
The Lagrange dual problem
Def 2.1. Lagrange dual problem:
where optimal value denoted \(d^*\). \(\lambda,\nu\in\mathbf{dom}~g\),
Making dual constraints explicit
we can form an equivalent problem, in which these equality constraints are given explicitly as constraints. The following examples demonstrate this idea.
Ex 2.1. LP :
and
form an equivalent problem by making these equality constraints \((g(\lambda,\nu)> -\infty)\) explicit:
Weak duality
Def 2.2. weak duality: \(d^*\leq p^*\)
- always holds (for convex and nonconvex problems)
Strong duality and Slater’s constraint qualification
Def 2.3. strong duality: \(d^* = p^*\)
- does not hold in general
- (usually, but not always) holds for convex problems
- conditions that guarantee strong duality in convex problems are called constraint qualifications (\(e.g.\) Slater’s condition)
Thm 2.1. Slater’s constraint qualifification:
strong duality holds for a convex problem
if it is strictly feasible, \(i.e.\),
- also guarantees that the dual optimum is attained (if \(p^*> -\infty\))
- can be sharpened: \(e.g.\) can replace \(\mathbf{int}~\mathcal{D}\) with \(\mathbf{reint}~\mathcal{D}\). If \(f_k\) is affine, \(f_k(x)\leq 0\) is enough (linear inequalities do not need to hold with strict inequality).
Ex 2.2. Inequality form LP (\(\exists~x~s.t.~Ax\prec b\)), Quadratic program (always) \(p^*=d^*\).
nonconvex example
\[\begin{align*} \min \quad &x^T Px\\ s.t. \quad &Ax\preceq b \end{align*} \]
Next part: BoydC5pt2