【SI152笔记】part2：线性规划

SI152: Numerical Optimization

Lec 3. Linear Programming

General form of LP

\[\begin{aligned} \min_{x}~ & c^T x \\ \text{s.t. }~ & a_i^T x \geq b_i , i \in M_1\\ & a_i^T x \leq b_i , i \in M_2\\ & a_i^T x = b_i , i \in M_3\\ & x_j \geq 0, j \in N_1 \\ & x_j \leq 0, j \in N_2 \end{aligned} \]

Standard form

\[\begin{aligned} \min_{x}~ & c^T x \\ \text{s.t. }~ & Ax = b \\ & x \geq 0 \end{aligned} \]

Basic Concept

Definition 1 (Polyhedron):

A polyhedron is a set that can be described in the form \(\{x \in\mathbb{R}^n | Ax \geq b\}\),
where \(A \in\mathbb{R}^{m\times n}\) matrix and \(b\in \mathbb{R}^n\).

Definition 2 (Bounded set)

A set $S \subset \mathbb{R}^n $ is bounded if there exists a constant \(K\) such that \(|x| \leq K\) for all \(x\in S\).

Definition 3

Let a be a nonzero vector in \(\mathbb{R}^n\) and let \(b\) be a scaler.

  • The set \(\{x \in\mathbb{R}^n | a^T x = b\}\) is called a hyperplane.
  • The set \(\{x \in\mathbb{R}^n | a^T x \geq b\}\) is called a halfspace.
Definition 4

A set $S \subset \mathbb{R}^n $ is convex if for any \(x, y \in S\), and any \(\lambda\in[0, 1]\), we have \(\lambda x + (1-\lambda) y \in S\).

Definition 5 (Extreme points)

Let \(P\) be a convex set. A vector \(x \in P\) is an extreme point of P if we cannot find two vectors \(y, z \in P\), both different from \(x\), and a scalar \(\lambda\in[0, 1]\), such that \(x = \lambda y + (1 − \lambda)z\).

Definition 6

Let \(P\) be a polyhedron. A vector \(x\in P\) is a vertex of \(P\) if there exists some \(c\) such that \(c^T x < c^T y\) for all \(y\) satisfying \(y ∈ P\) and \(y \neq x\).

Solution

At a given point \(x^*\), constraints (e.g. \(a^T x \leq b\)) can be classified:

  • active constraints: \(a^T x^* = b\)
  • inactive constraints: \(a^T x^* < b\)
  • violated constraints: \(a^T x^* > b\)

Basic solution: (1) All equality constaints are active. (2) \(n\) linearly independent active constaints.
Basic feasible solution: a basic solution that satisfies all of the constaints.

In fact, in LP, Vertex \(\iff\) Extreme points \(\iff\) Basic feasible solution.

Adjacent basic solutions: two basic solutions sharing \(n − 1\) basis.
Degenerate basic feasible solutions: more than \(n\) constraints are active.

Classify the problem by solution:

  • Infeasible: Doesn’t have optimal solution (like constrains unsatisfied)
  • Feasible:
    • Unbounded: unbounded optimal solution (\(x\to\infty\))
    • Bounded:
      • No extreme point:
        • multiple solution (all \(x\) in a hyperplane)
      • Extreme point exist:
        • multiple solution & unique solution:

Based on Full rank assumption (\(m < n\), and the rows are linearly independent):

  • let \(B\) be the matrix of \(m\) linearly independent columns, and the rest is \(N\).
  • The solution of \(Bx_B = b, x_N = 0\) is a basic solution.
  • If the solution of \(Bx_B = b, x_N = 0\) is nonnegative, it is a basic feasible solution.
  • \(B\) is called the basic matrix; \(N\) is the nonbasic matrix.
  • The variables in \(x_B\) are the basic variables. The variables in \(x_N\) are the nonbasic variables.
Theorem 7

Consider the standard form \(\{x | Ax = b, x\geq 0\}, A \in\mathbb{R}^{m\times n}, rank(A) = m\). If the problems is feasible, then there exists a basic feasible solution.

Theorem 8

Consider the standard form \(\{x | Ax = b, x\geq 0\}, A \in\mathbb{R}^{m\times n}, rank(A) = m\). If the problems is feasible, then there exists an optimal basic feasible solution.

Lec 4. Simplex Method

Introduction

  • Applicable form: standard form
  • Basic idea: start from a BFS, move consecutively to the adjacent BFS,
    until hit the optimal solution, or determine the infeasibility.

For \(Bx_B + Nx_N = b\), i.e. \(x_B = B^{-1}b - B^{-1}Nx_N\).

$f = c^T x = c_B^T x_B + c_N^T x_N = c_B^T B^{-1}b + (c_N^T - c_B^T B^{-1} N)^Tx_N = c_B^T B^{-1}b + r^T x_N $

Theorem 1 (Optimality Criterion)

At a BFS, if \(r \geq 0\), then it is optimal.

Pivot: \(x_q\) enter the basis and \(x_p\) leaves the basis, obtain a new BFS.
From current BFS to an adjacent BFS, and the objective decreases.

Simplex Tableau:

\(x_B\) \(x_N\) \(\bar{b}\)
\(Y\) \(B\) \(N\) \(b\)
\(r\) \(c_B^T\) \(c_N^T\) \(0\)

To

\(x_B\) \(x_N\) \(\bar{b}\)
\(Y\) \(I\) \(B^{-1}N\) \(B^{-1}b\)
\(r\) \(0\) \(c_N^T - c_B^T B^{-1}N\) \(-f\)

Simplex Method:

  1. Step 1: Create the first tableau associated with the initial BFS.
  2. Step 2: If \(r_j \geq 0, \forall j\), stop; current BFS is optimal.
  3. Step 3: Choose \(r_q = \min\{r_j | r_j < 0, j = 1, ..., n\}\).
  4. Step 4: If \(y_q \leq 0\), stop. The problem is infinite. Otherwise, choose \(p\)

    \[\dfrac{\bar{b}_p}{y_{p,q}} = \min\{ \dfrac{\bar{b}_i}{y_{i,q}} | y_{i,q} > 0, i = 1, ..., m\} \]

  5. Step 5: Pivot using \(y_{p,q}\) (i.e. choose by Bland’s rule). Update the tableau. Go to Step 1

Convergence and Degeneracy

Degenerate BFS \(\implies\) degenerate pivot\(\implies\) objective the same\(\implies\) cycling

Bland’s rule:
  • Rule for entering the basis: choose the smallest index with a negative (reduced) cost
  • Rule for leaving the basis: choose the smallest ratio (smallest index if there is a tie)

Complexity

Klee-Minty Problem: TODO. need \(2^n -1\) pivots.

The simplex method:

  • Iteration: \(O(2^n)\) (worst-case), \(O(n^3)\) (average)
  • Pivot: \(O(mn)\) (naive), \(O(m^2)\) (efficient)
  • Total: \(O(m^2\cdot 2^n)\) (worst-case), \(O(n^3 m^2)\) (average)

Initialization

Adding auxiliary variables: detect whether \(Ax = b, x\geq 0\) is feasible, if it is, remove the redundant constraint, find the initial BFS.

solve:

\[\begin{aligned} \min_{x,y}~ & f = \sum y_i \\ \text{s.t. }~ & Ax +y = b \\ & x \geq 0, y\geq 0 \end{aligned} \]

  • Starting from \(x = 0, y = b\) as the initial BFS, use the simplex method to find an optimal solution \((x_0 , y_0 )\), optimal value \(f_0\), optimal basis \(B_0\).
  • \(f_0 > 0\), the original LP is infeasible.
  • \(f_0 = 0\), the original LP is feasible, and \(x_0\) is a potential BFS.
    • no \(y_i\) is in the basis \(\implies\) \(x_0\) is BFS, \(B_0\) is the basis.
    • some \(y_i\) is in the basis \(\implies\) pivot, kick \(y_i\) out of the basis.
    • some \(y_i\) is in the basis with all nonnegative coefficients in this row \(\implies\) remove this redundant row.

Revised simplex method

Lec 5. Duality

Lagrange function (Lagrangian) of \(\min_x c^T x \text{ s.t. } a^T_i x \leq b_i\):

\[ L(x, λ) = c^T x + \sum_{i=1}^{m} λ_i(a^T_i x − b_i) \]

(\(\lambda\): the price per unit of violation that \(a^T_i x > b_i\))

The primal problem

Primal objective: \(f(x) := \max_{\lambda\geq 0} L(x, λ)\). Primal variable: \(x\)
Primal problem (min-max): \(\min_x f(x) = \min_x\max_{\lambda\geq 0} L(x, λ)\).

The dual problem

Dual objective: \(g(\lambda) := \min_x L(x, λ)\). Dual variable: \(\lambda\). Dual feasibility: \(λ ≥ 0\)
Dual problem: \(\max_{\lambda\geq 0} f(x) = \max_{\lambda\geq 0}\min_x L(x, λ)\).

Standard form of LP

The primal problem:

\[\begin{aligned} \min_{x}~ & c^T x \\ \text{s.t. }~ & Ax = b , x \geq 0 \end{aligned} \]

The dual problem:

\[\begin{aligned} \max_{\lambda}~ & b^T \lambda \\ \text{s.t. }~ & A^T \lambda \leq c \end{aligned} \]

Primal Prob. & Dual Prob

Primal Problem Dual Problem
$\min_x c^T x, $ $\max_{\lambda} b^T \lambda, $
$ \text{s.t.} Ax\sim b, x\sim 0 $ $ \text{s.t.} A^T\lambda\sim c, \lambda\sim 0 $
Constraints Variables
\(\sum_{j=1}^n a_{ij}x_j \geq b_i\) \(\lambda_i \geq 0\)
\(\sum_{j=1}^n a_{ij}x_j = b_i\) \(\lambda_i\) is free
\(\sum_{j=1}^n a_{ij}x_j \leq b_i\) \(\lambda_i \leq 0\)
Variables Constraints
\(x_j\) is free \(a_j^T \lambda = c_j\)
\(x_j \geq 0\) \(a_j^T \lambda \leq c_j\)
\(x_j \leq 0\) \(a_j^T \lambda \geq c_j\)
\(x_j = 0\) no constraint

Weak duality & Strong Duality

Theorem 1 (Weak duality): \(c^T x ≥ λ^T b\) for any primal-dual feasible \((x, λ)\).

Corollary 2: If \(c^T x = b^T λ\) holds at primal-dual feasible \((x, λ)\) (duality gap \(= 0\)), then \(x\) is primal optimal and \(λ\) is dual optimal.
Corollary 3: If the primal (dual) problem is unbounded, then the dual (primal) problem is infeasible.

Theorem 4 (Strong Duality): Let \(x^∗\) be the optimal solution of the standard form LP and \(B\) be the optimal basis. Then \(λ^∗ = (c^T_BB^{−1})^T\) is the optimal dual variable.
($ r = c − AT(cT_BB{−1})T = c − A^T B^{−T} c_B = c − A^T λ $)

Theorem 5: If the primal (dual) problem is feasible, then the dual (primal) is also feasible and the optimal values are equal (duality gap \(= 0\)).

Relationship

Another form of the dual problem:

\[\begin{aligned} \max_{\lambda}~ & b^T \lambda \\ \text{s.t. }~ & A^T \lambda + r = c, r\geq 0 \end{aligned} \]

\(c^T x = b^T λ \implies r^T x =0\)

posted @ 2025-01-01 22:07  Coinred  阅读(46)  评论(0)    收藏  举报