Part Ⅰ: Theory — Basic Concept
Preface
Optimization Object: Things with non-negative curvature
Only One Goal — Transform the actual problem into a convex problem.
- Problem complexity is not continuous in the intuition norm.
Problem Formulation
-
Objective and constraint functions are convex:
\[f_i(\alpha \mathbf{x} + \beta \mathbf{y}) \le \alpha f_i(\mathbf{x}) + \beta f_i(\mathbf{y}) \]if \(\alpha + \beta = 1\), \(\alpha,\beta\ge 0\), which means the problem has a non-negative curvature.
-
Least-Square Problem and Linear Programming are included as special cases.
1. Affine
PS: A development of Linear concept.
Well, although viewing linear space as a set of points is easier for understanding, I recommend to view linear space as a set of vectors, which means the space is spanned by lots of vectors. This will help understanding some information about linear transformation.
It's better having an awareness to combine a point with a vector.
1.1 Affine Set
Def :
A set \(P\in\mathbb{R}^n\), If \(\forall\, x,y \in P\), \(\forall\, \theta \in \mathbb{R}\), it always holds that \(\theta x + (1-\theta)y \in P\).
-
\(\theta x + (1-\theta)y\) represents a line through points of \(x,y\). This shows the set on a line is absolutely an affine set.
Proof:
Let vector \(z = \theta x + (1-\theta)y\)，so
\[z - y = \theta (x - y) \]We can know \(\overrightarrow{zy} = \theta \,\overrightarrow{xy}\).
-
The simplest Affine Set is the Empty Set \(\emptyset\). And, those sets that only contain a single element are affine as well.
Generally, the solution set of \(A\mathbf{x} = b\) is also an affine set.
-
Furtherly, every Affine Set can be expressed as the solution set of a system of linear equations.
Proof:
Assume a set \(C=\{\mathbf{x}\vert A\mathbf{x} = \mathbf{b}\},\,\forall\, \mathbf{x_1,x_2} \in C,\,\forall \, \theta\in \mathbb{R}\), we have
\[\begin{aligned} &A\bf \left[ \theta\, x_1 + \left(1-\theta\right)\,x_2 \right] \\ = &\bf \theta\, b + \left(1-\theta\right)\, b \\ = &\mathbf{b} \end{aligned} \]-
As an Affine Set minus any element in the set, it is still an Affine Set. What's more, it's a Subspace.
Proof:
Assume an affine set \(C=\{\mathbf{x}\vert A\mathbf{x} = \mathbf{b}\}\). For \(\forall\, \mathbf{x_0} \in C,\,\forall \, \theta\in \mathbb{R}\), we have a subspace \(V = C-\mathbf{x_0} = \{\mathbf{x}-\mathbf{x_0}\vert \mathbf{x} \in C\} = \{\mathbf{v}\vert A\mathbf{v} = \mathbf{0}\}\).
Apparently, \(\forall\,\mathbf{v_1,v_2} \in V, \,\mathbf{v_1+v_2} \in V\)
- Through the Proof, for linear equations \(A\mathbf{x} = b\), if we view coifficient matrix \(A\) corresponding to a linear transformation \(\mathcal{A}\), then any subspace \(V=C-x\,(x\in C)\) is the nullspace of \(A\), which means \(\ker{\mathcal{A}} = V\).
-
As an Affine Set plus any element in the set, it is still an Affine Set.
Proof:
Assume an affine set \(C=\{\mathbf{x}\vert A\mathbf{x} = \mathbf{b}\}\). For \(\forall\, \mathbf{x_0} \in C,\,\forall \, \theta\in \mathbb{R}\), we have the subspace \(V = C+\mathbf{x_0} = \{\mathbf{x}+\mathbf{x_0}\vert \mathbf{x} \in C\} = \{\mathbf{v}\vert A\mathbf{v} = 2\mathbf{b}\}\).
-
Conclution:
As an Affine Set given (plus or minus) any element in the set, it is still an Affine Set.
In a way, an Affine Set also can be described as a subspace plus an offset: \(C = V + \mathbf{x}\).
-
-
The dimension of an affine set \(C\) equals with that of sets \(V = C\pm\mathbf x\,(\mathbf x\in C)\), denoted as \(\dim{C} = \dim{V}\).
仿射集 C 的维数，等于它平移得到的线性子空间。
1.2 Affine Combination
Def:
Assume \(m\) elements in the set, like \(x_1, x_2, \cdots, x_m\), and their linear combination form
\[x = \theta_1 x_1 + \theta_2 x_2 + \cdots + \theta_m x_m \]When \(\theta_1 + \theta_2 + \cdots + \theta_m = 1\), the combination is called Affine Combination.
-
Every Affine Set \(\in \mathbb{R}^n\) contains all affine combinations of its elements.
-
For Linear Combination, the combination represents a space spanned by linearly independent vectors.
\[x = k_1 x_1 + k_2 x_2 + \cdots + k_m x_m \in \mathbf{span}(x_1, x_2, \cdots, x_m) \]But for Affine Combination, if linearly independent vectors are \(x_1, x_2, \cdots, x_m\), the combination
\[\begin{aligned} x &= \left( \sum_{n=1,n\ne i}^m\theta_n x_n \right) + \left( 1-\sum_{n=1,n\ne i}^m\theta_n\right) x_i \\ &= x_i + \left[ \sum_{n=1,n\ne i}^m\theta_n (x_n-x_i) \right] \in \mathbf{span}(x_1-x_i, \cdots, x_m-x_i) + x_i \end{aligned} \]represents a space, which contains all points of vectors. Apparently, except \(\theta_i\), other coefficients can be any number, which means the space is spanned by \(m-1\) vectors with \(x_i\) position as the origin.
-
By the way, the space has a translation with \(x_i\).
平移效果等同于 \(x_i\) 到张成空间的距离
-
Another Proof idea is covering space with lines, shown in 2.2
-
-
This also can illustrate the Affine Dimension Problrm in 1.4
1.3 Affine Hull
Def:
For a set \(S\), the set \(\{\theta_1 \alpha_1 + \theta_2 \alpha_2 + \cdots + \theta_n \alpha_n \,\vert\, \theta_i\in \mathbb{R},\sum{\theta_i} = 1, \alpha_i\in S,i=1,2,\cdots,n\}\), which contains all affine combinations of vectors in the set \(S \subseteq \mathbb{R}^n\), is called the Affine Hull of \(S\), denoted as \(\mathbf{aff}\, S\).
-
The Affine Hull \(\mathbf{aff}\, S\) is the smallest Affine Set that contains set \(S\).
- That means if \(C\) is any affine set with \(S\subseteq C\), then \(\mathbf{aff}\, S \subseteq C\).
Furtherly, that also means any affine set is the affine hull of itself.
1.4 Affine Dimension
Def:
The Affine Dimension of a set \(S\) is the dimension of its affine hull \(\mathbf{aff}\, S\).
-
e.g.:
A set is defined by points of two Intersecting lines, so its affine dimension is 2;
A set is defined by pionts on a ring, and its affine dimension is also 2.
1.5 Relative Interior
Def:
The Relative Interior of a set \(S\in \mathbb{R}^n\) is its interior relative to \(\mathbf{aff} S\), denoted as \(\mathbf{relint}\, S\):
\[\mathbf{relint}\, S = \{ x\in S\, \vert \,\forall \, r>0,\,\exists \, B(x,r) \cap \mathbf{aff}\, S \, \subseteq S \} \\ \text{or } \{ x\in S\, \vert \,\exists \, r >0,\, s.t.\, B(x,r) \cap \mathbf{aff}\, S \subseteq S \}, \]where \(B(x,r) = \{y\,\vert\, \Vert y-x\Vert\le r,\,y\in\mathbb{R}^n\}\), representing a closed ball ( \(r>0\) ),
- Based on concept of Relative Interior, We can define the Relative Boundary of a set \(S\) as \(\mathbf{cl}\, S\,\text{\\} \mathbf{relint}\, S\).
2. Convex
PS: Just similar with Affine, but still a little different.
2.1 Convex Set
Def:
A set contains every line segment between any two elements in the set.
-
Difference with Affine
In Affine, "line segment" is replaced by "line", so the parameter \(\theta\) can be any real number.
But in Convex, the parameter \(\theta\) can be only positive real number ( \(0\le \theta \le 1\) ), because points on the line segment can't exceed the end.
The condition of Affine is more strict than that of Convex, so every Affine Set is also a Convex Set, which means line is convex but line segment isn't affine.
-
Any ray is convex, and set with single element is also convex.
2.2 Convex Combination
Def:
An Affine Combination, when every coefficient is not less than 0 ( \(\theta_i \ge 0\) ), then the combination is called Convex Combination.
-
Every Convex Set \(\in \mathbb{R}^n\) contains all convex combinations of its elements.
-
The idea of a convex combination can be generalized to include infinite sums, integrals, and probability distributions (maybe the most general form).
-
A convex combination of vectors can be viewed as a mixture or weighted average of the vectors.
-
e.g.
Expectancy (in Probablity) is a Convex Combination, where \(x_1,x_2,\cdots,x_n\) are absolutely from convex set \(C\) (such as, \(\mathbb{R}\)):
- For discrete distribution, \(\sum p_i = 1,\, p_i\ge 0,\,\sum{p_ix_i} \in C\)
- For continuous distribution, function \(\int_C p(x) = 1,\, p(x)\ge 0,\, \int_C p(x)\cdot x\, dx \in C\).
-
-
The space boundary of convex combinations \(\Rightarrow\) Cover the space with line segments
-
证明任意三点构成的三角形区域内的点可以被三顶点凸表出（凸表出的凸表出依然是凸表出）
Proof:
Assume \(x_1,x_2,x_3\) are 3 points in the set, so points on the line segment \(x_1x_2\) is \(x' = \theta_1 x_1 + \theta_2 x_2\).
Also, there is a convex combination of the line segment between the point on \(x_1x_2\) and \(x_3\): \(x = k_1 x' + k_2 x_3\). So we have
\[x = k_1\theta_1 x_1 + k_2\theta_2 x_2 + k_2 x_3 \]Because \(\theta_1 + \theta_2 = k_1 + k_2 = 1\), we get \(k_1\theta_1 + k_2\theta_2 + k_2 = 1\).
-
As long as focus on every line segment of any two points in the set, the boundary of the space represented by the convex combination can be found.
-
2.3 Convex Hull
Def:
A set \(\{\theta_1 \alpha_1 + \theta_2 \alpha_2 + \cdots + \theta_n \alpha_n \,\vert\, \theta_i\in [0,1],\sum{\theta_i} = 1, \alpha_i\in S,i=1,2,\cdots,n\}\), which contains any convex combination of vectors in the set \(S \subseteq \mathbb{R}^n\), is called the Convex Hull of \(S\), denoted as \(\mathbf{conv}\, S\).
-
The Convex Hull \(\mathbf{conv}\, S\) is the smallest Convex Set that contains set \(S\).
- That means if \(C\) is any convex set with \(S\subseteq C\), then \(\mathbf{conv}\, S \subseteq C\).
Furtherly, that also means any convex set is the convex hull of itself.
3 Cone
PS: Just similsr with Convex, But basic concept is Convex Cone.
Def:
A set \(S\) holds that \(\forall \, x\in S, \, \forall \, \theta \in \mathbb{R}_+,\, s.t. \, \theta x \in S\).
-
A Cone is Nonnegative Homogeneous, which usually means a rays set.
Not a usualiy useful concept. Convex Cone is More useful.
3.1 Convex Cone
Def:
A set \(C\) is not only a convex set, but also a cone. \(\forall \, x_1,x_2\in C, \, \forall \, \theta_1,\theta_2 \in \mathbb{R}_+,\, s.t. \, \theta_1\,x_1 + \theta_2\,x_2 \in S\)
-
This form can be described geometrically as forming the two-dimensional pie slice and edges are \(\bf x_1\) and \(\bf x_2\).
-
Any line is affine. If it passes through Origin, it is a subspace, so also a convex cone.
Thus, Any subspace is affine, convex and a convex cone.
-
A ray is a convex cone only when it passes through Origin.
3.2 Conic Combination
Def:
A Convex Combination, when the sum of coefficients \(\sum{\theta_i}\) is not constrainted any more, then the combination is called Conic Combination.
-
Only ask that every coefficient \(\theta_i\) is nonnegative.
A Conic Combination Also can be called as Nonnegative Linear Combination.
-
Every Convex Cone \(\in \mathbb{R}^n\) contains all conic combinations of its elements.
-
The idea of conic combination can be generalized to infinite sums and integrals.
- Generalization of Convex Conbination
-
关于证明锥组合张成空间的边界，思想类似 1.1.2.2
3.3 Conic Hull
Def:
For a set \(S \subseteq \mathbb{R}^n\), the set \(\{\theta_1 \alpha_1 + \theta_2 \alpha_2 + \cdots + \theta_n \alpha_n \,\vert\, \theta_i\ge 0, \alpha_i\in S,i=1,2,\cdots,n\}\), containing all conic combinations of vectors in \(S\), is called the Conic Hull of \(S\), denoted as \(\mathbf{conic}\, S\).
-
The Conic Hull \(\mathbf{conic}\, S\) is the smallest Convex Cone that contains set \(S\).
- That means if \(C\) is any convex cone with \(S\subseteq C\), then \(\mathbf{conic}\, S \subseteq C\).
Obviously, that also means any convex cone is the conic hull of itself.
4 Hyperplanes
Def:
浙公网安备 33010602011771号