Part Ⅱ: Application
Least-Square
Ⅰ. Standard
A function connects a series of input variables like \((a_1, a_2, \cdots, a_n)\) with an output number \(b\) through a map.
Now we have a table of data:
\[\left( \begin{matrix} a_{11} & a_{12} & \cdots & a_{1n} \\ a_{21} & a_{22} & \cdots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & \cdots & a_{mn} \\ \end{matrix} \right. \left | \begin{matrix} b_1 \\ b_2 \\ \vdots \\ b_m \end{matrix} \right) = \left( A \right. \left | \mathbf{b} \right) \]Assume that there exist a linear relationship between \(\mathbf{\alpha}\) and \(b\), a set of coefficients \(\mathbf{x} = (x_1, x_2, \cdots, x_n)\)，we wanna find the optimal \(\mathbf{x}\), so that \(\Vert A\mathbf{x} - \mathbf{b} \Vert_2\) can achieve the minimum.
-
perpendicularity - Geometry
View the \(A\mathbf{x}\) as a linear combination of vectors, so:
\[\begin{pmatrix} a_{11} & a_{12} & \cdots & a_{1n} \\ a_{21} & a_{22} & \cdots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & \cdots & a_{mn} \\ \end{pmatrix} = \begin{pmatrix} \alpha_1 & \alpha_2 & \cdots & \alpha_n \\ \end{pmatrix} \]Because \(\mathbf{b}\) is a vector in m-dim space \(V\), and similarly, \(A\mathbf{x}\) constructs a space \(V'\) (a span of \(\alpha_1,\alpha_2, \cdots, \alpha_m\)) in \(V\). So when \(\Vert A\mathbf{x} - \mathbf{b} \Vert\) is the minimum, the vector \(A\mathbf{x} - \mathbf{b}\) should be perpendicular to every base vector of \(V'\), so that we can get
\[{\alpha_i}^T \cdot (A\mathbf{x} - \mathbf{b}) = 0, \text{for } i=1,2,\cdots, m \\ \longrightarrow A^T\cdot (A\mathbf{x} - \mathbf{b}) = 0 \]So the optimal solution of \(\mathbf{x}\) is from the equation
\[A^T A \mathbf{x} = A^T \mathbf{b} \] -
Matrix - Algebra
if \(\Vert A\mathbf{x} - \mathbf{b} \Vert\) achieves the minimum, then \((A\mathbf{x} - \mathbf{b})^T (A\mathbf{x} - \mathbf{b})\) also achieves the minimum. So we can get a function:
\[f(\mathbf{x}) = (A\mathbf{x} - \mathbf{b})^T (A\mathbf{x} - \mathbf{b}) \]This is a scalar function, we wanna take its derivative with respect to vector \(\mathbf{x}\).
\[\begin{aligned} \frac{\partial f(\mathbf{x})}{\partial \mathbf{x}} &= \frac{\partial \mathbf{x}^T A^T A \mathbf{x}}{\partial \mathbf{x}} - \frac{\partial \mathbf{b}^T A \mathbf{x}}{\partial \mathbf{x}} - \frac{\partial \mathbf{x}^T A^T \mathbf{b}}{\partial \mathbf{x}} + \frac{\partial \mathbf{b}^T \mathbf{b}}{\partial \mathbf{x}} \\ &= 2A^TA \mathbf{x} - 2A^T \mathbf{b} \\ &= 2A^T(A \mathbf{x} - \mathbf{b}) \end{aligned} \]When \(\frac{\partial f(\mathbf{x})}{\partial \mathbf{x}} = \mathbf{O}\), we can ensure \(\Vert A\mathbf{x} - \mathbf{b} \Vert\) is the minimum (Because the maximun doesn't exist), So we can also get:
\[A^T A \mathbf{x} = A^T \mathbf{b} \]
浙公网安备 33010602011771号