神经网络

神经网络背后的对数几率回归模型

公式推导

Forward

各个量的维度如图所示：

输入层到隐含层：

\[x' = g_1 (Ux + b_1) \]

隐含层到输出层：

\[\hat{y} = g_2 (Vx' + b_2) \]

注：\(g_1( \cdot )\) 简记为 \(g_1\)，\(g_2( \cdot )\) 简记为 \(g_2\)。

Backward

假设损失函数对输出值的偏导数为 \(\Delta\)，则有 \(\mathrm{d}J = tr(\Delta ^T \mathrm{d} \hat{y} )\)

计算损失对 \(b_2\) 的偏导数

\(\mathrm{d} J = tr[\Delta ^T (g_2' \odot \mathrm{d} b_2)]\)
\(\ \ \ \ = tr[(\Delta \odot g_2')^T \mathrm{d} b_2]\)
至此可得：

\[\frac{\partial J}{\partial b_2} = \Delta \odot g_2' \]

计算损失对 V 的偏导数

\(\mathrm{d} J = tr[\Delta ^T (g_2' \odot \mathrm{d} V x')]\)
\(\ \ \ \ = tr[(\Delta \odot g_2')^T \mathrm{d} V x']\)
\(\ \ \ \ = tr[x' \times (\Delta \odot g_2') \mathrm{d} V]\)
至此可得：

\[\frac{\partial J}{\partial V} = [x' \times (\Delta \odot g_2')]^T \]

计算损失对 \(b_1\) 的偏导数

\(\mathrm{d} J = tr[\Delta ^T (g_2' \odot V \mathrm{d} x')]\)
\(\ \ \ \ = tr[(\Delta \odot g_2')^T V g_1' \odot \mathrm{d} b_1]\)
\(\ \ \ \ = tr \{ [V^T (\Delta \odot g_2') \odot g_1']^T \mathrm{d} b_1 \}\)
至此可得：

\[\frac{\partial J}{\partial b_1} = V^T (\Delta \odot g_2') \odot g_1' \]

计算损失对 U 的偏导数

\(\mathrm{d} J = tr[\Delta ^T (g_2' \odot V \mathrm{d} x')]\)
\(\ \ \ \ = tr[(\Delta \odot g_2')^T V g_1' \odot (\mathrm{d} U x)]\)
\(\ \ \ \ = tr \{ [V^T (\Delta \odot g_2') \odot g_1']^T \mathrm{d}Ux \}\)
\(\ \ \ \ = tr[x \times V^T(\Delta \odot g_2') \odot g_1' \mathrm{d}U]\)
至此可得：

\[\frac{\partial J}{\partial U} = \{x \times [V^T (\Delta \odot g_2') \odot g_1'] \}^T \]

仔细观察计算过程，可以发现其实 \(\frac{\partial J}{\partial b_2}\) 等价于 \(\frac{\partial J}{\partial Vx' + b_2}\)，\(\frac{\partial J}{\partial b_1}\) 等价于 \(\frac{\partial J}{\partial Ux+b_1}\)。\(Vx' + b_2\) 是激活之前的输出层，\(Ux + b_1\) 是激活之前的隐含层。于是记损失对激活前输出层的梯度为 \(\nabla_{* o} \ J = \Delta \odot g_2'\)，对激活前的隐含层的梯度为 \(\nabla_{ * h} \ J = V^T ( \Delta \odot g_2' ) \odot g_1'\)。其二者亦有关系：\(\nabla_{ * h} \ J = V^T\nabla_{* o} \ J \odot g_1'\) 则上述四个导数可写作：

\[\begin{matrix} \frac{\partial J}{\partial b_2} &=& \nabla_{*o} \ J \\ \frac{\partial J}{\partial V} &=& \nabla_{* o} \ J \times x' \\ \frac{\partial J}{\partial b_1} &=& \nabla_{* h} \ J \\ \frac{\partial J}{\partial U} &=& \nabla_{* h} \ J \times x \end{matrix} \]

理解

1. 是否设置 bias 要更具实际问题来决定。
2. 一个神经网络可以看做是若干个逻辑斯蒂回归模型的嵌套组合。
3. 神经网络可以看做，不用输入特征来训练，而是用输入特征的映射来训练逻辑回归，这个映射可以是很复杂的映射。