看了刘建平老师的博客https://www.cnblogs.com/pinard/p/6422831.html对如下其中两个公式进行详细推导
损失函数为（大写字母为矩阵，小写字母字母加粗为列向量，其中\(W^L\)的维度为\(M_L*M_{L-1}\),即第\(L\)层神经元个数乘以第\(L-1\)层神经元个数）：
\[J(W, \mathbf{b}, \mathbf {x}, \mathbf{y})=\frac{1}{2}\left\|\mathbf{a}^{L}-\mathbf{y}\right\|_{2}^{2}=\frac{1}{2}\left\|\sigma\left(W^{L} \mathbf {a}^{L-1}+\mathbf {b}^{L}\right)-\mathbf {y}\right\|_{2}^{2}
\]
推导的两个公式如下：
\[\frac{\partial J(W, \mathbf{b}, \mathbf {x}, \mathbf{y})}{\partial W^{L}}=\left[\left(\mathbf {a}^{L}-\mathbf {y}\right) \odot \sigma^{\prime}\left(\mathbf {z}^{L}\right)\right]\left(\mathbf {a}^{L-1}\right)^{T}
\]
\[\frac{\partial J(W, \mathbf{b}, \mathbf {x}, \mathbf{y})}{\partial \mathbf {b}^{L}}=\left(\mathbf {a}^{L}-\mathbf {y}\right) \odot \sigma^{\prime}\left(\mathbf{z}^{L}\right)
\]
用到两个链式求导法则如下（都来源于刘建平老师博客，链接在文章末尾）
当标量对n个向量进行链式求导，即\(\mathbf{y}_{1} \rightarrow \mathbf{y}_{2} \rightarrow \ldots \rightarrow \mathbf{y}_{\mathbf{n}} \rightarrow z_{1}\)，链式求导法则如下：
\[\begin{equation}\frac{\partial z}{\partial \mathbf{y}_{\mathbf{1}}}=\left(\frac{\partial \mathbf{y}_{\mathbf{n}}}{\partial \mathbf{y}_{\mathbf{n}-\mathbf{1}}} \frac{\partial \mathbf{y}_{\mathbf{n}-\mathbf{1}}}{\partial \mathbf{y}_{\mathbf{n}-\mathbf{2}}} \ldots \frac{\partial \mathbf{y}_{\mathbf{2}}}{\partial \mathbf{y}_{\mathbf{1}}}\right)^{T} \frac{\partial z}{\partial \mathbf{y}_{\mathbf{n}}}\tag {1}\end{equation}
\]
当\(z=f(\mathbf {y}),\mathbf {y}=X\mathbf {a}+\mathbf {b}\)即\(X\rightarrow \mathbf{y}\rightarrow z\) 其中\(X\)为矩阵，\(\mathbf {y}\)为向量，链式求导结果如下：
\[\begin{equation}\frac{\partial z}{\partial X}=\frac {\partial z}{\partial {\mathbf{y}}}a^T\tag {2}\end{equation}
\]
先推导第一个公式，考虑如下复合结构(注意最后所求的\(J\)是标量)
\[W^L\rightarrow \mathbf{z}^L\rightarrow \mathbf{u}^L\rightarrow J
\]
其中$$J=\frac{1}{2}\Vert \mathbf{u}^L \Vert_2^2 $$
\[\mathbf{u}^L=\mathbf{a}^L-\mathbf{y}=\sigma (\mathbf{z}^L)-\mathbf{y}
\]
\[\mathbf{z}^L=W^L\mathbf{a}^{L-1}+\mathbf{b}^L
\]
由公式\((2)\)可得
\[\frac{\partial J}{\partial W^{L}}=\frac{\partial J}{\partial \mathbf {z}^{L}}(\mathbf a^{L-1})^T
\]
又有公式\((1)\)可得
\[\frac{\partial J}{\partial \mathbf {z}^{L}}=(\frac {\partial \mathbf{u}^L}{\partial \mathbf{z}^L})^T\frac {\partial J}{\partial \mathbf{u}^L}
\]
其中后半部分比较简单
\[\frac {\partial J}{\partial \mathbf{u}^L}=\mathbf {u}^L=\mathbf{a}^L-\mathbf{y}
\]
前半部分向量对向量求导，布局为雅克比矩阵形式，结果如下：
\[\frac{\partial \mathbf{u}^{L}}{\partial \mathbf{z}^{L}}=\frac{\partial\left(\sigma\left(\mathbf{z}^{L}\right)-\mathbf{y}\right)}{\partial \mathbf{z}^{L}}=\left(\begin{array}{lllc}
\frac{\partial \sigma\left(z_{1}^{L}\right)}{\partial z_{1}^{L}} & 0 & \cdots & 0 \\
0 & \frac{\partial \sigma\left(z_{2}^{L}\right)}{\partial z_{2}^{L}} & \cdots & 0 \\
\vdots & \vdots & \ddots & \vdots \\
0 & 0 & \cdots & \frac{\partial \sigma\left(z_{M_{L}}^{L}\right)}{\partial z_{M_{L}}^{L}}
\end{array}\right)=\operatorname{diag}\left(\sigma^{\prime}\left(\mathbf{z}^{L}\right)\right)
\]
得到的矩阵为对称矩阵，带入到上式，结果为：
\[\frac{\partial J}{\partial \mathbf {z}^{L}}=\operatorname{diag}\left(\sigma^{\prime}\left(\mathbf{z}^{L}\right)\right)(\mathbf{a}^L-\mathbf{y})=\sigma^{\prime}\left(\mathbf{z}^{L}\right)\odot (\mathbf{a}^L-\mathbf{y})=(\mathbf{a}^L-\mathbf{y})\odot\sigma^{\prime}\left(\mathbf{z}^{L}\right)
\]
故
\[\frac{\partial J}{\partial W^{L}}=\left[\left(\mathbf {a}^{L}-\mathbf {y}\right) \odot \sigma^{\prime}\left(\mathbf {z}^{L}\right)\right]\left(\mathbf {a}^{L-1}\right)^{T}
\]
第二个式子推导就很简单，由公式\((1)\)可得
\[\frac {\partial J}{\partial \mathbf{b}^L}=(\frac {\partial \mathbf{z}^L}{\partial \mathbf{b}^L})^T\frac {\partial J}{\partial \mathbf{z}^L}
\]
前半部分为单位矩阵\(E\)后半部分求第一个式子时已经求过，故
\[\frac{\partial J}{\partial \mathbf {b}^{L}}=\left(\mathbf {a}^{L}-\mathbf {y}\right) \odot \sigma^{\prime}\left(\mathbf{z}^{L}\right)
\]
参考博客（矩阵向量求导的知识）：
求导定义与布局
矩阵向量求导之定义法
矩阵向量求导之微分法法
矩阵向量求导链式法则
浙公网安备 33010602011771号