2022年互联网营销师如何报名seo是什么服务器
深度学习笔记之循环神经网络——GRU的反向传播过程
引言
上一节介绍了门控循环单元 ( Gate Recurrent Unit,GRU ) (\text{Gate Recurrent Unit,GRU}) (Gate Recurrent Unit,GRU)，本节我们参照 LSTM \text{LSTM} LSTM反向传播的格式，观察 GRU \text{GRU} GRU的反向传播过程。
回顾： GRU \text{GRU} GRU的前馈计算过程
GRU \text{GRU} GRU的前馈计算过程表示如下：
为了后续的反向传播过程，将过程分解的细致一些。其中 Z ~ ( t ) , r ~ ( t ) \widetilde{\mathcal Z}^{(t)},\widetilde{r}^{(t)} Z (t),r (t)分别表示更新门、重置门的线性计算过程。
{ Z ~ ( t ) = W H ⇒ Z ⋅ h ( t − 1 ) + W X ⇒ Z ⋅ x ( t ) + b Z Z ( t ) = σ ( Z ~ ( t ) ) r ~ ( t ) = W H ⇒ r ⋅ h ( t − 1 ) + W X ⇒ r ⋅ x ( t ) + b r r ( t ) = σ ( r ~ ( t ) ) h ~ ( t ) = Tanh [ W H ⇒ H ~ ⋅ ( r ( t ) ∗ h ( t − 1 ) ) + W X ⇒ H ~ ⋅ x ( t ) + b H ~ ] h ( t ) = ( 1 − Z ( t ) ) ∗ h ( t − 1 ) + Z ( t ) ∗ h ~ ( t ) \begin{cases} \begin{aligned} & \widetilde{\mathcal Z}^{(t)} = \mathcal W_{\mathcal H \Rightarrow \mathcal Z} \cdot h^{(t-1)} + \mathcal W_{\mathcal X \Rightarrow \mathcal Z} \cdot x^{(t)} + b_{\mathcal Z} \\ & \mathcal Z^{(t)} = \sigma(\widetilde{\mathcal Z}^{(t)}) \\ & \widetilde{r}^{(t)} = \mathcal W_{\mathcal H \Rightarrow r} \cdot h^{(t-1)} + \mathcal W_{\mathcal X \Rightarrow r} \cdot x^{(t)} + b_{r} \\ & r^{(t)} = \sigma(\widetilde{r}^{(t)}) \\ & \widetilde{h}^{(t)} = \text{Tanh} \left[\mathcal W_{\mathcal H \Rightarrow \widetilde{\mathcal H}} \cdot (r^{(t)} * h^{(t-1)}) + \mathcal W_{\mathcal X \Rightarrow \widetilde{\mathcal H}} \cdot x^{(t)} + b_{\widetilde{\mathcal H}}\right] \\ & h^{(t)} = (1 -\mathcal Z^{(t)}) * h^{(t-1)} + \mathcal Z^{(t)} * \widetilde{h}^{(t)} \end{aligned} \end{cases} ⎩ ⎨ ⎧Z (t)=WH⇒Z⋅h(t−1)+WX⇒Z⋅x(t)+bZZ(t)=σ(Z (t))r (t)=WH⇒r⋅h(t−1)+WX⇒r⋅x(t)+brr(t)=σ(r (t))h (t)=Tanh[WH⇒H ⋅(r(t)∗h(t−1))+WX⇒H ⋅x(t)+bH ]h(t)=(1−Z(t))∗h(t−1)+Z(t)∗h (t)
场景设计
上述仅描述的是 GRU \text{GRU} GRU关于序列信息 h ( t ) ( t = 1 , 2 , ⋯ , T ) h^{(t)}(t=1,2,\cdots,\mathcal T) h(t)(t=1,2,⋯,T)的迭代过程。各时刻的输出特征以及损失函数于循环神经网络相同：
- 使用 Softmax \text{Softmax} Softmax激活函数，其输出结果作为模型对 t t t时刻的预测结果：
{ C ( t ) = W H ⇒ C ⋅ h ( t ) + b h y ^ ( t ) = Softmax ( C ( t ) ) \begin{cases} \mathcal C^{(t)} = \mathcal W_{\mathcal H \Rightarrow \mathcal C} \cdot h^{(t)} + b_{h} \\ \hat y^{(t)} = \text{Softmax}(\mathcal C^{(t)}) \end{cases} {C(t)=WH⇒C⋅h(t)+bhy^(t)=Softmax(C(t)) - 关于 t t t时刻预测结果 y ^ ( t ) \hat y^{(t)} y^(t)与真实分布 y ( t ) y^{(t)} y(t)之间的偏差信息使用交叉熵 ( CrossEntropy ) (\text{CrossEntropy}) (CrossEntropy)进行表示：
其中n Y n_{\mathcal Y} nY表示预测/真实分布的维数。
L ( t ) = L [ y ^ ( t ) , y ( t ) ] = − ∑ j = 1 n Y y j ( t ) log [ y ^ j ( t ) ] \mathcal L^{(t)} = \mathcal L \left[\hat y^{(t)},y^{(t)}\right] = - \sum_{j=1}^{n_\mathcal Y} y_j^{(t)} \log \left[\hat y_j^{(t)}\right] L(t)=L[y^(t),y(t)]=−j=1∑nYyj(t)log[y^j(t)] - 所有时刻交叉熵结果的累加和构成完整的损失函数 L \mathcal L L：
L = ∑ t = 1 T L ( t ) = ∑ t = 1 T L [ y ^ ( t ) , y ( t ) ] \begin{aligned} \mathcal L & = \sum_{t=1}^{\mathcal T} \mathcal L^{(t)}\\ & = \sum_{t=1}^{\mathcal T} \mathcal L \left[\hat y^{(t)},y^{(t)}\right] \end{aligned} L=t=1∑TL(t)=t=1∑TL[y^(t),y(t)]
反向传播过程
T \mathcal T T时刻的反向传播过程
以 T \mathcal T T时刻重置门 ∂ L ∂ W h ( T ) ⇒ Z ( T ) \begin{aligned}\frac{\partial \mathcal L}{\partial \mathcal W_{\mathcal h^{(\mathcal T)} \Rightarrow \mathcal Z^{(\mathcal T)}}}\end{aligned} ∂Wh(T)⇒Z(T)∂L的反向传播为例：
- 计算梯度 ∂ L ∂ L ( T ) \begin{aligned}\frac{\partial \mathcal L}{\partial \mathcal L^{(\mathcal T)}}\end{aligned} ∂L(T)∂L：
其中仅有L ( T ) \mathcal L^{(\mathcal T)} L(T)一项存在梯度，其余项均视作常数。
∂ L ∂ L ( T ) = ∂ ∂ L ( T ) [ ∑ t = 1 T L ( t ) ] = 0 + 0 + ⋯ + 1 = 1 \frac{\partial \mathcal L}{\partial \mathcal L^{(\mathcal T)}} = \frac{\partial}{\partial \mathcal L^{(\mathcal T)}} \left[\sum_{t=1}^{\mathcal T} \mathcal L^{(t)}\right] = 0 + 0 + \cdots + 1 = 1 ∂L(T)∂L=∂L(T)∂[t=1∑TL(t)]=0+0+⋯+1=1 - 计算梯度 ∂ L ( T ) ∂ C ( T ) \begin{aligned}\frac{\partial \mathcal L^{(\mathcal T)}}{\partial \mathcal C^{(\mathcal T)}}\end{aligned} ∂C(T)∂L(T)：
关于Softmax \text{Softmax} Softmax激活函数与交叉熵组合的梯度描述，见循环神经网络—— Softmax \text{Softmax} Softmax函数的反向传播过程一节，这里不再赘述。
{ L ( T ) = − ∑ j = 1 n Y y j ( T ) log [ y ^ j ( T ) ] y ^ ( T ) = Softmax [ C ( T ) ] ⇒ ∂ L ( T ) ∂ C ( T ) = y ^ ( T ) − y ( T ) \begin{aligned} & \begin{cases} \mathcal L^{(\mathcal T)} = -\sum_{j=1}^{n_{\mathcal Y}}y_j^{(\mathcal T)} \log \left[\hat y_j^{(\mathcal T)}\right] \\ \hat y^{(\mathcal T)} = \text{Softmax}[\mathcal C^{(\mathcal T)}] \end{cases} \\ & \Rightarrow \frac{\partial \mathcal L^{(\mathcal T)}}{\partial \mathcal C^{(\mathcal T)}} = \hat y^{(\mathcal T)} - y^{(\mathcal T)} \end{aligned} {L(T)=−∑j=1nYyj(T)log[y^j(T)]y^(T)=Softmax[C(T)]⇒∂C(T)∂L(T)=y^(T)−y(T) - 继续计算梯度 ∂ C ( T ) ∂ h ( T ) \begin{aligned}\frac{\partial \mathcal C^{(\mathcal T)}}{\partial h^{(\mathcal T)}}\end{aligned} ∂h(T)∂C(T)：
∂ C ( T ) ∂ h ( T ) = ∂ ∂ h ( T ) [ W H ⇒ C ⋅ h ( T ) + b h ] = W H ⇒ C \frac{\partial \mathcal C^{(\mathcal T)}}{\partial h^{(\mathcal T)}} = \frac{\partial}{\partial h^{(\mathcal T)}} \left[\mathcal W_{\mathcal H \Rightarrow \mathcal C} \cdot h^{(\mathcal T)} + b_{h}\right] = \mathcal W_{\mathcal H \Rightarrow \mathcal C} ∂h(T)∂C(T)=∂h(T)∂[WH⇒C⋅h(T)+bh]=WH⇒C
至此，关于梯度 ∂ L ∂ h ( T ) \begin{aligned}\frac{\partial \mathcal L}{\partial h^{(\mathcal T)}}\end{aligned} ∂h(T)∂L可表示为：
∂ L ∂ h ( T ) = ∂ L ∂ L ( T ) ⋅ ∂ L ( T ) ∂ C ( T ) ⋅ ∂ C ( T ) ∂ h ( T ) = 1 ⋅ [ W H ⇒ C ] T ⋅ ( y ^ ( T ) − y ( T ) ) \begin{aligned} \frac{\partial \mathcal L}{\partial h^{(\mathcal T)}} & = \frac{\partial \mathcal L}{\partial \mathcal L^{(\mathcal T)}} \cdot \frac{\partial \mathcal L^{(\mathcal T)}}{\partial \mathcal C^{(\mathcal T)}} \cdot \frac{\partial \mathcal C^{(\mathcal T)}}{\partial h^{(\mathcal T)}} \\ & = 1 \cdot \left[\mathcal W_{\mathcal H \Rightarrow \mathcal C}\right]^T \cdot (\hat y^{(\mathcal T)} - y^{(\mathcal T)}) \end{aligned} ∂h(T)∂L=∂L(T)∂L⋅∂C(T)∂L(T)⋅∂h(T)∂C(T)=1⋅[WH⇒C]T⋅(y^(T)−y(T))
观察：从 h ( T ) h^{(\mathcal T)} h(T)开始，从 h ( T ) ⇒ W h ( T ) ⇒ Z ( T ) h^{(\mathcal T)} \Rightarrow \mathcal W_{h^{(\mathcal T)} \Rightarrow \mathcal Z^{(\mathcal T)}} h(T)⇒Wh(T)⇒Z(T)的传播路径都有哪些。
只有唯一一条，其前馈计算路径表示为：
{ h ( t ) = ( 1 − Z ( t ) ) ∗ h ( t − 1 ) + Z ( t ) ∗ h ~ ( t ) Z ( t ) = σ ( Z ~ ( t ) ) Z ~ ( t ) = W H ⇒ Z ⋅ h ( t − 1 ) + W X ⇒ Z ⋅ x ( t ) + b Z \begin{cases} \begin{aligned} & h^{(t)} = (1 -\mathcal Z^{(t)}) * h^{(t-1)} + \mathcal Z^{(t)} * \widetilde{h}^{(t)} \\ & \mathcal Z^{(t)} = \sigma(\widetilde{\mathcal Z}^{(t)}) \\ & \widetilde{\mathcal Z}^{(t)} = \mathcal W_{\mathcal H \Rightarrow \mathcal Z} \cdot h^{(t-1)} + \mathcal W_{\mathcal X \Rightarrow \mathcal Z} \cdot x^{(t)} + b_{\mathcal Z} \end{aligned} \end{cases} ⎩ ⎨ ⎧h(t)=(1−Z(t))∗h(t−1)+Z(t)∗h (t)Z(t)=σ(Z (t))Z (t)=WH⇒Z⋅h(t−1)+WX⇒Z⋅x(t)+bZ
对应反向传播结果表示为：
∂ h ( T ) ∂ W h ( T ) ⇒ Z ( T ) = ∂ h ( T ) ∂ Z ( T ) ⋅ ∂ Z ( T ) ∂ Z ~ ( T ) ⋅ ∂ Z ~ ( T ) ∂ W h ( T ) ⇒ Z ( T ) = [ h ~ ( T ) − h ( T − 1 ) ] ⋅ [ Sigmoid ( Z ~ ( T ) ) ] ′ ⋅ h ( T − 1 ) \begin{aligned} \frac{\partial h^{(\mathcal T)}}{\partial \mathcal W_{h^{(\mathcal T)} \Rightarrow \mathcal Z^{(\mathcal T)}}} & = \frac{\partial h^{(\mathcal T)}}{\partial \mathcal Z^{(\mathcal T)}} \cdot \frac{\partial \mathcal Z^{(\mathcal T)}}{\partial \widetilde{\mathcal Z}^{(\mathcal T)}} \cdot \frac{\partial \widetilde{\mathcal Z}^{(\mathcal T)}}{\partial \mathcal W_{h^{(\mathcal T)} \Rightarrow \mathcal Z^{(\mathcal T)}}} \\ & = \left[\widetilde{h}^{(\mathcal T)} - h^{(\mathcal T - 1)}\right] \cdot \left[\text{Sigmoid}(\widetilde{\mathcal Z}^{(\mathcal T)})\right]' \cdot h^{(\mathcal T - 1)} \end{aligned} ∂Wh(T)⇒Z(T)∂h(T)=∂Z(T)∂h(T)⋅∂Z (T)∂Z(T)⋅∂Wh(T)⇒Z(T)∂Z (T)=[h (T)−h(T−1)]⋅[Sigmoid(Z (T))]′⋅h(T−1)
最终，关于 ∂ L ∂ W h ( T ) ⇒ Z ( T ) \begin{aligned}\frac{\partial \mathcal L}{\partial \mathcal W_{\mathcal h^{(\mathcal T)} \Rightarrow \mathcal Z^{(\mathcal T)}}}\end{aligned} ∂Wh(T)⇒Z(T)∂L的反向传播结果为：
这里更主要的是描述它的反向传播路径，它的具体展开在后续不再赘述。
∂ L ∂ W h ( T ) ⇒ Z ( T ) = ∂ L ∂ h ( T ) ⋅ ∂ h ( T ) ∂ W h ( T ) ⇒ Z ( T ) = { [ W H ⇒ C ] T ⋅ ( y ^ ( T ) − y ( T ) ) } ⋅ { [ h ~ ( T ) − h ( T − 1 ) ] ⋅ [ Sigmoid ( Z ~ ( T ) ) ] ′ ⋅ h ( T − 1 ) } \begin{aligned} \begin{aligned}\frac{\partial \mathcal L}{\partial \mathcal W_{\mathcal h^{(\mathcal T)} \Rightarrow \mathcal Z^{(\mathcal T)}}}\end{aligned} & = \frac{\partial \mathcal L}{\partial h^{(\mathcal T)}} \cdot \frac{\partial h^{(\mathcal T)}}{\partial \mathcal W_{h^{(\mathcal T)}\Rightarrow \mathcal Z^{(\mathcal T)}}} \\ & = \left\{\left[\mathcal W_{\mathcal H \Rightarrow \mathcal C}\right]^T \cdot (\hat y^{(\mathcal T)} - y^{(\mathcal T)})\right\} \cdot \left\{ \left[\widetilde{h}^{(\mathcal T)} - h^{(\mathcal T - 1)}\right] \cdot \left[\text{Sigmoid}(\widetilde{\mathcal Z}^{(\mathcal T)})\right]' \cdot h^{(\mathcal T - 1)}\right\} \end{aligned} ∂Wh(T)⇒Z(T)∂L=∂h(T)∂L⋅∂Wh(T)⇒Z(T)∂h(T)={[WH⇒C]T⋅(y^(T)−y(T))}⋅{[h (T)−h(T−1)]⋅[Sigmoid(Z (T))]′⋅h(T−1)}
T − 1 \mathcal T - 1 T−1时刻的反向传播路径
关于 T − 1 \mathcal T - 1 T−1时刻的重置门梯度 ∂ L ∂ W h ( T − 1 ) ⇒ Z ( T − 1 ) \begin{aligned} \frac{\partial \mathcal L}{\partial \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}}} \end{aligned} ∂Wh(T−1)⇒Z(T−1)∂L，它的路径主要包含两大类：
第一类路径：同 T \mathcal T T时刻路径，从对应的 L ( T − 1 ) \mathcal L^{(\mathcal T - 1)} L(T−1)直接传至 W h ( T − 1 ) ⇒ Z ( T − 1 ) \mathcal W_{h^{(\mathcal T - 1)}\Rightarrow \mathcal Z^{(\mathcal T - 1)}} Wh(T−1)⇒Z(T−1)：
该路径与上述 T \mathcal T T时刻的路径类型相同，将对应的上标 T \mathcal T T改为 T − 1 \mathcal T - 1 T−1即可。
∂ L ( T − 1 ) ∂ W h ( T − 1 ) ⇒ Z ( T − 1 ) = ∂ L ( T − 1 ) ∂ h ( T − 1 ) ⋅ ∂ h ( T − 1 ) ∂ W h ( T − 1 ) ⇒ Z ( T − 1 ) \begin{aligned} \frac{\partial \mathcal L^{(\mathcal T - 1)}}{\partial \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}}} & = \frac{\partial \mathcal L^{(\mathcal T - 1)}}{\partial h^{(\mathcal T - 1)}} \cdot \frac{\partial h^{(\mathcal T - 1)}}{\partial \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}}} \end{aligned} ∂Wh(T−1)⇒Z(T−1)∂L(T−1)=∂h(T−1)∂L(T−1)⋅∂Wh(T−1)⇒Z(T−1)∂h(T−1)
第二类路径：重新观察 W h ( T − 1 ) ⇒ Z ( T − 1 ) \mathcal W_{h^{(\mathcal T - 1)}\Rightarrow \mathcal Z^{(\mathcal T - 1)}} Wh(T−1)⇒Z(T−1)只会出现在 Z ( T − 1 ) \mathcal Z^{(\mathcal T - 1)} Z(T−1)中，并且 Z ( T − 1 ) \mathcal Z^{(\mathcal T - 1)} Z(T−1)只会出现在 h ( T − 1 ) h^{(\mathcal T - 1)} h(T−1)中。因此：仅需要找出与 h ( T − 1 ) h^{(\mathcal T - 1)} h(T−1)相关的所有路径即可，最终都可以使用 ∂ h ( T − 1 ) ∂ W h ( T − 1 ) ⇒ Z ( T − 1 ) \begin{aligned}\frac{\partial h^{(\mathcal T - 1)}}{\partial \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}}}\end{aligned} ∂Wh(T−1)⇒Z(T−1)∂h(T−1)将梯度传递给 W h ( T − 1 ) ⇒ Z ( T − 1 ) \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}} Wh(T−1)⇒Z(T−1)。
其中‘第一类路径’就是其中一种情况。只不过它是从当前 T − 1 \mathcal T - 1 T−1时刻直接传递得到的梯度结果。而第二类路径我们关注从 T \mathcal T T时刻传递产生的梯度信息。
从 T ⇒ T − 1 \mathcal T \Rightarrow \mathcal T - 1 T⇒T−1时刻中，关于 h ( T − 1 ) h^{(\mathcal T - 1)} h(T−1)的梯度路径一共包含 4 4 4条：
- 第一条：通过 T \mathcal T T时刻 h ( T ) h^{(\mathcal T)} h(T)中的 h ( T − 1 ) h^{(\mathcal T - 1)} h(T−1)进行传递。
{ Forword : h ( T ) = ( 1 − Z ( T ) ) ∗ h ( T − 1 ) + Z ( T ) ∗ h ~ ( T ) Backward : ∂ L ( T ) ∂ W h ( T − 1 ) ⇒ Z ( T − 1 ) ⇒ ∂ L ( T ) ∂ h ( T ) ⋅ ∂ h ( T ) ∂ h ( T − 1 ) ⋅ ∂ h ( T − 1 ) ∂ W h ( T − 1 ) ⇒ Z ( T − 1 ) \begin{cases} \text{Forword : } h^{(\mathcal T)} = (1 -\mathcal Z^{(\mathcal T)}) * h^{(\mathcal T -1)} + \mathcal Z^{(\mathcal T)} * \widetilde{h}^{(\mathcal T)} \\ \quad \\ \text{Backward : }\begin{aligned} \frac{\partial \mathcal L^{(\mathcal T)}}{\partial \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}}} \Rightarrow \frac{\partial \mathcal L^{(\mathcal T)}}{\partial h^{(\mathcal T)}} \cdot \frac{\partial h^{(\mathcal T)}}{\partial h^{(\mathcal T - 1)}} \cdot \frac{\partial h^{(\mathcal T - 1)}}{\partial \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}}} \end{aligned} \end{cases} ⎩ ⎨ ⎧Forword : h(T)=(1−Z(T))∗h(T−1)+Z(T)∗h (T)Backward : ∂Wh(T−1)⇒Z(T−1)∂L(T)⇒∂h(T)∂L(T)⋅∂h(T−1)∂h(T)⋅∂Wh(T−1)⇒Z(T−1)∂h(T−1) - 第二条：通过 T \mathcal T T时刻 h ( T ) h^{(\mathcal T)} h(T)中的 Z ( T ) \mathcal Z^{(\mathcal T)} Z(T)向 h ( T − 1 ) h^{(\mathcal T-1)} h(T−1)进行传递。
{ Forward : { h ( T ) = ( 1 − Z ( T ) ) ∗ h ( T − 1 ) + Z ( T ) ∗ h ~ ( T ) Z ( T ) = σ [ W H ⇒ Z ⋅ h ( T − 1 ) + W X ⇒ Z ⋅ x ( T ) + b Z ] Backward : ∂ L ( T ) ∂ W h ( T − 1 ) ⇒ Z ( T − 1 ) ⇒ ∂ L ( T ) ∂ h ( T ) ⋅ ∂ h ( T ) ∂ Z ( T ) ⋅ ∂ Z ( T ) ∂ h ( T − 1 ) ⋅ ∂ h ( T − 1 ) ∂ W h ( T − 1 ) ⇒ Z ( T − 1 ) \begin{cases} \text{Forward : } \begin{cases} h^{(\mathcal T)} = (1 -\mathcal Z^{(\mathcal T)}) * h^{(\mathcal T -1)} + \mathcal Z^{(\mathcal T)} * \widetilde{h}^{(\mathcal T)} \\ \mathcal Z^{(\mathcal T)} = \sigma \left[\mathcal W_{\mathcal H \Rightarrow \mathcal Z} \cdot h^{(\mathcal T -1)} + \mathcal W_{\mathcal X \Rightarrow \mathcal Z} \cdot x^{(\mathcal T)} + b_{\mathcal Z}\right] \end{cases} \quad \\ \text{Backward : } \begin{aligned} \frac{\partial \mathcal L^{(\mathcal T)}}{\partial \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}}} \Rightarrow \frac{\partial \mathcal L^{(\mathcal T)}}{\partial h^{(\mathcal T)}} \cdot \frac{\partial h^{(\mathcal T)}}{\partial \mathcal Z^{(\mathcal T)}} \cdot \frac{\partial \mathcal Z^{(\mathcal T)}}{\partial h^{(\mathcal T - 1)}} \cdot \frac{\partial h^{(\mathcal T - 1)}}{\partial \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}}} \end{aligned} \end{cases} ⎩ ⎨ ⎧Forward : {h(T)=(1−Z(T))∗h(T−1)+Z(T)∗h (T)Z(T)=σ[WH⇒Z⋅h(T−1)+WX⇒Z⋅x(T)+bZ]Backward : ∂Wh(T−1)⇒Z(T−1)∂L(T)⇒∂h(T)∂L(T)⋅∂Z(T)∂h(T)⋅∂h(T−1)∂Z(T)⋅∂Wh(T−1)⇒Z(T−1)∂h(T−1) - 第三条：通过 T \mathcal T T时刻 h ( T ) h^{(\mathcal T)} h(T)中的 h ~ ( T ) \widetilde{h}^{(\mathcal T)} h (T)向 h ( T − 1 ) h^{(\mathcal T - 1)} h(T−1)进行传递。
{ Forward : { h ( T ) = ( 1 − Z ( T ) ) ∗ h ( T − 1 ) + Z ( T ) ∗ h ~ ( T ) h ~ ( T ) = Tanh [ W H ⇒ H ~ ⋅ ( r ( T ) ∗ h ( T − 1 ) ) + W X ⇒ H ~ ⋅ x ( T ) + b H ~ ] Backward : ∂ L ( T ) ∂ W h ( T − 1 ) ⇒ Z ( T − 1 ) ⇒ ∂ L ( T ) ∂ h ( T ) ⋅ ∂ h ( T ) ∂ h ~ ( T ) ⋅ h ~ ( T ) ∂ h ( T − 1 ) ⋅ ∂ h ( T − 1 ) ∂ W h ( T − 1 ) ⇒ Z ( T − 1 ) \begin{cases} \text{Forward : } \begin{cases} h^{(\mathcal T)} = (1 -\mathcal Z^{(\mathcal T)}) * h^{(\mathcal T -1)} + \mathcal Z^{(\mathcal T)} * \widetilde{h}^{(\mathcal T)} \\ \widetilde{h}^{(\mathcal T)} = \text{Tanh} \left[\mathcal W_{\mathcal H \Rightarrow \widetilde{\mathcal H}} \cdot (r^{(\mathcal T)} * h^{(\mathcal T -1)}) + \mathcal W_{\mathcal X \Rightarrow \widetilde{\mathcal H}} \cdot x^{(\mathcal T)} + b_{\widetilde{\mathcal H}}\right] \end{cases}\\ \quad \\ \text{Backward : } \begin{aligned} \frac{\partial \mathcal L^{(\mathcal T)}}{\partial \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}}} \Rightarrow \frac{\partial \mathcal L^{(\mathcal T)}}{\partial h^{(\mathcal T)}} \cdot \frac{\partial h^{(\mathcal T)}}{\partial \widetilde{h}^{(\mathcal T)}} \cdot \frac{\widetilde{h}^{(\mathcal T)}}{\partial h^{(\mathcal T - 1)}} \cdot \frac{\partial h^{(\mathcal T - 1)}}{\partial \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}}} \end{aligned} \end{cases} ⎩ ⎨ ⎧Forward : {h(T)=(1−Z(T))∗h(T−1)+Z(T)∗h (T)h (T)=Tanh[WH⇒H ⋅(r(T)∗h(T−1))+WX⇒H ⋅x(T)+bH ]Backward : ∂Wh(T−1)⇒Z(T−1)∂L(T)⇒∂h(T)∂L(T)⋅∂h (T)∂h(T)⋅∂h(T−1)h (T)⋅∂Wh(T−1)⇒Z(T−1)∂h(T−1) - 第四条：与第三条路径类似，只不过从 h ~ ( T ) \widetilde{h}^{(\mathcal T)} h (T)中的 r ( T ) r^{(\mathcal T)} r(T)向 h ( T − 1 ) h^{(\mathcal T - 1)} h(T−1)进行传递。
{ Forward : { h ( T ) = ( 1 − Z ( T ) ) ∗ h ( T − 1 ) + Z ( T ) ∗ h ~ ( T ) h ~ ( T ) = Tanh [ W H ⇒ H ~ ⋅ ( r ( T ) ∗ h ( T − 1 ) ) + W X ⇒ H ~ ⋅ x ( T ) + b H ~ ] r ~ ( T ) = W H ⇒ r ⋅ h ( T − 1 ) + W X ⇒ r ⋅ x ( T ) + b r Backward : ∂ L ( T ) ∂ W h ( T − 1 ) ⇒ Z ( T − 1 ) ⇒ ∂ L ( T ) ∂ h ( T ) ⋅ ∂ h ( T ) ∂ h ~ ( T ) ⋅ ∂ h ~ ( T ) ∂ r ( T ) ⋅ ∂ r ( T ) ∂ h ( T − 1 ) ⋅ ∂ h ( T − 1 ) ∂ W h ( T − 1 ) ⇒ Z ( T − 1 ) \begin{cases} \text{Forward : } \begin{cases} h^{(\mathcal T)} = (1 -\mathcal Z^{(\mathcal T)}) * h^{(\mathcal T -1)} + \mathcal Z^{(\mathcal T)} * \widetilde{h}^{(\mathcal T)} \\ \widetilde{h}^{(\mathcal T)} = \text{Tanh} \left[\mathcal W_{\mathcal H \Rightarrow \widetilde{\mathcal H}} \cdot (r^{(\mathcal T)} * h^{(\mathcal T -1)}) + \mathcal W_{\mathcal X \Rightarrow \widetilde{\mathcal H}} \cdot x^{(\mathcal T)} + b_{\widetilde{\mathcal H}}\right] \\ \widetilde{r}^{(\mathcal T)} = \mathcal W_{\mathcal H \Rightarrow r} \cdot h^{(\mathcal T -1)} + \mathcal W_{\mathcal X \Rightarrow r} \cdot x^{(\mathcal T)} + b_{r} \end{cases}\\ \quad \\ \text{Backward : } \begin{aligned} \frac{\partial \mathcal L^{(\mathcal T)}}{\partial \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}}} \Rightarrow \frac{\partial \mathcal L^{(\mathcal T)}}{\partial h^{(\mathcal T)}} \cdot \frac{\partial h^{(\mathcal T)}}{\partial \widetilde{h}^{(\mathcal T)}} \cdot \frac{\partial \widetilde{h}^{(\mathcal T)}}{\partial r^{(\mathcal T)}} \cdot \frac{\partial r^{(\mathcal T)}}{\partial h^{(\mathcal T - 1)}} \cdot \frac{\partial h^{(\mathcal T - 1)}}{\partial \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}}} \end{aligned} \end{cases} ⎩ ⎨ ⎧Forward : ⎩ ⎨ ⎧h(T)=(1−Z(T))∗h(T−1)+Z(T)∗h (T)h (T)=Tanh[WH⇒H ⋅(r(T)∗h(T−1))+WX⇒H ⋅x(T)+bH ]r (T)=WH⇒r⋅h(T−1)+WX⇒r⋅x(T)+brBackward : ∂Wh(T−1)⇒Z(T−1)∂L(T)⇒∂h(T)∂L(T)⋅∂h (T)∂h(T)⋅∂r(T)∂h (T)⋅∂h(T−1)∂r(T)⋅∂Wh(T−1)⇒Z(T−1)∂h(T−1)
至此， T ⇒ T − 1 \mathcal T \Rightarrow \mathcal T - 1 T⇒T−1时刻的 5 5 5条路径已全部找全。其中：
- 1 1 1条是 T − 1 \mathcal T - 1 T−1时刻自身路径；
- 剩余 4 4 4条均是 T ⇒ T − 1 \mathcal T \Rightarrow \mathcal T - 1 T⇒T−1的传播路径。
T − 2 \mathcal T - 2 T−2时刻的反向传播路径
再往下走一步，观察它路径传播数量的规律：
- 第 1 1 1条依然是 L ( T − 2 ) \mathcal L^{(\mathcal T - 2)} L(T−2)向 W h ( T − 2 ) ⇒ Z ( T − 2 ) \mathcal W_{h^{(\mathcal T - 2)} \Rightarrow \mathcal Z^{(\mathcal T - 2)}} Wh(T−2)⇒Z(T−2)直接传递的梯度：
∂ L ( T − 2 ) ∂ W h ( T − 2 ) ⇒ Z ( T − 2 ) = ∂ L ( T − 2 ) ∂ h ( T − 2 ) ⋅ ∂ h ( T − 2 ) ∂ W h ( T − 2 ) ⇒ Z ( T − 2 ) \begin{aligned} \frac{\partial \mathcal L^{(\mathcal T - 2)}}{\partial \mathcal W_{h^{(\mathcal T - 2)} \Rightarrow \mathcal Z^{(\mathcal T - 2)}}} = \frac{\partial \mathcal L^{(\mathcal T - 2)}}{\partial h^{(\mathcal T - 2)}} \cdot \frac{\partial h^{(\mathcal T - 2)}}{\partial \mathcal W_{h^{(\mathcal T - 2)} \Rightarrow \mathcal Z^{(\mathcal T - 2)}}} \end{aligned} ∂Wh(T−2)⇒Z(T−2)∂L(T−2)=∂h(T−2)∂L(T−2)⋅∂Wh(T−2)⇒Z(T−2)∂h(T−2) - 存在 4 4 4条是从 L ( T − 1 ) \mathcal L^{(\mathcal T - 1)} L(T−1)开始，从 T − 1 ⇒ T − 2 \mathcal T - 1 \Rightarrow \mathcal T - 2 T−1⇒T−2时刻传递的路径：
∂ L ( T − 1 ) ∂ W h ( T − 2 ) ⇒ Z ( T − 2 ) = { ∂ L ( T − 1 ) ∂ h ( T − 1 ) ⋅ ∂ h ( T − 1 ) ∂ h ( T − 2 ) ⋅ ∂ h ( T − 2 ) ∂ W h ( T − 2 ) ⇒ Z ( T − 2 ) ∂ L ( T − 1 ) ∂ h ( T − 1 ) ⋅ ∂ h ( T − 1 ) ∂ Z ( T − 1 ) ⋅ ∂ Z ( T − 1 ) ∂ h ( T − 2 ) ⋅ ∂ h ( T − 2 ) ∂ W h ( T − 2 ) ⇒ Z ( T − 2 ) ∂ L ( T − 1 ) ∂ h ( T − 1 ) ⋅ ∂ h ( T − 1 ) ∂ h ~ ( T − 1 ) ⋅ ∂ h ~ ( T − 1 ) ∂ h ( T − 2 ) ⋅ ∂ h ( T − 2 ) ∂ W h ( T − 2 ) ⇒ Z ( T − 2 ) ∂ L ( T − 1 ) ∂ h ( T − 1 ) ⋅ ∂ h ( T − 1 ) ∂ h ~ ( T − 1 ) ⋅ ∂ h ~ ( T − 1 ) ∂ r ( T − 1 ) ⋅ ∂ r ( T − 1 ) ∂ h ( T − 2 ) ⋅ ∂ h ( T − 2 ) ∂ W h ( T − 2 ) ⇒ Z ( T − 2 ) \frac{\partial \mathcal L^{(\mathcal T - 1)}}{\partial \mathcal W_{h^{(\mathcal T - 2)} \Rightarrow \mathcal Z^{(\mathcal T - 2)}}} = \begin{cases} \begin{aligned} & \frac{\partial \mathcal L^{(\mathcal T - 1)}}{\partial h^{(\mathcal T - 1)}} \cdot \frac{\partial h^{(\mathcal T - 1)}}{\partial h^{(\mathcal T - 2)}} \cdot \frac{\partial h^{(\mathcal T - 2)}}{\partial \mathcal W_{h^{(\mathcal T - 2)} \Rightarrow \mathcal Z^{(\mathcal T - 2)}}} \\ & \frac{\partial \mathcal L^{(\mathcal T - 1)}}{\partial h^{(\mathcal T - 1)}} \cdot \frac{\partial h^{(\mathcal T - 1)}}{\partial \mathcal Z^{(\mathcal T - 1)}} \cdot \frac{\partial \mathcal Z^{(\mathcal T - 1)}}{\partial h^{(\mathcal T - 2)}} \cdot \frac{\partial h^{(\mathcal T - 2)}}{\partial \mathcal W_{h^{(\mathcal T - 2)} \Rightarrow \mathcal Z^{(\mathcal T - 2)}}}\\ & \frac{\partial \mathcal L^{(\mathcal T - 1)}}{\partial h^{(\mathcal T - 1)}} \cdot \frac{\partial h^{(\mathcal T - 1)}}{\partial \widetilde{h}^{(\mathcal T - 1)}} \cdot \frac{\partial \widetilde{h}^{(\mathcal T - 1)}}{\partial h^{(\mathcal T - 2)}} \cdot \frac{\partial h^{(\mathcal T - 2)}}{\partial \mathcal W_{h^{(\mathcal T - 2)} \Rightarrow \mathcal Z^{(\mathcal T - 2)}}} \\ & \frac{\partial \mathcal L^{(\mathcal T - 1)}}{\partial h^{(\mathcal T - 1)}} \cdot \frac{\partial h^{(\mathcal T - 1)}}{\partial \widetilde{h}^{(\mathcal T - 1)}} \cdot \frac{\partial \widetilde{h}^{(\mathcal T - 1)}}{\partial r^{(\mathcal T - 1)}} \cdot \frac{\partial r^{(\mathcal T - 1)}}{\partial h^{(\mathcal T - 2)}} \cdot \frac{\partial h^{(\mathcal T - 2)}}{\partial \mathcal W_{h^{(\mathcal T - 2)} \Rightarrow \mathcal Z^{(\mathcal T - 2)}}} \end{aligned} \end{cases} ∂Wh(T−2)⇒Z(T−2)∂L(T−1)=⎩ ⎨ ⎧∂h(T−1)∂L(T−1)⋅∂h(T−2)∂h(T−1)⋅∂Wh(T−2)⇒Z(T−2)∂h(T−2)∂h(T−1)∂L(T−1)⋅∂Z(T−1)∂h(T−1)⋅∂h(T−2)∂Z(T−1)⋅∂Wh(T−2)⇒Z(T−2)∂h(T−2)∂h(T−1)∂L(T−1)⋅∂h (T−1)∂h(T−1)⋅∂h(T−2)∂h (T−1)⋅∂Wh(T−2)⇒Z(T−2)∂h(T−2)∂h(T−1)∂L(T−1)⋅∂h (T−1)∂h(T−1)⋅∂r(T−1)∂h (T−1)⋅∂h(T−2)∂r(T−1)⋅∂Wh(T−2)⇒Z(T−2)∂h(T−2) - 存在 4 × 4 4 \times 4 4×4条是从 L ( T ) \mathcal L^{(\mathcal T)} L(T)开始，从 T ⇒ T − 2 \mathcal T \Rightarrow \mathcal T- 2 T⇒T−2时刻传递的路径。
这个就不写了，太墨迹了。
这仅仅是 T ⇒ T − 2 \mathcal T \Rightarrow \mathcal T - 2 T⇒T−2时刻的路径数量， T − 3 \mathcal T - 3 T−3时刻关于 L ( T ) \mathcal L^{(\mathcal T)} L(T)相关的梯度路径有 4 × 4 × 4 = 64 4 \times 4 \times 4 = 64 4×4×4=64条，以此类推。
总结
和 LSTM \text{LSTM} LSTM的反向传播路径相比， LSTM \text{LSTM} LSTM仅仅从 T ⇒ T − 2 \mathcal T \Rightarrow \mathcal T - 2 T⇒T−2时刻传递的路径就有 24 24 24条，而 GRU \text{GRU} GRU仅有 16 16 16条，相比之下，极大地减小了反向传播路径的数量；
降低了时间、空间复杂度;
其次， GRU \text{GRU} GRU相比 LSTM \text{LSTM} LSTM减少了模型参数的更新数量，降低了过拟合 ( OverFitting ) (\text{OverFitting}) (OverFitting)的风险。
它的抑制梯度消失原理与 LSTM \text{LSTM} LSTM思想相同。首先随着反向传播深度的加深，相关梯度路径依然会呈指数级别增长，但规模明显小于 LSTM \text{LSTM} LSTM；并且其梯度计算过程依然有更新门、重置门自身参与梯度运算，从而调节各梯度分量的配置情况。
相关参考：
GRU循环神经网络 —— LSTM的轻量级版本