# encoding

Sinusoidal Position Encoding

\begin{aligned} P E_{(p o s, 2 i)} &=\sin \left(\frac{p o s}{10000^{\frac{2 i}{d_{\text {model}}}}}\right) \\ P E_{(p o s, 2 i+1)} &=\cos \left(\frac{p o s}{10000^{\frac{2 i}{d_{\text {model}}}}}\right) \end{aligned}

pos + k 位置的encoding可以通过pos位置的encoding线性表示。它们的关系可以通过三角函数公式体现：

$\begin{array}{l} \sin (\alpha+\beta)=\sin \alpha \cdot \cos \beta+\cos \alpha \cdot \sin \beta \\ \cos (\alpha+\beta)=\cos \alpha \cdot \cos \beta-\sin \alpha \cdot \sin \beta \end{array}$

$\begin{array}{l} P E_{(p o s+k, 2 i)}=\sin \left(w_{i} \cdot(p o s+k)\right)=\sin \left(w_{i} p o s\right) \cos \left(w_{i} k\right)+\cos \left(w_{i} p o s\right) \sin \left(w_{i} k\right) \\ P E_{(p o s+k, 2 i+1)}=\cos \left(w_{i} \cdot(p o s+k)\right)=\cos \left(w_{i} p o s\right) \cos \left(w_{i} k\right)-\sin \left(w_{i} p o s\right) \sin \left(w_{i} k\right) \end{array} \\ w_{i}=\frac{1}{10000^{2 i / d_{\text {model}}}}$

\begin{aligned} P E_{(p o s+k, 2 i)} &=\cos \left(w_{i} k\right) P E_{(p o s, 2 i)}+\sin \left(w_{i} k\right) P E_{(p o s, 2 i+1)} \\ P E_{(p o s+k, 2 i+1)} &\left.=\cos \left(w_{i} k\right) P E_{(p o s, 2 i+1)}-\sin \left(w_{i} k\right) P E_{(p o s, 2 i)}\right) \end{aligned}

$P E_{(p o s, 2 i)} =\sin \left(pos \cdot \frac{1}{10000^{\frac{2 i}{d_{\text {model}}}}}\right) \\ T = 2 \pi \cdot 10000^{\frac{2i}{d_model}}$

# Bert 中的 positional encoding

class BertEmbeddings(nn.Module):

def __init__(self, config):
super().__init__()