[吴恩达团队自然语言处理第3课_1]神经网络与情感分析 RNN

[吴恩达团队自然语言处理第3课_1]神经网络与情感分析 RNN

Neural networks

Initial Representation

空的补0

Summary

• Structure for sentiment analysis
• Classify complex tweets
• Initial representation

Trax neural networks

基于tensorflow

Advantages of using frameworks

• Run fast on CPUs,GPUs and TPUs
• Parallel computing
• Record algebraic computations for gradient evaluation

主要框架

Tensorflow Pytorch JAX

Trax layers

Classes

Classed in Python

class MyClass(Object): 
    def _init_(self,y): 
        self.y=y 
    def my_method(self,x): 
        return x+self.y 
    def _call_(self,x): 
        return self. my_method(x)
    
f = MyClass(7)
print(f(3))
#10

Subclasses

class SubClass(MyClass): 
    def my_method(self,x): 
        returnx+self.y**2
        
f = SubClass(7)
print(f(3))
#52

Dense Layer and ReLu Layer

Dense Layer

ReLu layer

Summary

• Dense Layer ->

  \[z^{[i]}=W^{[i]}a^{[i-1]} \]

• ReLu Layer ->

  \[g(z^{[i]})=max(0,z^{[i]}) \]

serial layer

• Serial layer is a composition of sublayers

  ------------>

Trax: Other Layers

Embeddding Layer

Mean Layer

减少进入下一步的数据

Summary

• Embedding is trainable using an embedding layer
• Mean layer gives a vector representation

Training

Computing gradients in Trax

Training with grad()

Summary

• grad() allows much easier training
• Forward and backpropagation in one line!

RNN

Traditional Language Models

N-grams

\[P(w_2|w_1)=\frac{count(w_1,w_2)}{count(w_1)} →Bigrams\\ P(w_3|w_1,w_2)=\frac{count(w_1,w_2,w_3)}{count(w_1,w_2)} →Trigrams\\ P(w_1,w_2,w_3)=P(w_1)\times P(w_2|w_1)\times P(w_3|w_2) \]

• Large N-grams to capture dependencies between distant words 没有大型语料库很难估计
• Need a lot of space and RAM 即使有大型语料库，也需要大量存储空间

Advantages of RNNs

have在这里并没有意义

如果使用n-grams长度会特别长

RNNS Basic Structure

可学习的参数

\[W_h,W_x,W \]

Summary

• RNNs model relationships among distant words
• In RNNs a lot of computations share parameters

Tasks

按输入输出的性质分组

One to One

输入一组不相关特征X,返回单个输出Y

如预测球队在排行榜的位置，仅具有一个隐藏状态h^<t0>,在此类任务RNN并不是那么有用

Many to One

如情感分析，tweet :I am very happy !

Many to Many

如机器翻译，RNN效果会很好，因为他们从头到尾传播信息

Encoder以单个表现形式编码单词序列，记录句子的整体含义

再解码为另一个语言的单词序列

Summary

• RNNs can be implemented for a variety of NLP tasks
• Applications include Machine translation and caption generation

Math in simple RNNs

A Vanilla RNN

$$ h^{}=g(W_{hh}h^{}\oplus W_{hx}x^{}+b_h) $$ 先关注RNN中的第一个单元 $$ W_{hh}h^{}\oplus W_{hx}x^{}+偏置\ 再通过激活函数\ 得h^{}\\ $$ $$ \hat y^{}=g(w_{yh}h^{}+by) $$ **Summary**

• Hidden states propagate information through time
• Basic recurrent units have two inputs at each time:h^<t-1>,x^<t>

Cross Entropy Loss

$$ J=-\frac{1}{T}\sum_{t=1}^T\sum_{j=1}^K y_j^{}\log \hat y_j^{}\\ $$ **Summary**

For RNNs the loss function is just an average through time!

Implementation notes

Frameworks like Tensorflow need this type of abstraction
Parallel computations and GPU usage

Summary

• Frameworks require abstractions
• tf. scan() mimics RNNs

Gated recurrent units (GRU)

Outline

• Gated recurrent unit(GRU) structure
• Comparison between GRUs and vanilla RNNs

GRU会保留主题的相关信息，如They对应ants的复数

Relevance and update gates to remember important prior information

\[\Gamma_r:relevance \ gate\ 相关门\\ \Gamma_u:update\ gate\ 更新门 \]

这些门计算Sigmode，将值压缩到0到1

Vanilla RNN vs GRUs

• RNN: 较长的序列前面的信息会丢失，即梯度消失

  

• GRUs:更长处理时间和内存使用，更新门和相关门确定之前的隐藏状态的哪些信息是相关的和应该更新哪些信息；hidden state candidates （h')存储可能用来覆盖一个从先前隐藏状态传递过来的信息；当前隐藏状态计算并更新来自是一个隐藏状态的信息；y_hat都用更新的隐藏状态得出

​

这些计算使网络能够学习什么信息需要保留，以及何时覆盖它

Summary

• GRUs "decide" how to update the hidden state

• GRUs help preserve important information

  GRU是LSTM的简化版本

deep and Bi-directional RNNs

Outline

• How bidirectional RNNs propagate information
• Forward propagation in deep RNNs

<img src="https://img2022.cnblogs.com/blog/1586717/202202/1586717-20220225153205607-313797841.png" alt="image-20220225151714944" style="zoom:67%;" /

Bi-directional

Deep RNNs

多个RNN一起

Summary

• In bidirectional RNNs,,the outputs take information from the past and the future
• Deep RNNs have more than one layer,which helps in complex tasks