HuggingFace

Pipeline  

pipeline 模块把所有东西都封装死了，只需要传进去原始输入，就能得到输出. 

例：遮掩词填空，可以看出 pipeline function 给出了 5 个备选答案   

from transformers import pipeline   

classifier = pipeline("fill-mask")    
y_pred = classifier("I love <mask> very much.") 

print(y_pred)  
"""
[
    {'score': 0.09382506459951401, 'token': 123, 'token_str': ' him', 'sequence': 'I love him very much.'}, 
    {'score': 0.06408175826072693, 'token': 47, 'token_str': ' you', 'sequence': 'I love you very much.'}, 
    {'score': 0.056255027651786804, 'token': 69, 'token_str': ' her', 'sequence': 'I love her very much.'}, 
    {'score': 0.017606642097234726, 'token': 106, 'token_str': ' them', 'sequence': 'I love them very much.'}, 
    {'score': 0.016162296757102013, 'token': 24, 'token_str': ' it', 'sequence': 'I love it very much.'}
] 
"""

　　

 

Tokenizer  

tokenizer 是分词器，对输入的单词进行预处理，可能会将单词拆开（例如，dogs 拆成 dog + s)  

一般来说，tokenizer 的处理结果和后面的大模型应当是配套的（显然，不同大模型有不同的拆分方案）  

一般来说，会有 input_ids 和 attention_mask 这两项，前面的 input_ids 就是拆分后词在语料库中的编号，然后后面 attention_mask 为 0 代表着没东西（是被 padding 的位置） 

from transformers import pipeline   
from transformers import AutoTokenizer 


checkpoint = "distilbert-base-uncased-finetuned-sst-2-English"   
tokenizer = AutoTokenizer.from_pretrained(checkpoint)    

raw_inputs = [
    "I've been waiting this for a lifetime!", 
    "I love Tom Brady."  
]

tokenized_inputs = tokenizer(raw_inputs, padding = True)  
print(tokenized_inputs) 


"""
{
    'input_ids': 
        [
            [101, 1045, 1005, 2310, 2042, 3403, 2023, 2005, 1037, 6480, 999, 102], 
            [101, 1045, 2293, 3419, 10184, 1012, 102, 0, 0, 0, 0, 0]
        ], 
    'attention_mask': 
        [
            [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 
            [1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0]
        ]
    }

"""

　　

Model   

输入的句子经过 tokenizer 的预处理后就可以喂给 model（真正的大模型） 了. 

model 的输出是未经过标准化/激活函数的向量，所以说想要得到最后的结果还需要自己写一下.   

from transformers import pipeline   
from transformers import AutoTokenizer 
from transformers import AutoModel 

checkpoint = "distilbert-base-uncased-finetuned-sst-2-English"   
tokenizer = AutoTokenizer.from_pretrained(checkpoint)    

raw_inputs = [
    "I've been waiting this for a lifetime!", 
    "I love Tom Brady."  
]

tokenized_inputs = tokenizer(raw_inputs, padding = True, return_tensors = "pt")   


model = AutoModel.from_pretrained(checkpoint)  
outputs = model(**tokenized_inputs)  

print(outputs.last_hidden_state.shape) 
# torch.Size([2, 12, 768])

例如，可以针对单词分类这个任务写一个 softmax ： 

from transformers import pipeline   
from transformers import AutoTokenizer 
from transformers import AutoModel, AutoModelForTokenClassification
import torch 

checkpoint = "distilbert-base-uncased-finetuned-sst-2-English"   
tokenizer = AutoTokenizer.from_pretrained(checkpoint)    

raw_inputs = [
    "I've been waiting this for a lifetime!", 
    "I love Tom Brady."  
]

tokenized_inputs = tokenizer(raw_inputs, padding = True, return_tensors = "pt")   


model = AutoModelForTokenClassification.from_pretrained(checkpoint)  
outputs = model(**tokenized_inputs)  
print(outputs.logits) 

predictions = torch.nn.functional.softmax(outputs.logits, dim = -1)    
print(predictions)