循环神经网络（数据思维赛-安全文本信息抽取）

训练集有四列数据：

 id为序号，sentence_idx为语句的序号，words为从一条语句中解析出的单词，tag为每个word对应的tag标签。

 

test集给出前三列数据，求每一个word对应的tag数据。

 

使用循环神经网络算法：

import pandas as pd
from sklearn.model_selection import train_test_split
from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences
from keras.models import Sequential
from keras.layers import Embedding, Dense, Flatten
from tensorflow.keras.utils import to_categorical

# 加载训练数据
train_data = pd.read_csv('train.csv',converters={i: str for i in range(0, 100)})

# 创建词汇表和标签表
tokenizer = Tokenizer()
tokenizer.fit_on_texts(train_data['words'])
word_vocab_size = len(tokenizer.word_index) + 1
tag_vocab = sorted(list(set(train_data['tag'].values)))

tag2idx = {t: i for i, t in enumerate(tag_vocab)}

# 将句子和标签转换为序列
X = tokenizer.texts_to_sequences(train_data['words'])
y = [tag2idx[tag] for tag in train_data['tag']]
sentence_idx = train_data['sentence_idx'].values

# 对序列进行填充
max_len = max(len(sentence) for sentence in X)
X_pad = pad_sequences(X, maxlen=max_len)

# 将标签转换为 one-hot 编码
y_onehot = to_categorical(y)

# 划分训练集和验证集
X_train, X_val, y_train, y_val, sentence_idx_train, sentence_idx_val = train_test_split(
    X_pad, y_onehot, sentence_idx, test_size=0.2, random_state=42)

# 构建神经网络模型
model = Sequential()
model.add(Embedding(word_vocab_size, 100, input_length=max_len))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dense(len(tag_vocab), activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

# 训练模型
model.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=10, batch_size=32)

# 加载测试数据
test_data = pd.read_csv('test.csv',converters={i: str for i in range(0, 100)})

# 将测试数据转换为序列并填充
X_test = tokenizer.texts_to_sequences(test_data['words'])
X_test_pad = pad_sequences(X_test, maxlen=max_len)

# 使用训练好的模型进行预测
predictions = model.predict_classes(X_test_pad)

# 将预测结果保存到result.csv中
result_df = pd.DataFrame({'id': test_data['id'], 'sentence_idx': test_data['sentence_idx'],
                          'words': test_data['words'], 'tag': [tag_vocab[prediction] for prediction in predictions]})
result_df.to_csv('result.csv', index=False)