【代码粗读】Simultaneously Self-Attending to All Mentions for Full-Abstract Biological Relation Extraction

代码地址：https://github.com/patverga/bran

今天下载这篇论文的代码，着实把我惊住了，工程量庞大，脚本文件太多看花眼，而且代码使用python2 写的，和我python3环境不搭。

先记录下调试过程中的坑。

Setup Environment Variables

From this directory call: source set_environment.sh
Note: this will only set the paths for this session.

#!/usr/bin/env bash

export PYTHONPATH=$PYTHONPATH:`pwd`
export CDR_IE_ROOT=`pwd`

主要是通过'pwd'获得当前文件夹的目录。用shell中变量CDR_IE_ROOT保存

Processing Data

CDR

Process the CDR dataset
${CDR_IE_ROOT}/bin/process_CDR/process_CDR.sh

Process the CDR dataset including additional weakly labeled data
${CDR_IE_ROOT}/bin/process_CDR/process_CDR_extra_data.sh

These scripts will use byte-pair encoding (BPE) tokenization. There are also scripts to tokenize using the Genia tokenizer.

#!/usr/bin/env bash

#CDR_IE_ROOT = 'D:/python/wgy_jupyter/bran-master'
word_piece_vocab=data/cdr/word_piece_vocabs/just_train_2500/word_pieces.txt
input_dir=${CDR_IE_ROOT}/data/cdr
processed_dir=${input_dir}/processed/just_train_2500
proto_dir=${processed_dir}/protos
max_len=500000
# replace infrequent tokens with <UNK>
min_count=5


# process train, dev, and test data
mkdir -p ${processed_dir}
echo "Processing Training data"
python ${CDR_IE_ROOT}/src/processing/utils/process_CDR_data.py \
        --input_file ${input_dir}/CDR_TrainingSet.PubTator.txt.gz \
        --output_dir ${processed_dir} --output_file_suffix CDR_train.txt \
        --max_seq ${max_len} \
        --full_abstract True --word_piece_codes ${word_piece_vocab}

for f in CDR_dev CDR_train CDR_test; do
python ${CDR_IE_ROOT}/src/processing/utils/filter_hypernyms.py \
　　　　 -p ${processed_dir}/positive_0_${f}.txt \
　　　　 -n ${processed_dir}/negative_0_${f}.txt \
　　　　 -m data/2017MeshTree.txt \
　　　　 -o ${processed_dir}/negative_0_${f}_filtered.txt
done

拿训练集来举例：根据sh脚本文件的命令我们通过加载TraningSet.PubTator.txt.gz并与用分词工具BPE，得到关于训练集的命名实体训练集，正样本，负样本（过滤/未过滤）

word_pieces是一些切开分词的内容：

#version: 0.2
i n
t i
r e
t h
a n
e n
e r
e d</w>
o n
o f</w>
a t
th e</w>
r o
i n</w>
a ti
i t
an d</w>
a r
o n</w>
s t
s i
a l
a l</w>
o r
i c
a s
e c
d i
a c
en t
l o
in g</w>
t o</w>

生成的数据集文件目录：

其中因为python2与python3的语法问题，Error: a bytes-like object is required, not 'str' 

问题出在python3.5和Python2.7在套接字返回值解码上有区别:
python bytes和str两种类型可以通过函数encode()和decode()相互转换，
str→bytes：encode()方法。str通过encode()方法可以转换为bytes。
bytes→str：decode()方法。如果我们从网络或磁盘上读取了字节流，那么读到的数据就是bytes。要把bytes变为str，就需要用decode()方法。

解决方法：

将line.strip().split(",")  改为  line.decode().strip().split(",")【这分明和报错写的意思相反！】

也不知道为什么同一文件夹目录下

from word_piece_tokenizer import WordPieceTokenizer

语句导入同级别的包会标出红线报错，但是不影响使用，但在Pycharm中无法ctrl+鼠标索引；

from .word_piece_tokenizer import WordPieceTokenizer

这样同级别包下虽然不会红线报错，但代码无法运行，但在Pycharm中能够ctrl+鼠标索引.

代码使用Tensorflow 1.X写的所以没太看懂，只找到对应论文的模型部分看下作者如何实现的：

2.1 Inputs

    def get_token_embeddings(self, token_embeddings, position_embeddings, token_attention=None):
        selected_words = tf.nn.embedding_lookup(token_embeddings, self.text_batch)
        if self.project_inputs:
            params = {"inputs": selected_words, "filters": self.embed_dim, "kernel_size": 1,
                      "activation": tf.nn.relu, "use_bias": True}
            selected_words = tf.layers.conv1d(**params)
        if self.position_dim > 0:
            selected_e1_dists = tf.nn.embedding_lookup(position_embeddings, self.e1_dist_batch)
            selected_e2_dists = tf.nn.embedding_lookup(position_embeddings, self.e2_dist_batch)
            token_embeds = tf.concat(axis=2, values=[selected_words, selected_e1_dists, selected_e2_dists])
        else:
            token_embeds = selected_words

        if self.encode_position:
            pad_mask = tf.expand_dims(tf.cast(tf.not_equal(self.text_batch, self.pad_idx), tf.float32), [2])
            pos_encoding = pad_mask * tf.nn.embedding_lookup(self.pos_encoding, self.pos_encode_batch)
            token_embeds = tf.add(token_embeds, pos_encoding)

        dropped_embeddings = tf.nn.dropout(token_embeds, self.word_dropout_keep)
        # keep pad tokens as 0 vectors
        if self.filter_pad:
            print('Filtering pad tokens')
            dropped_embeddings = tf.multiply(dropped_embeddings,
                                             tf.expand_dims(tf.cast(tf.not_equal(self.text_batch, self.pad_idx),
                                                                    tf.float32), [2]))

        return dropped_embeddings

2.2 Transformer

    def forward(self, input_feats, middle_dropout_keep_prob, hidden_dropout_keep_prob, batch_size, max_seq_len,
                reuse, block_num=0):
        initial_in_dim =self.embed_dim if self.project_inputs else self.token_dim+(2*self.position_dim)
        input_feats *= (initial_in_dim ** 0.5)     #开根号
        self.attention_weights = []
        for i in range(self.block_repeats):     #B个transformer块
            with tf.variable_scope("num_blocks_{}".format(i)):
                ### Multihead Attention
                in_dim = self.embed_dim if i > 0 else initial_in_dim
                input_feats = self.multihead_attention(queries=input_feats,
                                                  keys=input_feats,
                                                  num_units=in_dim,
                                                  num_heads=self.num_heads,
                                                  dropout_rate=middle_dropout_keep_prob,
                                                  causality=False,
                                                  reuse=reuse)

                ### Feed Forward
                input_feats = self.feedforward(input_feats, num_units=[in_dim*self.ff_scale, in_dim], reuse=reuse)
        return input_feats

2.2.1 Multi-head Attention

    def multihead_attention(self, queries,
                            keys,
                            num_units=None,
                            num_heads=8,
                            dropout_rate=0,
                            is_training=True,
                            causality=False,
                            scope="multihead_attention",
                            reuse=None):
        '''
        June 2017 by kyubyong park.
        kbpark.linguist@gmail.com.
        https://www.github.com/kyubyong/transformer
        '''
        '''Applies multihead attention.

        Args:
          queries: A 3d tensor with shape of [N, T_q, C_q].
          keys: A 3d tensor with shape of [N, T_k, C_k].
          num_units: A scalar. Attention size.
          dropout_rate: A floating point number.
          is_training: Boolean. Controller of mechanism for dropout.
          causality: Boolean. If true, units that reference the future are masked.
          num_heads: An int. Number of heads.
          scope: Optional scope for `variable_scope`.
          reuse: Boolean, whether to reuse the weights of a previous layer
            by the same name.

        Returns
          A 3d tensor with shape of (N, T_q, C)
        '''
        with tf.variable_scope(scope, reuse=reuse):
            # Set the fall back option for num_units
            if num_units is None:
                num_units = queries.get_shape().as_list[-1]

            # Linear projections
            Q = tf.layers.dense(queries, num_units, activation=tf.nn.relu) # (N, T_q, C)
            K = tf.layers.dense(keys, num_units, activation=tf.nn.relu) # (N, T_k, C)
            V = tf.layers.dense(keys, num_units, activation=tf.nn.relu) # (N, T_k, C)

            # Split and concat
            Q_ = tf.concat(tf.split(Q, num_heads, axis=2), axis=0) # (h*N, T_q, C/h)
            K_ = tf.concat(tf.split(K, num_heads, axis=2), axis=0) # (h*N, T_k, C/h)
            V_ = tf.concat(tf.split(V, num_heads, axis=2), axis=0) # (h*N, T_k, C/h)

            # Multiplication
            outputs = tf.matmul(Q_, tf.transpose(K_, [0, 2, 1])) # (h*N, T_q, T_k)

            # Scale
            outputs = outputs / (K_.get_shape().as_list()[-1] ** 0.5)

            # Key Masking
            key_masks = tf.sign(tf.abs(tf.reduce_sum(keys, axis=-1))) # (N, T_k)
            key_masks = tf.tile(key_masks, [num_heads, 1]) # (h*N, T_k)
            key_masks = tf.tile(tf.expand_dims(key_masks, 1), [1, tf.shape(queries)[1], 1]) # (h*N, T_q, T_k)

            paddings = tf.ones_like(outputs)*(-1e8)
            outputs = tf.where(tf.equal(key_masks, 0), paddings, outputs) # (h*N, T_q, T_k)

            # Activation
            attention_weights = tf.nn.softmax(outputs) # (h*N, T_q, T_k)
            # store the attention weights for analysis
            batch_size = tf.shape(queries)[0]
            seq_len = tf.shape(queries)[1]
            save_attention = tf.reshape(attention_weights, [self.num_heads, batch_size, seq_len, seq_len])
            save_attention = tf.transpose(save_attention, [1, 0, 2, 3])
            self.attention_weights.append(save_attention)

            # Query Masking
            query_masks = tf.sign(tf.abs(tf.reduce_sum(queries, axis=-1))) # (N, T_q)
            query_masks = tf.tile(query_masks, [num_heads, 1]) # (h*N, T_q)
            query_masks = tf.tile(tf.expand_dims(query_masks, -1), [1, 1, tf.shape(keys)[1]]) # (h*N, T_q, T_k)
            outputs = attention_weights * query_masks # broadcasting. (N, T_q, C)

            # Dropouts
            outputs = tf.nn.dropout(outputs, dropout_rate)

            # Weighted sum
            outputs = tf.matmul(outputs, V_) # ( h*N, T_q, C/h)

            # Restore shape
            outputs = tf.concat(tf.split(outputs, num_heads, axis=0), axis=-1 ) # (N, T_q, C)

            # Residual connection
            outputs += tf.nn.dropout(queries, dropout_rate)

            # Normalize
            outputs = self.normalize(outputs) # (N, T_q, C)

        return outputs

2.2.2  Convolutions

    def feedforward(self, inputs,
                    num_units=[2048, 512],
                    scope="multihead_attention",
                    reuse=None):

        '''Point-wise feed forward net.

        Args:
          inputs: A 3d tensor with shape of [N, T, C].
          num_units: A list of two integers.
          scope: Optional scope for `variable_scope`.
          reuse: Boolean, whether to reuse the weights of a previous layer
            by the same name.

        Returns:
          A 3d tensor with the same shape and dtype as inputs
        '''
        with tf.variable_scope(scope, reuse=reuse):
            outputs = inputs
            layer_params = self.layer_str.split(',')
            for i, l_params in enumerate(layer_params):
                width, dilation = [int(x) for x in l_params.split(':')]
                dim = num_units[1] if i == (len(layer_params)-1) else num_units[0]

                print('dimension: %d  width: %d  dilation: %d' % (dim, width, dilation))
                params = {"inputs": outputs, "filters": dim, "kernel_size": width,
                          "activation": tf.nn.relu, "use_bias": True, "padding": "same", "dilation_rate": dilation}
                outputs = tf.layers.conv1d(**params)
            # mask padding
            outputs *= tf.expand_dims(tf.cast(tf.not_equal(self.text_batch, self.pad_idx), tf.float32), [2])
            # Residual connection
            inputs += outputs

            # Normalize
            outputs = self.normalize(outputs)

        return outputs

2.3  Bi-affine Pairwise Scores

    def aggregate_tokens(self, encoded_tokens, batch_size, max_seq_len,
                         attention_vector, e1_dist_batch, e2_dist_batch, seq_lens,
                         middle_dropout_keep_prob, hidden_dropout_keep_prob, final_dropout_keep,
                         scope_name='text', reuse=False, aggregation='attention'):

        reduction = tf.reduce_logsumexp
        # # aggregation='attention'
        with tf.variable_scope(scope_name, reuse=reuse):
            input_feats = encoded_tokens

            e1_mask = tf.cast(tf.expand_dims(tf.equal(self.e1_dist_batch, self.entity_index), 2), tf.float32)
            e2_mask = tf.cast(tf.expand_dims(tf.equal(self.e2_dist_batch, self.entity_index), 2), tf.float32)

            # # b x s x (d*l)
            e1 = tf.layers.dense(tf.layers.dense(input_feats, self.embed_dim, activation=tf.nn.relu), self.embed_dim)
            e2 = tf.layers.dense(tf.layers.dense(input_feats, self.embed_dim, activation=tf.nn.relu), self.embed_dim)

            e1 = tf.nn.dropout(e1, final_dropout_keep)
            e2 = tf.nn.dropout(e2, final_dropout_keep)

            # result = self.diagonal_bilinear(e1, e2, num_labels)
            pairwise_scores = self.bilinear(e1, e2, self.num_labels)
            # self.attention_weights = tf.split(self.bilinear_scores, self.num_labels, 2)[1]
            self.pairwise_scores = tf.nn.softmax(pairwise_scores, dim=2)
            result = tf.transpose(pairwise_scores, [0, 1, 3, 2])
            # mask result
            result += tf.expand_dims(self.ep_dist_batch, 3)
            outputs = reduction(result, [1, 2])
            print(outputs.get_shape())

            return outputs

2.4  Entity Level Prediction

    def bilinear(self, inputs1, inputs2, output_size, add_bias2=True, add_bias1=True, add_bias=False,
                 initializer=None, scope=None, moving_params=None):
        """"""
        with tf.variable_scope(scope or 'Bilinear'):
            # Reformat the inputs
            ndims = len(inputs1.get_shape().as_list())
            inputs1_shape = tf.shape(inputs1)
            inputs1_bucket_size = inputs1_shape[ndims-2]
            inputs1_size = inputs1.get_shape().as_list()[-1]

            inputs2_shape = tf.shape(inputs2)
            inputs2_bucket_size = inputs2_shape[ndims-2]
            inputs2_size = inputs2.get_shape().as_list()[-1]
            output_shape = []
            batch_size = 1
            for i in range(ndims-2):
                batch_size *= inputs1_shape[i]
                output_shape.append(inputs1_shape[i])
            output_shape.append(inputs1_bucket_size)
            output_shape.append(output_size)
            output_shape.append(inputs2_bucket_size)
            inputs1 = tf.reshape(inputs1, [batch_size, inputs1_bucket_size, inputs1_size])
            inputs2 = tf.reshape(inputs2, [batch_size, inputs2_bucket_size, inputs2_size])
            if add_bias1:
                inputs1 = tf.concat([inputs1, tf.ones([batch_size, inputs1_bucket_size, 1])], 2)
            if add_bias2:
                inputs2 = tf.concat([inputs2, tf.ones([batch_size, inputs2_bucket_size, 1])], 2)

            # Get the matrix
            if initializer is None and moving_params is None:
                mat = orthonormal_initializer(inputs1_size+add_bias1, inputs2_size+add_bias2)[:,None,:]
                mat = np.concatenate([mat]*output_size, axis=1)
                initializer = tf.constant_initializer(mat)
            weights = tf.get_variable('Weights', [inputs1_size+add_bias1, output_size, inputs2_size+add_bias2], initializer=initializer)
            if moving_params is not None:
                weights = moving_params.average(weights)
            else:
                tf.add_to_collection('Weights', weights)

            # Do the multiplications
            # (bn x d) (d x rd) -> (bn x rd)
            lin = tf.matmul(tf.reshape(inputs1, [-1, inputs1_size+add_bias1]),
                            tf.reshape(weights, [inputs1_size+add_bias1, -1]))
            # (b x nr x d) (b x n x d)T -> (b x nr x n)
            bilin = tf.matmul(tf.reshape(lin, [batch_size, inputs1_bucket_size*output_size, inputs2_size+add_bias2]),
                                    inputs2, adjoint_b=True)
            # (bn x r x n)
            bilin = tf.reshape(bilin, [-1, output_size, inputs2_bucket_size])
            # (b x n x r x n)
            bilin = tf.reshape(bilin, output_shape)

            # Get the bias
            if add_bias:
                bias = tf.get_variable('Biases', [output_size], initializer=tf.zeros_initializer)
                if moving_params is not None:
                    bias = moving_params.average(bias)
                bilin += tf.expand_dims(bias, 1)

            return bilin

由aggregate_tokens()得到的实体对score结果传入下列第19行代码：

    def embed_text_from_tokens(self, selected_col_embeddings, attention_vector, e1_dist_batch, e2_dist_batch, seq_lens,
                               middle_dropout_keep_prob, hidden_dropout_keep_prob, final_dropout_keep,
                               scope_name='text', reuse=False, aggregation='piecewise', return_tokens=False):
        batch_size = tf.shape(selected_col_embeddings)[0]
        max_seq_len = tf.shape(selected_col_embeddings)[1]

        output = []
        last_output = selected_col_embeddings
        if not reuse:
            print('___aggregation type:  %s filter %d  block repeats: %d___'
                  % (aggregation, self.filter_width, self.block_repeats))
        for i in range(1):
            block_reuse = (reuse if i == 0 else True)
            encoded_tokens = self.forward(last_output, middle_dropout_keep_prob,
                                          hidden_dropout_keep_prob, batch_size, max_seq_len, block_reuse, i)
            if return_tokens:
                output.append(encoded_tokens)
            else:
                encoded_seq = self.aggregate_tokens(encoded_tokens, batch_size, max_seq_len,
                                                    attention_vector, e1_dist_batch, e2_dist_batch, seq_lens,
                                                    middle_dropout_keep_prob, hidden_dropout_keep_prob, final_dropout_keep,
                                                    scope_name=scope_name, reuse=block_reuse, aggregation=aggregation)
                output.append(encoded_seq)
            last_output = encoded_tokens
        return output

由aggregate_tokens()得到的结果传入下列第12行代码：

    def embed_text(self, token_embeddings, position_embeddings, attention_vector,   #最终结果
                   scope_name='text', reuse=False, aggregation='attention', no_dropout=False,
                   return_tokens=False, token_attention=None,):
        selected_col_embeddings = self.get_token_embeddings(token_embeddings, position_embeddings, token_attention)
        if no_dropout:
            middle_dropout_keep_prob, hidden_dropout_keep_prob, final_dropout_keep = 1.0, 1.0, 1.0
        else:
            middle_dropout_keep_prob = self.middle_dropout_keep_prob
            hidden_dropout_keep_prob = self.hidden_dropout_keep_prob
            final_dropout_keep = self.final_dropout_keep

        output = self.embed_text_from_tokens(
                selected_col_embeddings, attention_vector,
                self.e1_dist_batch, self.e2_dist_batch, self.seq_len_batch,
                middle_dropout_keep_prob, hidden_dropout_keep_prob, final_dropout_keep,
                scope_name, reuse, aggregation, return_tokens=return_tokens)
        return output

使用embed_text()的预测结果与标签做梯度下降优化模型：

encoded_text_list = self.text_encoder.embed_text(self.token_embeddings, self.position_embeddings,
                                                             self.attention_vector, token_attention=token_attention)

self.loss = self.calculate_loss(encoded_text_list, predictions_list, self.label_batch,
                                            FLAGS.l2_weight, FLAGS.dropout_loss_weight, no_drop_output_list)