青岛胶南做网站的有多少,网页升级访问自动,wordpress怎么禁用多域名,工作室官网模板引言
今天我们来实现RE2进行文本匹配#xff0c;模型实现参考了官方代码https://github.com/alibaba-edu/simple-effective-text-matching-pytorch。
模型实现 RE2模型架构如上图所示。它的输入是两个文本片段#xff0c;所有组件参数除了预测层和对齐层外都是共享的。上图…引言
今天我们来实现RE2进行文本匹配模型实现参考了官方代码https://github.com/alibaba-edu/simple-effective-text-matching-pytorch。
模型实现 RE2模型架构如上图所示。它的输入是两个文本片段所有组件参数除了预测层和对齐层外都是共享的。上图虚线框出来的为一个Block堆叠了N个block文本片段之间的block内部通过对齐层进行交互。block之间通过增加的残差层进行连接。
下面我们从底向上依次实现实现过程中参考了官方实现。
Embedding
嵌入层很简单没有使用字符嵌入就是简单的单词嵌入。
class Embedding(nn.Module):def __init__(self, vocab_size: int, embedding_dim: int, dropout: float) - None:super().__init__()self.embedding nn.Embedding(vocab_size, embedding_dim, padding_idx0)self.dropout nn.Dropout(dropout)def forward(self, x: Tensor) - Tensor:Args:x (Tensor): (batch_size, seq_len)Returns:Tensor: (batch_size, seq_len, embedding_dim)return self.dropout(self.embedding(x))Encoder
GeLU
首先实现GeLU它是RELU的变种后来被用到BERT中。其函数图像如下所示 class GeLU(nn.Module):def forward(self, x: Tensor) - Tensor:return 0.5 * x * (1.0 torch.tanh(x * 0.7978845608 * (1.0 0.044715 * x * x)))Linear
重写了线性层activations开启GeLU激活。
class Linear(nn.Module):def __init__(self, in_features: int, out_features: int, activations: bool True) - None:super().__init__()linear nn.Linear(in_features, out_features)modules [weight_norm(linear)]if activations:modules.append(GeLU())self.model nn.Sequential(*modules)self.reset_parameters(activations)def reset_parameters(self, activations: bool) - None:linear self.model[0]nn.init.normal_(linear.weight,stdmath.sqrt((2.0 if activations else 1.0) / linear.in_features),)nn.init.zeros_(linear.bias)def forward(self, x):return self.model(x)nn.Conv1d
我们在比较聚合模型的实现中详细了解了torch.nn.Conv2d的实现以及CNN的一些基础概念。
这里我们通过torch.nn.Conv1d来实现论文中的多层卷积网络本小结来详细了解Conv1d实现。
torch.nn.Conv1din_channels: 输入的通道数文本中为嵌入维度out_channels 一个卷积核产生一个输出通道kernel_size: 卷积核的大小stride: 卷积步长,默认为1padding: 填充默认为0bias(bool): 是否添加偏置默认为True我们以一个例子来说明它的计算过程假设对于输入W B G 是 冠 军随机得到的嵌入为 希望今天下午S13 WBG可以战胜T1。 import numpy as np
import torch.nn as nn
import torchbatch_size 1
seq_len 6
embed_size 3input_tensor torch.rand(batch_size, seq_len, embed_size)
print(input_tensor)
print(input_tensor.shape)tensor([[[0.9291, 0.8333, 0.5160],[0.0543, 0.8149, 0.5704],[0.7831, 0.2263, 0.9279],[0.0898, 0.0758, 0.4401],[0.4321, 0.2098, 0.6666],[0.6183, 0.0609, 0.2330]]])
torch.Size([1, 6, 3])此时每个字符对应一个3维的嵌入向量分别为
W — [0.9291, 0.8333, 0.5160]
B — [0.0543, 0.8149, 0.5704]
G — [0.7831, 0.2263, 0.9279]
是 — [0.0898, 0.0758, 0.4401]
冠 — [0.4321, 0.2098, 0.6666]
军 — [0.6183, 0.0609, 0.2330]但是Conv1d需要in_channels即嵌入维度为仅在batch_size后第一个位置由[1, 6, 3]变成[1, 3, 6]。
input_tensor input_tensor.permute(0, 2, 1)
# (batch_size, embed_size, seq_len)图示如下 文章还没发结果被3:0了。 然后我们定义一个一维卷积
input_channels embed_size # 等于embed_size
output_channels 2
kernel_size 2 # kernel_sizeconv1d nn.Conv1d(in_channelsinput_channels, out_channelsoutput_channels, kernel_sizekernel_size)我们可以打印出来filter权重矩阵
print(conv1d.weight)
print(conv1d.weight.shape)Parameter containing:
tensor([[[ 0.0025, 0.3353],[ 0.0620, -0.3916],[-0.3458, -0.0610]],[[-0.1731, -0.0787],[-0.0419, -0.2555],[-0.1429, 0.1656]]], requires_gradTrue)
torch.Size([2, 3, 2])filter 权重的大小为 (2,3,2) shape[0]2是filter个数shape[1]3是输入嵌入大小shape[2]2是filter大小。
默认是添加了偏置一个filter一个偏置
Parameter containing:
tensor([ 0.3760, -0.2881], requires_gradTrue)
torch.Size([2])我们这里有两个filter所以有两个偏置。因为这里kernel_size2且步长stride1所以一个filter是如下的方式框住了两个字符嵌入并且每次向右移动一格 此时第一个filter的卷积操作计算为
sum([[0.9291, 0.0543], [[0.0025, 0.3353],[0.8333, 0.8149], * [0.0620, -0.3916], 0.3760(bias)[0.5160, 0.5704]] [-0.3458, -0.0610])第一个filter权重和这两个嵌入进行逐位置乘法产生一个标量(sum)最后加上第一个filter的偏置。
通过代码实现为
# 开始计算卷积
# 前两个嵌入与卷积核权重逐元素乘法
result input_tensor[:,:,:2]*conv1d.weight
print(result)
# 结果求和再加上偏置
print(torch.sum(result[0]) conv1d.bias[0])
print(torch.sum(result[1]) conv1d.bias[1])tensor([[[ 0.0024, 0.0182],[ 0.0517, -0.3191],[-0.1784, -0.0348]],[[-0.1608, -0.0043],[-0.0349, -0.2082],[-0.0737, 0.0944]]], grad_fnMulBackward0)tensor(-0.0841, grad_fnAddBackward0) # 第一个filter的结果
tensor(-0.6756, grad_fnAddBackward0) # 第二个filter的结果这是第一次卷积的结果第二次卷积把红框向右移动一格又会有一个结果。 最终移动到输入的最后一个位置计算完毕 共需要计算5次因此最终一个filter会输出5个标量共有2个filter批大小为1。
如果用代码实现的话
output conv1d(input_tensor)
print(output)
print(output.shape)tensor([[[-0.0841, 0.3468, 0.0447, 0.2508, 0.3288],[-0.6756, -0.3790, -0.5193, -0.3470, -0.4926]]],grad_fnConvolutionBackward0)
torch.Size([1, 2, 5])可以看到output的形状为[1, 2, 5]第一列的计算结果和我们上面的一致。
shape[0]1是批次内样本个数shape[1]2是filter个数也是想要输出的channel数shape[2]5是卷积后的维度。
这里(忽略dilation)卷积后的维度大小由卷积核大小kernel_size、步长stride、填充padding以及输入序列长度seq_len决定 ⌊ seq_len 2 × padding − kernel_size stride 1 ⌋ \left\lfloor \frac{\text{seq\_len} 2 \times\text{padding} - \text{kernel\_size}}{\text{stride}} 1\right\rfloor ⌊strideseq_len2×padding−kernel_size1⌋
其中 ⌊ ∗ ⌋ \lfloor * \rfloor ⌊∗⌋表示向下取整。
我们可以代入上面的参数验证一下 6 2 × 0 − 2 1 1 6 0 − 2 1 5 \frac{6 2\times 0 - 2}{1} 1 60-215 162×0−2160−215 结果是5和输出一致。
Conv1d
下面实现RE2的多层卷积网络首先是一个改写的Conv1d用weight_norm进行权重归一化采用GeLU激活函数。
class Conv1d(nn.Module):def __init__(self, in_channels: int, out_channels: int, kernel_sizes: list[int]) - None:Args:in_channels (int): the embedding_dimout_channels (int): number of filterskernel_sizes (list[int]): the size of kernelsuper().__init__()out_channels out_channels // len(kernel_sizes)convs []# L_in is seq_len, L_out is output_dim of conv# L_out (L_in 2 * padding - kernel_size 1)# and padding(kernel_size - 1) // 2# L_out (L_in kernel_size - 1 - kernel_size 1) L_infor kernel_size in kernel_sizes:conv nn.Conv1d(in_channels, out_channels, kernel_size, padding(kernel_size - 1) // 2)convs.append(nn.Sequential(weight_norm(conv), GeLU()))# output shape of each conv is (batch_size, out_channels(new), seq_len)self.model nn.ModuleList(convs)self.reset_parameters()def reset_parameters(self) - None:for seq in self.model:conv seq[0]nn.init.normal_(conv.weight,stdmath.sqrt(2.0 / (conv.in_channels * conv.kernel_size[0])),)nn.init.zeros_(conv.bias)def forward(self, x: Tensor) - Tensor:Args:x (Tensor): shape (batch_size, embedding_dim, seq_len)Returns:Tensor:# back to (batch_size, out_channels, seq_len)return torch.cat([encoder(x) for encoder in self.model], dim1)用out_channels // len(kernel_sizes)将输出大小拆分最后用torch.cat将它们拼接回out_channels
padding(kernel_size - 1) // 2目的是使得卷积后的维度大小和输入的seq_len一致这里需要kernel_size 是奇数因为padding只能接收整数。
weight_norm将权重分解为大小和方向可以加速训练过程并提高模型的泛化能力。保留原先的权重方向大小由权重归一化层自己学习 w g v ∣ ∣ v ∣ ∣ \pmb w g\frac{\pmb v}{||\pmb v||} wg∣∣v∣∣v
Encoder实现
class Encoder(nn.Module):def __init__(self,input_size: int,hidden_size: int,kernel_sizes: list[int],encoder_layers: int,dropout: float,) - None:_summary_Args:input_size (int): embedding_dim or embedding_dim hidden_sizehidden_size (int): hidden sizekernel_sizes (list[int]): the size of kernelsencoder_layers (int): number of conv layersdropout (float): dropout ratiosuper().__init__()self.encoders nn.ModuleList([Conv1d(in_channelsinput_size if i 0 else hidden_size,out_channelshidden_size,kernel_sizeskernel_sizes,)for i in range(encoder_layers)])self.dropout nn.Dropout(dropout)def forward(self, x: Tensor, mask: Tensor) - Tensor:forward in encoderArgs:x (Tensor): (batch_size, seq_len, input_size)mask (Tensor): (batch_size, seq_len, 1)Returns:Tensor: _description_# x (batch_size, input_size, seq_len)x x.transpose(1, 2)# mask (batch_size, 1, seq_len)mask mask.transpose(1, 2)for i, encoder in enumerate(self.encoders):# fills elements of x with 0.0 where mask is Falsex.masked_fill_(~mask, 0.0)# using dropoutif i 0:x self.dropout(x)# returned x (batch_size, hidden_size, seq_len)x encoder(x)# apply dropoutx self.dropout(x)# (batch_size, seq_len, hidden_size)return x.transpose(1, 2)这里用多层Conv1d作为编码器要注意第0层和其他层的区别第0层的嵌入维度是input_size即embedding_size经过第0层的Conv1d后维度变成两hidden_size所以后续层参数in_channels为hidden_size。
这里用x.masked_fill_(~mask, 0.0)设置mask矩阵中的填充位为0。
不采用RNN作为编码器作者认为RNN速度慢且没有带来性能上的提升。
Alignment
然后实现对齐层所谓的对齐就是让两个序列进行交互这里采用基于注意力交互的方式。
class Alignment(nn.Module):def __init__(self, input_size: int, hidden_size: int, dropout: float, project_func: str) - None:Args:input_size (int): embedding_dim hidden_size or embedding_dim hidden_size * 2hidden_size (int): hidden sizedropout (float): dropout ratioproject_func (str): identity or linearsuper().__init__()self.temperature nn.Parameter(torch.tensor(1 / math.sqrt(hidden_size)))if project_func ! identity:self.projection nn.Sequential(nn.Dropout(dropout), Linear(input_size, hidden_size))else:self.projection nn.Identity()def forward(self, a: Tensor, b: Tensor, mask_a: Tensor, mask_b: Tensor) - Tensor:Args:a (Tensor): (batch_size, seq_len, input_size)b (Tensor): (batch_size, seq_len, input_size)mask_a (Tensor): (batch_size, seq_len, 1)mask_b (Tensor): (batch_size, seq_len, 1)Returns:Tensor: _description_# if projection linear : self.projection(*) - (batch_size, seq_len, hidden_size) - transpose(*) - (batch_size, hidden_size, seq_len)# if projection identity : self.projection(*) - (batch_size, seq_len, input_size) - transpose(*) - (batch_size, input_size, seq_len)# attn (batch_size, seq_len_a, seq_len_b)attn (torch.matmul(self.projection(a), self.projection(b).transpose(1, 2))* self.temperature)# mask (batch_size, seq_len_a, seq_len_b)mask torch.matmul(mask_a.float(), mask_b.transpose(1, 2).float())mask mask.bool()# fills elements of x with 0.0(after exp) where mask is Falseattn.masked_fill_(~mask, -1e7)# attn_a (batch_size, seq_len_a, seq_len_b)attn_a F.softmax(attn, dim1)# attn_b (batch_size, seq_len_a, seq_len_b)attn_b F.softmax(attn, dim2)# feature_b (batch_size, seq_len_b, seq_len_a) x (batch_size, seq_len_a, input_size)# - (batch_size, seq_len_b, input_size)feature_b torch.matmul(attn_a.transpose(1, 2), a)# feature_a (batch_size, seq_len_a, seq_len_b) x (batch_size, seq_len_b, input_size)# - (batch_size, seq_len_a, input_size)feature_a torch.matmul(attn_b, b)return feature_a, feature_b
增强残差连接 class AugmentedResidualConnection(nn.Module):def __init__(self) - None:super().__init__()def forward(self, x: Tensor, res: Tensor, i: int) - Tensor:Args:x (Tensor): the output of pre block (batch_size, seq_len, hidden_size)res (Tensor): (batch_size, seq_len, embedding_size) or (batch_size, seq_len, embedding_size hidden_size)res[:,:,hidden_size:] is the output of Embedding layerres[:,:,:hidden_size] is the output of previous two blocki (int): layer indexReturns:Tensor: (batch_size, seq_len, hidden_size embedding_size)if i 1:# (batch_size, seq_len, hidden_size embedding_size)return torch.cat([x, res], dim-1)hidden_size x.size(-1)# (res[:, :, :hidden_size] x) is the summation of the output of previous two blocks# x (batch_size, seq_len, hidden_size)x (res[:, :, :hidden_size] x) * math.sqrt(0.5)# (batch_size, seq_len, hidden_size embedding_size)return torch.cat([x, res[:, :, hidden_size:]], dim-1)为了给对齐处理提供更丰富的特征RE2采用了一个增强版的残差连接用于每个块之间。
对于一个长为 l l l的序列标记第 n n n个块的输入和输出分别为 x ( n ) ( x 1 ( n ) , x 2 ( n ) , ⋯ , x l ( n ) ) x^{(n)} (x^{(n)}_1,x^{(n)}_2,\cdots,x^{(n)}_l) x(n)(x1(n),x2(n),⋯,xl(n))和 o ( n ) ( o 1 ( n ) , o 2 ( n ) , ⋯ , o l ( n ) ) o^{(n)} (o^{(n)}_1,o^{(n)}_2,\cdots,o^{(n)}_l) o(n)(o1(n),o2(n),⋯,ol(n)) o ( 0 ) o^{(0)} o(0)表示零向量序列。
第一个块的输入 x ( 1 ) x^{(1)} x(1)是嵌入层的输出由图1中的空心矩形表示第 n ( n ≥ 2 ) n(n\geq 2) n(n≥2)块的输入 x ( n ) x^{(n)} x(n)是第一块的输入 x ( 1 ) x^{(1)} x(1)和前面两块的输出进行求和后的拼接(图中的对角斜线矩形) x i ( n ) [ x i ( 1 ) ; o i ( n − 1 ) o i ( n − 2 ) ] x^{(n)}_i [x^{(1)}_i;o^{(n-1)}_i o^{(n-2)}_i ] xi(n)[xi(1);oi(n−1)oi(n−2)] 公式更加清楚一点第 n n n块的输入是由两个向量拼接而来第一个向量是第一块的输入第二个向量是第 n n n块前面两块的输出进行(元素级)累加。这个就是增强的残差连接。
融合层
class Fusion(nn.Module):def __init__(self, input_size: int, hidden_size: int, dropout: float) - None:Args:input_size (int): embedding_dim hidden_size or embedding_dim hidden_size * 2hidden_size (int): hidden sizedropout (float): dropout ratiosuper().__init__()self.dropout nn.Dropout(dropout)self.fusion1 Linear(input_size * 2, hidden_size, activationsTrue)self.fusion2 Linear(input_size * 2, hidden_size, activationsTrue)self.fusion3 Linear(input_size * 2, hidden_size, activationsTrue)self.fusion Linear(hidden_size * 3, hidden_size, activationsTrue)def forward(self, x: Tensor, align: Tensor) - Tensor:Args:x (Tensor): input (batch_size, seq_len, input_size)align (Tensor): output of Alignment (batch_size, seq_len, input_size)Returns:Tensor: (batch_size, seq_len, hidden_size)# x1 (batch_size, seq_len, hidden_size)x1 self.fusion1(torch.cat([x, align], dim-1))# x2 (batch_size, seq_len, hidden_size)x2 self.fusion1(torch.cat([x, x - align], dim-1))# x3 (batch_size, seq_len, hidden_size)x3 self.fusion1(torch.cat([x, x * align], dim-1))# x (batch_size, seq_len, hidden_size * 3)x torch.cat([x1, x2, x3], dim-1)x self.dropout(x)# (batch_size, seq_len, hidden_size)return self.fusion(x)融合层通过三个方面比较了局部和对齐表示(分别为对齐层的输入和输出)然后对它们进行融合。
对于第一个序列来说融合层的输出 a ˉ \bar a aˉ计算为 a ˉ i 1 G 1 ( [ a i ; a i ′ ] ) , a ˉ i 2 G 2 ( [ a i ; a i − a i ′ ] ) , a ˉ i 3 G 3 ( [ a i ; a i ∘ a i ′ ] ) , a ˉ i G ( [ a ˉ i 1 ; a ˉ i 2 ; a ˉ i 3 ] ) , \begin{aligned} \bar a_i^1 G_1([a_i;a_i^\prime]), \\ \bar a_i^2 G_2([a_i;a_i - a_i^\prime]), \\ \bar a_i^3 G_3([a_i;a_i \circ a_i^\prime]), \\ \bar a_i G([\bar a_i^1;\bar a_i^2;\bar a_i^3]), \end{aligned} aˉi1aˉi2aˉi3aˉiG1([ai;ai′]),G2([ai;ai−ai′]),G3([ai;ai∘ai′]),G([aˉi1;aˉi2;aˉi3]), 这里 G 1 , G 2 , G 3 G_1,G_2,G_3 G1,G2,G3和 G G G都是参数独立的单层前馈网络 ∘ \circ ∘表示元素级乘法。
差操作( − - −)强调了两个向量的区别而乘操作强调了它们的相似。对于另一个序列 b ˉ \bar b bˉ的计算是类似的。
这些操作和ESIM有点类似增加了一个前馈网络。
完了之后通过一个池化层得到定长向量。
池化层
class Pooling(nn.Module):def forward(self, x: Tensor, mask: Tensor) - Tensor:Args:x (Tensor): (batch_size, seq_len, hidden_size)mask (Tensor): (batch_size, seq_len, 1)Returns:Tensor: (batch_size, hidden_size)# max returns a namedtuple (values, indices), we only need valuesreturn x.masked_fill(~mask, -float(inf)).max(dim1)[0]池化层取时间步维度上的最大值。
预测层
class Prediction(nn.Module):def __init__(self, hidden_size: int, num_classes: int, dropout: float) - None:super().__init__()self.dense nn.Sequential(nn.Dropout(dropout),Linear(hidden_size * 4, hidden_size, activationsTrue),nn.Dropout(dropout),Linear(hidden_size, num_classes),)def forward(self, a: Tensor, b: Tensor) - Tensor:Args:a (Tensor): (batch_size, hidden_size)b (Tensor): (batch_size, hidden_size)Returns:Tensor: (batch_size, num_classes)return self.dense(torch.cat([a, b, a - b, a * b], dim-1))
预测层比较简单再次对输入向量进行了一个融合 y ^ H ( [ v 1 ; v 2 ; v 1 − v 2 ; v 1 ∘ v 2 ] ) \hat {\pmb y} H([v_1;v_2;v_1-v_2;v1 \circ v_2]) y^H([v1;v2;v1−v2;v1∘v2])
RE2实现
RE2的实现时上述层的堆叠
class RE2(nn.Module):def __init__(self, args) - None:super().__init__()self.embedding Embedding(args.vocab_size, args.embedding_dim, args.dropout)self.connection AugmentedResidualConnection()self.blocks nn.ModuleList([nn.ModuleDict({encoder: Encoder(args.embedding_dimif i 0else args.embedding_dim args.hidden_size,args.hidden_size,args.kernel_sizes,args.encoder_layers,args.dropout,),alignment: Alignment(args.embedding_dim args.hidden_sizeif i 0else args.embedding_dim args.hidden_size * 2,args.hidden_size,args.dropout,args.project_func,),fusion: Fusion(args.embedding_dim args.hidden_sizeif i 0else args.embedding_dim args.hidden_size * 2,args.hidden_size,args.dropout,),})for i in range(args.num_blocks)])self.pooling Pooling()self.prediction Prediction(args.hidden_size, args.num_classes, args.dropout)def forward(self, a: Tensor, b: Tensor, mask_a: Tensor, mask_b: Tensor) - Tensor:Args:a (Tensor): (batch_size, seq_len)b (Tensor): (batch_size, seq_len)mask_a (Tensor): (batch_size, seq_len, 1)mask_b (Tensor): (batch_size, seq_len, 1)Returns:Tensor: (batch_size, num_classes)# a (batch_size, seq_len, embedding_dim)a self.embedding(a)# b (batch_size, seq_len, embedding_dim)b self.embedding(b)res_a, res_b a, bfor i, block in enumerate(self.blocks):if i 0:# a (batch_size, seq_len, embedding_dim hidden_size)a self.connection(a, res_a, i)# b (batch_size, seq_len, embedding_dim hidden_size)b self.connection(b, res_b, i)# now embeddings saved to res_a[:,:,hidden_size:]res_a, res_b a, b# a_enc (batch_size, seq_len, hidden_size)a_enc block[encoder](a, mask_a)# b_enc (batch_size, seq_len, hidden_size)b_enc block[encoder](b, mask_b)# concating the input and output of encoder# a (batch_size, seq_len, embedding_dim hidden_size or embedding_dim hidden_size * 2)a torch.cat([a, a_enc], dim-1)# b (batch_size, seq_len, embedding_dim hidden_size or embedding_dim hidden_size * 2)b torch.cat([b, b_enc], dim-1)# align_a (batch_size, seq_len, embedding_dim hidden_size or embedding_dim hidden_size * 2)# align_b (batch_size, seq_len, embedding_dim hidden_size or embedding_dim hidden_size * 2)align_a, align_b block[alignment](a, b, mask_a, mask_b)# a (batch_size, seq_len, hidden_size)a block[fusion](a, align_a)# b (batch_size, seq_len, hidden_size)b block[fusion](b, align_b)# a (batch_size, hidden_size)a self.pooling(a, mask_a)# b (batch_size, hidden_size)b self.pooling(b, mask_b)# (batch_size, num_classes)return self.prediction(a, b)
注意不同块之间输入维度的区别。
数据准备
在→文章←中数据准备这部分内容有详细的解释。
from collections import defaultdict
from tqdm import tqdm
import numpy as np
import json
from torch.utils.data import Dataset
import pandas as pd
from typing import TupleUNK_TOKEN UNK
PAD_TOKEN PADclass Vocabulary:Class to process text and extract vocabulary for mappingdef __init__(self, token_to_idx: dict None, tokens: list[str] None) - None:Args:token_to_idx (dict, optional): a pre-existing map of tokens to indices. Defaults to None.tokens (list[str], optional): a list of unique tokens with no duplicates. Defaults to None.assert any([tokens, token_to_idx]), At least one of these parameters should be set as not None.if token_to_idx:self._token_to_idx token_to_idxelse:self._token_to_idx {}if PAD_TOKEN not in tokens:tokens [PAD_TOKEN] tokensfor idx, token in enumerate(tokens):self._token_to_idx[token] idxself._idx_to_token {idx: token for token, idx in self._token_to_idx.items()}self.unk_index self._token_to_idx[UNK_TOKEN]self.pad_index self._token_to_idx[PAD_TOKEN]classmethoddef build(cls,sentences: list[list[str]],min_freq: int 2,reserved_tokens: list[str] None,) - Vocabulary:Construct the Vocabulary from sentencesArgs:sentences (list[list[str]]): a list of tokenized sequencesmin_freq (int, optional): the minimum word frequency to be saved. Defaults to 2.reserved_tokens (list[str], optional): the reserved tokens to add into the Vocabulary. Defaults to None.Returns:Vocabulary: a Vocubulary instanetoken_freqs defaultdict(int)for sentence in tqdm(sentences):for token in sentence:token_freqs[token] 1unique_tokens (reserved_tokens if reserved_tokens else []) [UNK_TOKEN]unique_tokens [tokenfor token, freq in token_freqs.items()if freq min_freq and token ! UNK_TOKEN]return cls(tokensunique_tokens)def __len__(self) - int:return len(self._idx_to_token)def __getitem__(self, tokens: list[str] | str) - list[int] | int:Retrieve the indices associated with the tokens or the index with the single tokenArgs:tokens (list[str] | str): a list of tokens or single tokenReturns:list[int] | int: the indices or the single indexif not isinstance(tokens, (list, tuple)):return self._token_to_idx.get(tokens, self.unk_index)return [self.__getitem__(token) for token in tokens]def lookup_token(self, indices: list[int] | int) - list[str] | str:Retrive the tokens associated with the indices or the token with the single indexArgs:indices (list[int] | int): a list of index or single indexReturns:list[str] | str: the corresponding tokens (or token)if not isinstance(indices, (list, tuple)):return self._idx_to_token[indices]return [self._idx_to_token[index] for index in indices]def to_serializable(self) - dict:Returns a dictionary that can be serializedreturn {token_to_idx: self._token_to_idx}classmethoddef from_serializable(cls, contents: dict) - Vocabulary:Instantiates the Vocabulary from a serialized dictionaryArgs:contents (dict): a dictionary generated by to_serializableReturns:Vocabulary: the Vocabulary instancereturn cls(**contents)def __repr__(self):return fVocabulary(size{len(self)})class TMVectorizer:The Vectorizer which vectorizes the Vocabularydef __init__(self, vocab: Vocabulary, max_len: int) - None:Args:vocab (Vocabulary): maps characters to integersmax_len (int): the max length of the sequence in the datasetself.vocab vocabself.max_len max_lenself.padding_index vocab.pad_indexdef _vectorize(self, indices: list[int], vector_length: int -1) - np.ndarray:Vectorize the provided indicesArgs:indices (list[int]): a list of integers that represent a sequencevector_length (int, optional): an arugment for forcing the length of index vector. Defaults to -1.Returns:np.ndarray: the vectorized index arrayif vector_length 0:vector_length len(indices)vector np.zeros(vector_length, dtypenp.int64)if len(indices) vector_length:vector[:] indices[:vector_length]else:vector[: len(indices)] indicesvector[len(indices) :] self.padding_indexreturn vectordef _get_indices(self, sentence: list[str]) - list[int]:Return the vectorized sentenceArgs:sentence (list[str]): list of tokensReturns:indices (list[int]): list of integers representing the sentencereturn [self.vocab[token] for token in sentence]def vectorize(self, sentence: list[str], use_dataset_max_length: bool True) - np.ndarray:Return the vectorized sequenceArgs:sentence (list[str]): raw sentence from the datasetuse_dataset_max_length (bool): whether to use the global max vector lengthReturns:the vectorized sequence with paddingvector_length -1if use_dataset_max_length:vector_length self.max_lenindices self._get_indices(sentence)vector self._vectorize(indices, vector_lengthvector_length)return vectorclassmethoddef from_serializable(cls, contents: dict) - TMVectorizer:Instantiates the TMVectorizer from a serialized dictionaryArgs:contents (dict): a dictionary generated by to_serializableReturns:TMVectorizer:vocab Vocabulary.from_serializable(contents[vocab])max_len contents[max_len]return cls(vocabvocab, max_lenmax_len)def to_serializable(self) - dict:Returns a dictionary that can be serializedReturns:dict: a dict contains Vocabulary instance and max_len attributereturn {vocab: self.vocab.to_serializable(), max_len: self.max_len}def save_vectorizer(self, filepath: str) - None:Dump this TMVectorizer instance to fileArgs:filepath (str): the path to store the filewith open(filepath, w) as f:json.dump(self.to_serializable(), f)classmethoddef load_vectorizer(cls, filepath: str) - TMVectorizer:Load TMVectorizer from a fileArgs:filepath (str): the path stored the fileReturns:TMVectorizer:with open(filepath) as f:return TMVectorizer.from_serializable(json.load(f))class TMDataset(Dataset):Dataset for text matchingdef __init__(self, text_df: pd.DataFrame, vectorizer: TMVectorizer) - None:Args:text_df (pd.DataFrame): a DataFrame which contains the processed data examplesvectorizer (TMVectorizer): a TMVectorizer instanceself.text_df text_dfself._vectorizer vectorizerdef __getitem__(self, index: int) - Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, int]:row self.text_df.iloc[index]vector1 self._vectorizer.vectorize(row.sentence1)vector2 self._vectorizer.vectorize(row.sentence2)mask1 vector1 ! self._vectorizer.padding_indexmask2 vector2 ! self._vectorizer.padding_indexreturn (vector1, vector2, mask1, mask2, row.label)def get_vectorizer(self) - TMVectorizer:return self._vectorizerdef __len__(self) - int:return len(self.text_df)
和之前的文章差不多唯一的区别增加了填充位置的mask。
模型训练
learning_rate1e-3,
batch_size256,
num_epochs10,
max_len50,
embedding_dim300,
hidden_size150,
encoder_layers2,
num_blocks2,
kernel_sizes[3],
dropout0.2,
min_freq2,
project_funclinear,
grad_clipping2.0,
print_every300,
num_classes2,经过几次实验表现最好的配置如上所示学习率为0.2梯度裁剪为2.0。
如论文所述增加了梯度裁剪学习率指数衰减通过用AdamW替换。
验证和训练函数为
def evaluate(data_iter: DataLoader, model: nn.Module
) - Tuple[float, float, float, float]:y_list, y_pred_list [], []model.eval()for x1, x2, mask1, mask2, y in tqdm(data_iter):x1 x1.to(device).long()x2 x2.to(device).long()mask1 mask1.to(device).bool().unsqueeze(2)mask2 mask2.to(device).bool().unsqueeze(2)y y.float().to(device)output model(x1, x2, mask1, mask2)pred torch.argmax(output, dim1).long()y_pred_list.append(pred)y_list.append(y)y_pred torch.cat(y_pred_list, 0)y torch.cat(y_list, 0)acc, p, r, f1 metrics(y, y_pred)return acc, p, r, f1def train(data_iter: DataLoader,model: nn.Module,criterion: nn.CrossEntropyLoss,optimizer: torch.optim.Optimizer,grad_clipping: float,print_every: int 500,verboseTrue,
) - None:model.train()for step, (x1, x2, mask1, mask2, y) in enumerate(tqdm(data_iter)):x1 x1.to(device).long()x2 x2.to(device).long()mask1 mask1.to(device).bool().unsqueeze(2)mask2 mask2.to(device).bool().unsqueeze(2)y torch.LongTensor(y).to(device)output model(x1, x2, mask1, mask2)loss criterion(output, y)optimizer.zero_grad()loss.backward()torch.nn.utils.clip_grad_norm_(model.parameters(), grad_clipping)optimizer.step()if verbose and (step 1) % print_every 0:pred torch.argmax(output, dim1).long()acc, p, r, f1 metrics(y, pred)print(f TRAIN iter{step1} loss{loss.item():.6f} accuracy{acc:.3f} precision{p:.3f} recal{r:.3f} f1 score{f1:.4f})核心训练代码为 model RE2(args)
optimizer torch.optim.AdamW(model.parameters(), lrargs.learning_rate)
criterion nn.CrossEntropyLoss()print(fModel: {model})for epoch in range(args.num_epochs):train(train_data_loader,model,criterion,optimizer,args.grad_clipping,print_everyargs.print_every,verboseargs.verbose,)print(Begin evalute on dev set.)with torch.no_grad():acc, p, r, f1 evaluate(dev_data_loader, model)print(fEVALUATE [{epoch1}/{args.num_epochs}] accuracy{acc:.3f} precision{p:.3f} recal{r:.3f} f1 score{f1:.4f})model.eval()acc, p, r, f1 evaluate(test_data_loader, model)
print(fTEST accuracy{acc:.3f} precision{p:.3f} recal{r:.3f} f1 score{f1:.4f})model RE2(args)print(fModel: {model})model_saved_path os.path.join(args.save_dir, args.model_state_file)if args.reload_model and os.path.exists(model_saved_path):model.load_state_dict(torch.load(args.model_saved_path))print(Reloaded model)else:print(New model)model model.to(device)model_save_path os.path.join(args.save_dir, f{datetime.now().strftime(%Y%m%d%H%M%S)}-model.pth)train_data_loader DataLoader(train_dataset, batch_sizeargs.batch_size, shuffleTrue)dev_data_loader DataLoader(dev_dataset, batch_sizeargs.batch_size)test_data_loader DataLoader(test_dataset, batch_sizeargs.batch_size)optimizer torch.optim.AdamW(model.parameters(), lrargs.learning_rate)criterion nn.CrossEntropyLoss()for epoch in range(args.num_epochs):train(train_data_loader,model,criterion,optimizer,args.grad_clipping,print_everyargs.print_every,verboseargs.verbose,)print(Begin evalute on dev set.)with torch.no_grad():acc, p, r, f1 evaluate(dev_data_loader, model)print(fEVALUATE [{epoch1}/{args.num_epochs}] accuracy{acc:.3f} precision{p:.3f} recal{r:.3f} f1 score{f1:.4f})model.eval()acc, p, r, f1 evaluate(test_data_loader, model)print(fTEST accuracy{acc:.3f} precision{p:.3f} recal{r:.3f} f1 score{f1:.4f})
Arguments : Namespace(dataset_csvtext_matching/data/lcqmc/{}.txt, vectorizer_filevectorizer.json, model_state_filemodel.pth, pandas_filedataframe.{}.pkl, save_dirD:\\workspace\\nlp-in-action\\text_matching\\re2\\model_storage, reload_modelFalse, cudaTrue, learning_rate0.001, batch_size256, num_epochs10, max_len50, embedding_dim300, hidden_size150, encoder_layers2, num_blocks2, kernel_sizes[3], dropout0.2, min_freq2, project_funclinear, grad_clipping2.0, print_every300, lr_decay_rate0.95, num_classes2, verboseTrue)
Using device: cuda:0.
Loads cached dataframes.
Loads vectorizer file.
Model: RE2((embedding): Embedding((embedding): Embedding(35925, 300, padding_idx0)(dropout): Dropout(p0.2, inplaceFalse))(connection): AugmentedResidualConnection()(blocks): ModuleList((0): ModuleDict((encoder): Encoder((encoders): ModuleList((0): Conv1d((model): ModuleList((0): Sequential((0): Conv1d(300, 150, kernel_size(3,), stride(1,), padding(1,))(1): GeLU())))(1): Conv1d((model): ModuleList((0): Sequential((0): Conv1d(150, 150, kernel_size(3,), stride(1,), padding(1,))(1): GeLU()))))(dropout): Dropout(p0.2, inplaceFalse))(alignment): Alignment((projection): Sequential((0): Dropout(p0.2, inplaceFalse)(1): Linear((model): Sequential((0): Linear(in_features450, out_features150, biasTrue)(1): GeLU()))))(fusion): Fusion((dropout): Dropout(p0.2, inplaceFalse)(fusion1): Linear((model): Sequential((0): Linear(in_features900, out_features150, biasTrue)(1): GeLU()))(fusion2): Linear((model): Sequential((0): Linear(in_features900, out_features150, biasTrue)(1): GeLU()))(fusion3): Linear((model): Sequential((0): Linear(in_features900, out_features150, biasTrue)(1): GeLU()))(fusion): Linear((model): Sequential((0): Linear(in_features450, out_features150, biasTrue)(1): GeLU()))))(1): ModuleDict((encoder): Encoder((encoders): ModuleList((0): Conv1d((model): ModuleList((0): Sequential((0): Conv1d(450, 150, kernel_size(3,), stride(1,), padding(1,))(1): GeLU())))(1): Conv1d((model): ModuleList((0): Sequential((0): Conv1d(150, 150, kernel_size(3,), stride(1,), padding(1,))(1): GeLU()))))(dropout): Dropout(p0.2, inplaceFalse))(alignment): Alignment((projection): Sequential((0): Dropout(p0.2, inplaceFalse)(1): Linear((model): Sequential((0): Linear(in_features600, out_features150, biasTrue)(1): GeLU()))))(fusion): Fusion((dropout): Dropout(p0.2, inplaceFalse)(fusion1): Linear((model): Sequential((0): Linear(in_features1200, out_features150, biasTrue)(1): GeLU()))(fusion2): Linear((model): Sequential((0): Linear(in_features1200, out_features150, biasTrue)(1): GeLU()))(fusion3): Linear((model): Sequential((0): Linear(in_features1200, out_features150, biasTrue)(1): GeLU()))(fusion): Linear((model): Sequential((0): Linear(in_features450, out_features150, biasTrue)(1): GeLU())))))(pooling): Pooling()(prediction): Prediction((dense): Sequential((0): Dropout(p0.2, inplaceFalse)(1): Linear((model): Sequential((0): Linear(in_features600, out_features150, biasTrue)(1): GeLU()))(2): Dropout(p0.2, inplaceFalse)(3): Linear((model): Sequential((0): Linear(in_features150, out_features2, biasTrue)(1): GeLU()))))
)
New model32%|█████████████████████████████████████████████████████▌ | 299/933 [01:1602:38, 4.00it/s]
TRAIN iter300 loss0.273509 accuracy0.887 precision0.885 recal0.926 f1 score0.904964%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 599/933 [02:3101:23, 3.99it/s]
TRAIN iter600 loss0.296151 accuracy0.859 precision0.897 recal0.861 f1 score0.878496%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 899/933 [03:4600:08, 4.00it/s]
TRAIN iter900 loss0.262893 accuracy0.875 precision0.887 recal0.887 f1 score0.8873
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 933/933 [03:5400:00, 3.98it/s]
Begin evalute on dev set.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:0200:00, 14.60it/s]
EVALUATE [1/10] accuracy0.752 precision0.737 recal0.783 f1 score0.759232%|█████████████████████████████████████████████████████▌ | 299/933 [01:1402:37, 4.03it/s]
TRAIN iter300 loss0.272779 accuracy0.898 precision0.919 recal0.907 f1 score0.913364%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 599/933 [02:2901:23, 3.98it/s]
TRAIN iter600 loss0.238999 accuracy0.898 precision0.907 recal0.930 f1 score0.918796%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 899/933 [03:4400:08, 4.00it/s]
TRAIN iter900 loss0.225822 accuracy0.910 precision0.929 recal0.909 f1 score0.9187
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 933/933 [03:5200:00, 4.01it/s]
Begin evalute on dev set.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:0200:00, 14.59it/s]
EVALUATE [2/10] accuracy0.787 precision0.763 recal0.831 f1 score0.795632%|█████████████████████████████████████████████████████▌ | 299/933 [01:1402:37, 4.03it/s]
TRAIN iter300 loss0.260889 accuracy0.902 precision0.929 recal0.912 f1 score0.920664%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 599/933 [02:2901:22, 4.03it/s]
TRAIN iter600 loss0.216830 accuracy0.910 precision0.929 recal0.923 f1 score0.925696%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 899/933 [03:4300:08, 4.06it/s]
TRAIN iter900 loss0.162659 accuracy0.945 precision0.944 recal0.958 f1 score0.9510
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 933/933 [03:5100:00, 4.02it/s]
Begin evalute on dev set.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:0200:00, 14.73it/s]
EVALUATE [3/10] accuracy0.816 precision0.809 recal0.827 f1 score0.817932%|█████████████████████████████████████████████████████▌ | 299/933 [01:1402:36, 4.06it/s]
TRAIN iter300 loss0.228807 accuracy0.906 precision0.909 recal0.922 f1 score0.915564%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 599/933 [02:2801:22, 4.05it/s]
TRAIN iter600 loss0.186292 accuracy0.926 precision0.932 recal0.938 f1 score0.934796%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 899/933 [03:4200:08, 4.06it/s]
TRAIN iter900 loss0.160805 accuracy0.953 precision0.957 recal0.957 f1 score0.9568
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 933/933 [03:5000:00, 4.04it/s]
Begin evalute on dev set.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:0200:00, 14.73it/s]
EVALUATE [4/10] accuracy0.814 precision0.804 recal0.832 f1 score0.817632%|█████████████████████████████████████████████████████▌ | 299/933 [01:1302:36, 4.06it/s]
TRAIN iter300 loss0.190363 accuracy0.910 precision0.926 recal0.919 f1 score0.922664%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 599/933 [02:2801:22, 4.04it/s]
TRAIN iter600 loss0.190028 accuracy0.918 precision0.901 recal0.967 f1 score0.932596%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 899/933 [03:4200:08, 4.05it/s]
TRAIN iter900 loss0.170661 accuracy0.930 precision0.957 recal0.918 f1 score0.9375
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 933/933 [03:5000:00, 4.04it/s]
Begin evalute on dev set.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:0200:00, 14.73it/s]
EVALUATE [5/10] accuracy0.810 precision0.775 recal0.873 f1 score0.821232%|█████████████████████████████████████████████████████▌ | 299/933 [01:1402:40, 3.95it/s]
TRAIN iter300 loss0.125980 accuracy0.965 precision0.974 recal0.968 f1 score0.970964%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 599/933 [02:2801:22, 4.05it/s]
TRAIN iter600 loss0.160912 accuracy0.930 precision0.928 recal0.953 f1 score0.940496%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 899/933 [03:4200:08, 4.05it/s]
TRAIN iter900 loss0.159766 accuracy0.930 precision0.922 recal0.959 f1 score0.9400
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 933/933 [03:5000:00, 4.04it/s]
Begin evalute on dev set.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:0200:00, 14.74it/s]
EVALUATE [6/10] accuracy0.815 precision0.777 recal0.885 f1 score0.827132%|█████████████████████████████████████████████████████▌ | 299/933 [01:1302:36, 4.04it/s]
TRAIN iter300 loss0.144144 accuracy0.941 precision0.973 recal0.929 f1 score0.950864%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 599/933 [02:2801:22, 4.06it/s]
TRAIN iter600 loss0.149635 accuracy0.934 precision0.922 recal0.975 f1 score0.947796%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 899/933 [03:4200:08, 4.06it/s]
TRAIN iter900 loss0.151699 accuracy0.938 precision0.926 recal0.974 f1 score0.9497
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 933/933 [03:5000:00, 4.04it/s]
Begin evalute on dev set.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:0200:00, 14.73it/s]
EVALUATE [7/10] accuracy0.831 precision0.806 recal0.874 f1 score0.838332%|█████████████████████████████████████████████████████▌ | 299/933 [01:1402:36, 4.04it/s]
TRAIN iter300 loss0.191586 accuracy0.922 precision0.908 recal0.967 f1 score0.936764%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 599/933 [02:2901:23, 3.98it/s]
TRAIN iter600 loss0.188188 accuracy0.930 precision0.947 recal0.935 f1 score0.941296%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 899/933 [03:4400:08, 4.03it/s]
TRAIN iter900 loss0.196099 accuracy0.910 precision0.939 recal0.892 f1 score0.9151
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 933/933 [03:5300:00, 4.00it/s]
Begin evalute on dev set.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:0200:00, 14.66it/s]
EVALUATE [8/10] accuracy0.838 precision0.817 recal0.870 f1 score0.842632%|█████████████████████████████████████████████████████▌ | 299/933 [01:1502:36, 4.04it/s]
TRAIN iter300 loss0.136444 accuracy0.953 precision0.986 recal0.934 f1 score0.959264%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 599/933 [02:3001:22, 4.05it/s]
TRAIN iter600 loss0.137828 accuracy0.949 precision0.953 recal0.959 f1 score0.955996%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 899/933 [03:4500:08, 3.98it/s]
TRAIN iter900 loss0.148434 accuracy0.934 precision0.947 recal0.941 f1 score0.9439
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 933/933 [03:5300:00, 3.99it/s]
Begin evalute on dev set.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:0200:00, 14.39it/s]
EVALUATE [9/10] accuracy0.840 precision0.814 recal0.883 f1 score0.847132%|█████████████████████████████████████████████████████▌ | 299/933 [01:1502:38, 4.01it/s]
TRAIN iter300 loss0.223042 accuracy0.918 precision0.904 recal0.968 f1 score0.935064%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 599/933 [02:2901:23, 4.02it/s]
TRAIN iter600 loss0.105175 accuracy0.965 precision0.971 recal0.964 f1 score0.967796%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 899/933 [03:4500:08, 4.04it/s]
TRAIN iter900 loss0.110603 accuracy0.953 precision0.934 recal0.986 f1 score0.9592
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 933/933 [03:5300:00, 4.00it/s]
Begin evalute on dev set.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:0200:00, 14.66it/s]
EVALUATE [10/10] accuracy0.836 precision0.819 recal0.863 f1 score0.8406
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 49/49 [00:0300:00, 14.59it/s]
TEST accuracy0.822 precision0.762 recal0.936 f1 score0.8403这是在没有使用预训练的词向量前提下达到的准确率后面机会自己训练一个word2vec词向量然后结合起来用看下效果。
完整代码
https://github.com/nlp-greyfoss/nlp-in-action-public/blob/master/text_matching/re2/model.py 文章转载自： http://www.morning.jqwpw.cn.gov.cn.jqwpw.cn http://www.morning.jrwbl.cn.gov.cn.jrwbl.cn http://www.morning.dmfdl.cn.gov.cn.dmfdl.cn http://www.morning.sgfgz.cn.gov.cn.sgfgz.cn http://www.morning.kcxtz.cn.gov.cn.kcxtz.cn http://www.morning.nkllb.cn.gov.cn.nkllb.cn http://www.morning.ylyzk.cn.gov.cn.ylyzk.cn http://www.morning.dmwjl.cn.gov.cn.dmwjl.cn http://www.morning.llcsd.cn.gov.cn.llcsd.cn http://www.morning.kxrhj.cn.gov.cn.kxrhj.cn http://www.morning.itvsee.com.gov.cn.itvsee.com http://www.morning.lmzpk.cn.gov.cn.lmzpk.cn http://www.morning.fldsb.cn.gov.cn.fldsb.cn http://www.morning.xznrk.cn.gov.cn.xznrk.cn http://www.morning.rdmz.cn.gov.cn.rdmz.cn http://www.morning.mlpmf.cn.gov.cn.mlpmf.cn http://www.morning.knlbg.cn.gov.cn.knlbg.cn http://www.morning.lptjt.cn.gov.cn.lptjt.cn http://www.morning.gbybx.cn.gov.cn.gbybx.cn http://www.morning.ggnjq.cn.gov.cn.ggnjq.cn http://www.morning.fmtfj.cn.gov.cn.fmtfj.cn http://www.morning.mhnb.cn.gov.cn.mhnb.cn http://www.morning.pmjw.cn.gov.cn.pmjw.cn http://www.morning.hkcjx.cn.gov.cn.hkcjx.cn http://www.morning.hwlmy.cn.gov.cn.hwlmy.cn http://www.morning.rtlth.cn.gov.cn.rtlth.cn http://www.morning.fyglr.cn.gov.cn.fyglr.cn http://www.morning.bpmnz.cn.gov.cn.bpmnz.cn http://www.morning.hqgxz.cn.gov.cn.hqgxz.cn http://www.morning.hmfxl.cn.gov.cn.hmfxl.cn http://www.morning.wgbmj.cn.gov.cn.wgbmj.cn http://www.morning.qfkdt.cn.gov.cn.qfkdt.cn http://www.morning.ampingdu.com.gov.cn.ampingdu.com http://www.morning.dwzwm.cn.gov.cn.dwzwm.cn http://www.morning.rbhqz.cn.gov.cn.rbhqz.cn http://www.morning.lnckq.cn.gov.cn.lnckq.cn http://www.morning.jjwzk.cn.gov.cn.jjwzk.cn http://www.morning.zpstm.cn.gov.cn.zpstm.cn http://www.morning.lcplz.cn.gov.cn.lcplz.cn http://www.morning.byshd.cn.gov.cn.byshd.cn http://www.morning.gnzsd.cn.gov.cn.gnzsd.cn http://www.morning.yktwr.cn.gov.cn.yktwr.cn http://www.morning.nbfkk.cn.gov.cn.nbfkk.cn http://www.morning.yqtry.cn.gov.cn.yqtry.cn http://www.morning.gxtbn.cn.gov.cn.gxtbn.cn http://www.morning.clkyw.cn.gov.cn.clkyw.cn http://www.morning.qrqg.cn.gov.cn.qrqg.cn http://www.morning.coatingonline.com.cn.gov.cn.coatingonline.com.cn http://www.morning.ltxgk.cn.gov.cn.ltxgk.cn http://www.morning.qrmry.cn.gov.cn.qrmry.cn http://www.morning.lflsq.cn.gov.cn.lflsq.cn http://www.morning.lwyqd.cn.gov.cn.lwyqd.cn http://www.morning.jgzmr.cn.gov.cn.jgzmr.cn http://www.morning.rnmc.cn.gov.cn.rnmc.cn http://www.morning.kgltb.cn.gov.cn.kgltb.cn http://www.morning.kdnbf.cn.gov.cn.kdnbf.cn http://www.morning.cbczs.cn.gov.cn.cbczs.cn http://www.morning.dygsz.cn.gov.cn.dygsz.cn http://www.morning.sdecsd.cn.gov.cn.sdecsd.cn http://www.morning.mlhfr.cn.gov.cn.mlhfr.cn http://www.morning.nrbcx.cn.gov.cn.nrbcx.cn http://www.morning.ldgqh.cn.gov.cn.ldgqh.cn http://www.morning.bxqtq.cn.gov.cn.bxqtq.cn http://www.morning.gmwdl.cn.gov.cn.gmwdl.cn http://www.morning.wfttq.cn.gov.cn.wfttq.cn http://www.morning.srzhm.cn.gov.cn.srzhm.cn http://www.morning.ghfmd.cn.gov.cn.ghfmd.cn http://www.morning.sphft.cn.gov.cn.sphft.cn http://www.morning.pbtrx.cn.gov.cn.pbtrx.cn http://www.morning.mtmnk.cn.gov.cn.mtmnk.cn http://www.morning.bkppb.cn.gov.cn.bkppb.cn http://www.morning.zfqdt.cn.gov.cn.zfqdt.cn http://www.morning.kdpal.cn.gov.cn.kdpal.cn http://www.morning.bwttp.cn.gov.cn.bwttp.cn http://www.morning.ktnt.cn.gov.cn.ktnt.cn http://www.morning.xkyfq.cn.gov.cn.xkyfq.cn http://www.morning.msbpb.cn.gov.cn.msbpb.cn http://www.morning.qptbn.cn.gov.cn.qptbn.cn http://www.morning.duqianw.com.gov.cn.duqianw.com http://www.morning.znkls.cn.gov.cn.znkls.cn
