Improving named entity recognition in Vietnamese texts by a character-level deep lifelong learning model
Vietnam Journal of Computer Science
Vol. 6, No. 4 (2019) 471–487
c
# The Author(s)
.
Improving Named Entity Recognition in Vietnamese Texts
by a Character-Level Deep Lifelong Learning Model¤
,
,¶
,
||
Ngoc-Vu Nguyen† §, Thi-Lan Nguyen‡ , Cam-Van Nguyen Thi‡
,
,
,
,
Mai-Vu Tran‡ **, Tri-Thanh Nguyen‡ †† and Quang-Thuy Ha‡ ‡‡
†
Department of Information Technology
Ministry of Natural Resources and Environment (MONRE)
10, Ton That Thuyet, Cau Giay, Hanoi, Vietnam
‡
Vietnam National University
Hanoi (VNU), University of Engineering and Technology (UET)
144, Xuan Thuy, Cau Giay, Hanoi, Vietnam
§
¶
||
**
††
‡‡
Received 5 June 2019
Accepted 13 September 2019
Published 18 October 2019
Named entity recognition (NER) is a fundamental task which a®ects the performance of its
dependent task, e.g. machine translation. Lifelong machine learning (LML) is a continuous
learning process, in which the knowledge base accumulated from previous tasks will be used to
improve future learning tasks having few samples. Since there are a few studies on LML based
on deep neural networks for NER, especially in Vietnamese, we propose a lifelong learning model
based on deep learning with a CRFs layer, named DeepLML–NER, for NER in Vietnamese
texts. DeepLML–NER includes an algorithm to extract the knowledge of \pre¯x-features" of
named entities in previous domains. Then the model uses the knowledge in the knowledge base
to solve the current NER task. Preprocessing and model parameter tuning are also investigated
to improve the performance. The e®ect of the model was demonstrated by in-domain and cross-
domain experiments, achieving promising results.
Keywords: Named entity recognition; deep lifelong learning; DeepLML for NER in Vietnamese
texts.
*
This paper is an extended version of the paper \Ngoc-Vu Nguyen, Thi-Lan Nguyen, Cam-Van Nguyen
Thi, Mai-Vu Tran, Quang-Thuy Ha. A Character-Level Deep Lifelong Learning Model for Named Entity
Recognition in Vietnamese Text. ACIIDS (1) 2019: 90–102."
††
Corresponding author.
This is an Open Access article published by World Scienti¯c Publishing Company. It is distributed under
the terms of the Creative Commons Attribution 4.0 (CC BY) License which permits use, distribution and
reproduction in any medium, provided the original work is properly cited.
471
472 N.-V. Nguyen et al.
1. Introduction
Named entity recognition (NER), a sequence labeling problem, in textual documents
has become a highly challenging topic drawing great attention thanks to its appli-
cation in bigger tasks, e.g. machine translation. In recent years, some neural network
–
models have been proposed for sequence labeling and NER,1 4 and these models are
competitive with traditional models.
–
Lifelong machine learning (LML) is an advanced learning machine paradigm.5 8
In LML, the learning process is continuous, i.e. the accumulated knowledge learned
from previous tasks in the knowledge base is exploited to enhance future learning
tasks. So far, there have been studies that combine LML with deep learning in
di®erent tasks such as human action detection,9 image recognition.10 However, LML
models with deep learning methods for sequence labeling are limited, especially for
Vietnamese NER.
This paper presents a Vietnamese NER Deep LML model based on a Bidirectional
Long Short-term Memory (Bi-LSTM) network combined with CRFs (denoted as
DeepLML–NER model). This paper has four main contributions:
(1) Propose an algorithm for extracting the knowledge of \pre¯x-features" of NEs in
Vietnamese texts.
(2) Propose a Deep Lifelong Learning Model for NER in Vietnamese texts.
DeepLML–NER can perform well on the current domain by storing and trans-
ferring \pre¯x-features" knowledge simultaneously from the previous tasks to
the current one. Experiments were designed on both in-domain and cross-domain
data to evaluate the e®ectiveness of the model.
(3) Present a method to enhance the data quality in the previous tasks to improve
the knowledge quality.
(4) Investigate to select suitable model parameters for achieving better performance.
The rest of this paper is organized as follows. In Sec. 2, we brie°y outline funda-
mental concepts of NER, Bi-LSTM and the hybrid architecture which combines Bi-
LSTM with a CRFs layer for NER task. Section 3 describes the proposed Deep
Lifelong Learning Model for Vietnamese NER, which includes an algorithm for
extracting the knowledge of \pre¯x-features". Section 4 shows the experiments on 16
Vietnamese domains. The comparisons of performance of our Vietnamese NER
model with others have been shown. Some related work is introduced in Sec. 5. In
Sec. 6, we present conclusions and future work.
2. Background
In this section, we brie°y outline the fundamental concepts of the NER problem,
Bi-LSTM and the hybrid architecture, which combines Bi-LSTM with a CRFs layer
for NER task.
Improving Named Entity Recognition in Vietnamese Texts 473
2.1. Named entity recognition
Named entity is an entity that was assigned a certain name, e.g. John Smith is a
name assigned to a person, Microsoft Corporation is a name assigned to an orga-
nization, where person and organization are considered as entity types. Given an
input sentence, NER's target is to identify the name as well as entity type. For
example, given the input: \Microsoft Corporation was founded by Bill Gates and
Paul Allen on April 4, 1975, to develop and sell BASIC interpreters for the Altair
8800a". The goal of NER is to identify the set of named entities along with their
type: f[\Microsoft Corporation": ORG], [\Bill Gates": PER], [\Paul Allen": PER],
[\April 4, 1975": TIME], [\BASIC": MICS], [\Altair 8800": MICS]g, where PER,
ORG, and MICS are the acronyms of Person, Organization, and Miscellaneous,
correspondingly.
This can be considered as a sequence problem, i.e. given a sentence (a sequence of
tokens), the task is to assign a suitable label for each token. For example, the above
sentence should be labeled as
\MicrosoftB-ORG CorporationI-ORG wasO foundedO byO BillB-PER GatesI-PER
andO PaulB-PER AllenI-PERonO AprilB-TIME 4I-TIME, 1975I-TIME, toO developO andO
sellOBASICB-MICS interpretersO forO theO AltairB-MICS 8800I-MICS".
In case a named entity consists of multiple tokens, the ¯rst token is labeled as
B-Z, where Z is the type of the entity, and B means the beginning token of the
entity. The following tokens of the entity are labeled as I-Z. All tokens not included
in a named entity are labeled as O (other). Another example of NER in Vietnamese
texts is given in Fig. 1.
NER is a subtask of information extraction, and its alternative names are entity
identi¯cation, entity chunking, and named entity extraction. Common models for
NER are Conditional Random Fields (CRF),11 Hidden Markov Model (HMM), and
Maximum Entropy Markov Model (MEMM). Recently, deep learning methods with
Fig. 1. Bi-LSTM+CRF architecture for a sample Vietnamese sentence.
a
474 N.-V. Nguyen et al.
Neural Networks have also been incorporated in sequence labeling models as pre-
sented in Sec. 2.2.
2.2. Bidirectional LSTM+CRF
LSTM (Bi-LSTM) is a well known variant of the Recurrent Neural Networks (RNN),
which was originally introduced as a solution to overcome the problem of vanishing
and exploding gradient and hence allowing deeper networks to perform well in
practice.12 This idea was implemented in LSTM cells by creating an internal memory
state which is simply added to the processed input in order to reduce the multipli-
cative e®ect of small gradients.
For NER, traditional LSTM models have a big disadvantage since they do not use
the current su x of a sentence. Thus, Bi-LSTM13 was designed to address this issue.
A Bi-LSTM contains two single LSTM networks, including forward and backward
ones. For each input word, the forward LSTM learns the e®ect of the pre¯x of a word
and the backward LSTM learns the e®ect of the su x of the word.
In a Bi-LSTM model, the ¯nal label of the output is not decided by the softmax
function. It means that the ¯nal tagging of a token does not depend on the label of
its surrounding words. Thus, with the advantage of allowing the context to be
labeled in the CRFs,11 adding a CRF layer to an LSTM or Bi-LSTM model
will allow the model to learn the sequence label in the best way (hereafter Bi-
LSTMþCRF), thereby maximizing the accuracy of the model. Figure 1 illustrates
the architecture of a Bi-LSTMþCRF, it is used to label the words of a Vietnamese
input sentence \UBND
has decided).
" (Bình
province committee
3. The Proposed Character-Level Deep Lifelong Learning Model
Pre¯x patterns are words or phrases that usually precede named entities. For
example, the word Công ty (company) usually precedes the name of an organization
(ORG label); Ông (Mr.), Bà (Mrs.) usually stand in front of a person entity (PER
label), etc. Using a pre¯x list of labels, we can extract more features for each word by
calculating the correlation between the word and the list of pre¯xes. Besides, we
apply a scalable pre¯x list ꢀꢀꢀ an important feature that the model can apply for
lifelong learning.
Let K be a reliable pre¯x set extracted from previous tasks using a model M,
where a Bi-LSTMþCRF is used. The model M is trained based on a training dataset
Dt. Initially, let K be the set Kt, i.e. the set of all reliable pre¯xes in the training data
set Dt. M will deal with more tasks, and more reliable pre¯xes will be extracted, so
that the size of K is larger. When dealing with the Dnþ1 task, extra extracted pre¯xes
will be added to K, consequently, the model M can yield better results for the
new task.
Improving Named Entity Recognition in Vietnamese Texts 475
Fig. 2. DeepLML–NER: The character-level deep lifelong learning model for NER.
3.1. Model training
The proposed character-level deep lifelong learning model named DeepLML–NER,
consisting of \Lifelong Learning" and \Character-level Deep learning" models,
includes three phases, as depicted in Fig. 2: (1) Preprocessing, (2) Feature extraction,
and (3) Model training using Bi-LSTMþCRF.
3.1.1. Data preprocessing
For labeled data, we run a word segmentation tool on each sentence. Then run a POS
tagger for each word using VNCoreNLP,b and convert the resulted data into IOB2
format.14 Since the aim of this model is to detect only person, location, organization,
and miscellaneous named entity types, there are a total of seven possible labels for a
token: B-PER, I-PER, B-LOC, I-LOC, B-ORG, I-ORG, O.
3.1.2. Feature extraction
Assume that a sentence includes n words, i.e. S ¼ ½w1; w2; . . . ; wnꢁ. In this phase, the
features of each word wi are extracted and represented as a vector xi.
b
https://github.com/vncorenlp/VnCoreNLP.
476 N.-V. Nguyen et al.
Word embedding-based and POS tag feature: We apply a pretrained word vector
model for non-English languages named pretrained Word2vec.c This model was
trained on a Wikipedia repository of 74 million Vietnamese sentences, of which the
word embedding matrix is 10,087 words, each word is represented by a 100-dimen-
sional vector. We also randomly initialize a 100-dimensional vector in order to rep-
resent the words contained in the dataset but not in the embedded matrix. Each
word wi in the input sentence is searched in the embedded matrix to obtain a feature
vector wwi .
POS tag feature is transformed into one-hot vectors. A set with m POS tags were
encoded through an m ꢂ m identity matrix (i.e. the matrix in which only elements in
the diagonal are 1, the other elements are 0). POS tag feature vector wpi is the ith row
of the matrix, where i 2 ½0; m ꢀ 1ꢁ is the POS tag's index in the POS tag list.
Based on an assumption that the meaning of a word is synthesized from the
information of its characters, and the characters' information in the same word is
related to each other, we also combine character embedding and CNN15 to extract
character-level features. Speci¯cally, the character embedded matrix is initialized
randomly for the lifelong learning model, and each letter of a word is encoded based
on this matrix, and then passed to the CNN network to obtain the feature vector wCi .
Pre¯x feature: A set of pre¯xes extracted in advance, i.e. by retrieving all pre¯xes
of words except the O label in the training dataset, is divided into subsets based on
the entity's label (i.e. the label of the entity after the pre¯x), and stored in the KB.
After that, we use word embedding to encode these words, and then calculate the
average vectors on each set of pre¯xes (called pre¯x cluster center).
The pre¯x cluster center vector is appended to the word embedded vectors of the
pre¯x of each word (see Fig. 3). The vector is then passed through a fully connected
layer with the activation function tanh to obtain the pre¯x feature vector wfi with
size of df.
Feature vector representation: Finally, all the extracted features are put into a
single vector for each word ith,
xi ¼ xiw ꢃ xip ꢃ xic ꢃ xif:
ð1Þ
3.1.3. Model training using Bi-LSTMþCRF
We extract and represent the training data in the form of vectors as previously
described. The training dataset is divided into chunks to be processed in batches. The
model calculates the loss and other measures (which will be presented in the model
evaluation phase), at the end of each batch, the weights will be updated once. To
update the weights of the model, we use the Adaptive Moment (ADAM) estimation
optimization function.
Training the Bi-LSTMþCRF neural network with the labeled datasets will result
in a model which will be used for extraction and inference in the next phase.
c
Improving Named Entity Recognition in Vietnamese Texts 477
Fig. 3. Pre¯x feature representation.
3.2. Unlabeled data enhancement
After an investigation on the unlabeled data retrieved from a Vietnamese news
website as well as sentence segmentation tool, we found some characteristics listed
below, and proposed solution for alleviation:
(1) There are short sentences, e.g. the title of a news article, the caption of a picture.
These are usually not expressed in a complete grammar sentences, thus they can
be seen as noise, which causes bad e®ects on the recognition process. To reduce
the bad e®ects, we proposed to remove them from the unlabeled dataset. Thanks
to the fact that the NER works on each sentence independently, this removal
does not produce any context loss. From the experiments, we found that a
sentence with less than six words should be removed.
(2) There are long sentences resulting from the mistake of word segmentation tool
which fails to detect the boundary of two or more consecutive sentences. Thus,
two or more sentences are combined as one. This also creates noise in the NER
module, and should be removed. A sentence is considered as an error if it has
more than 60 words.
^
(3) A syllable in Vietnamese has at most 7 characters, such as \nghieng" (italic). Due
to the error in the word segmentation tool, a very long word can be produced,
and cause noise. We will skip sentences having tokens consisting of more than 10
characters.
478 N.-V. Nguyen et al.
Table 1. Adjusted values of key parameters.
Default value Adjusted value
Parameter
The number of units in the two-levels of LSTM (word lstm units)
The dimension of the pre¯x feature (pre word feature size)
The batch size (batch size)
100
100
40
200
200
64
0.001
The learning rate (learning rate)
0.01
(4) Vietnamese compound words usually consist of two single words. Unexpected
errors of the word segmentation tool can result in a token containing several
ꢀ
^
^
words, e.g. the phrase \Theo bao Thanh nien" (according to Thanh nien
ꢀ
^
newspaper) is incorrectly segmented into one token Theo bao thanh nien con-
sisting of four words. The sentences having this error should be ignored.
3.3. Model parameter optimization
Following the Bi-LSTM-CRF model of Pham,15 in the published source code,d we get
the prede¯ned values of the key parameters as the default ones. We also adjusted
these parameters through experiments to get the most suitable values. Table 1 lists
the adjusted values of the parameters.
3.4. Lifelong pre¯x feature extraction
In this phase, the ¯rst step is to collect data from Dantri,e a Vietnamese online
newspaper. The data is preprocessed by VnCoreNLPf tool. The model M trained in
the previous phase is used to recognize the entities and extract pre¯xes from this
dataset (hereafter new task). M is unchanged and the data from new tasks are
unlabeled. All the results from the new domain are stored in the past information
store S. After M has extracted the pre¯xs from n domains, and when working with
the ðn þ 1Þth task, model M uses the reliable pre¯xes K
from S and
ðnþ1Þ
KtðK ¼ Kt [ Kðnþ1ÞÞ. The pre¯xes in Kt from the dataset are trusted because they
are manually labeled. It is not possible to use all of the pre¯xes obtained from
previous tasks as reliable data, because the labeling model may have errors. However,
if a pre¯x appears multiple times, the pre¯x is likely to be reliable. Thus, it is possible
to de¯ne a threshold ꢀ to select reliable pre¯xes if their frequency is above this
threshold.
In DeepLML–NER, the lifelong feature extraction algorithm for aspect mining
(L-CRF) proposed by Shu et al.16 is applied to extract lifelong feature, but instead
of using the general dependency feature, DeepLML–NER uses the pre¯x feature.
Algorithm 1 describes the lifelong feature extraction of our proposed model.
d
https://github.com/pth1993/NNVLP.
e
f
Improving Named Entity Recognition in Vietnamese Texts 479
The detail steps of the algorithm are explained as follows:
(1) Generate features F on the data Dnþ1 (line 3).
(2) Use the model M to produce a set of entities E
(Line 4).
ðnþ1Þ
(3) E
is added to S which is the past knowledge base. From S, we mine a set of
ðnþ1Þ
frequent pre¯xes K
using a threshold ꢀ (line 5).
is the same as Kp from the previous iteration that means no more
ðnþ1Þ
(4) If K
ðnþ1Þ
pre¯xes are found, then the loop will stop (line 7).
(5) Otherwise, some additional reliable pre¯xes are found. M can label data more
accurately, in the next iteration. Lines 10 and 11 update the two sets for the next
iteration (line 10–12).
3.5. Model evaluation
Testing dataset is also preprocessed by VnCoreNLP tool to extract the same features
as the above phase. The additional KB is used to extract the pre¯x feature. Then, the
model ¯lters the pre¯xes with the frequency above a threshold ꢀ, and evaluates the
performance with standard measures, as described in Secs. 3.5.1 and 3.5.2.
3.5.1. Evaluation method
We use the same training datasets and test datasets for all experiments. Domain
datasets (i.e. the unlabeled dataset) are combined for training and testing in two
ways: in-domain and cross-domain. Assume that the lifelong feature extraction phase
was performed on unlabeled domain datasets with thresholds ꢀ.
Cross-domain experiments: For each iteration, one domain is selected as the test
set, the remaining nine domains are used as the training dataset. This evaluation
480 N.-V. Nguyen et al.
helps to answer whether the trained model can be e®ectively used in di®erent
domains, thus saving the e®ort of manually making labeled data.
In-domain experiments: Training and test datasets of the same domains are used
for experiments. In addition, in order to evaluate the improvement of the model when
solving the new task after taking advantage of the knowledge learned from the
previous tasks, we compare the Deep LML model with two other models, which do
not use the knowledge from the previous tasks:
(1) CRFs: this model only uses feature F ¼ ðwi; wpÞ.
i
(2) Bi-LSTMþCRF: this model uses feature F ¼ ðwi; wp; wic; wfÞ which is similar to
i
i
Deep LML, however, it does not use prior knowledge.
For CRFs, we use CRFSuitg for implementing this model fast.
3.5.2. Evaluation measures
The model is evaluated with Precision (PÞ, Recall (RÞ and F1 measures,
F1 ¼ ð2 ꢂ P ꢂ RÞ ꢂ 100%=ðP þ RÞ;
where the precision is the percentage of named entities that are correctly found; the
recall is the percentage of correctly found named entities over the named entities that
existed in the testing dataset. The average of the two measures is F1 score, which
re°ects the overall quality of the model.
4. Experimental Results
4.1. Datasets
4.1.1. VLSP 2018 NER dataset
Experiments were conducted on the VLSP 2018 dataset for NER. This dataset is
provided by the Vietnamese Language and Speech Community, which are collected
from a Vietnamese electronic newspaper website, including 10 domains.
Four types of NEs are the same as those in the Computational Natural Language
Learning (CoNLL) Shared Task 2003,h i.e. PER (person), ORG (organization), LOC
(locations) and MISC (miscellaneous). The main aim of the VLSP 2018, Evaluation
Table 2. The statistics of VLSP 2018 NER dataset.
Dataset
PER
ORG
LOC
MICS
Total
Training
Development
Test
4,600 5,587 6,289
492 723 795
743
63
17,219
2,073
1,883 2,126 2,377
6,978 8,436 9,461
178
984
6,564
Total
25,856
g
h
Improving Named Entity Recognition in Vietnamese Texts 481
Table 3. Percentage of overlapping entities between the domains in the VLSP2018 dataset.
Science/
Domain
Life Entertainment Education Tech Economy Law World Sports Culture Society
Life
ꢀꢀꢀ
0.14
ꢀꢀꢀ
0.13
0.05
ꢀꢀꢀ
0.1
0.14
0.07
0.12
0.13
ꢀꢀꢀ
0.14 0.09 0.11
0.04 0.04 0.06
0.11 0.05 0.06
0.06 0.12 0.08
0.08 0.06 0.05
0.15
0.08
0.12
0.12
0.09
0.08
0.09
0.05
ꢀꢀꢀ
0.13
0.05
0.12
0.1
Entertainment
Education
Science/Tech
Economy
Law
0.05
0.06
0.06
0.04
0.05
0.03
0.03
0.03
0.04
0.05
0.06
ꢀꢀꢀ
0.07
0.08
0.06
0.04
0.05
0.05
0.05
0.05
0.07
0.08
0.08
0.04
0.04
0.06
0.08
0.07
0.04
0.08
0.04
0.05
0.05
0.11
0.1
ꢀꢀꢀ
0.09
0.08
0.05
0.07
0.11
0.04 0.03
World
ꢀꢀꢀ
0.04
0.05
ꢀꢀꢀ
0.07
0.04
0.08
ꢀꢀꢀ
Sports
0.03 0.04
Culture
Society
0.06 0.05 0.04
0.08 0.05 0.04
0.1
Campaign for NER, is to evaluate the ability of recognizing those entities. Input data
followed MUC-6 NER taski format: only tags of named entities chunks are provided,
without word segmentation and other information (POS tag, chunk tag, etc.). This
dataset is divided into training, development and test sets. The statistic information
of the dataset is described in Table 2, and the overlapping information among the
domains is listed in Table 3.
4.1.2. Dantri dataset
The second dataset is an unassigned dataset collected from 1600 articles in 16
domains from a Vietnamese news paper.j It contains 246,586 sentences including
6,682,201 tokens. The statistics of each domain is described in Table 4.
Table 4. The statistic of Dantri dataset.
Domain
#tokens
#sentences
Strange story
Entertainment
Education
Business
Young lifestyle
Auto and moto
Law
350,450
271,666
680,809
483,219
309,252
480,321
462,295
475,327
427,959
404,959
180,972
401,711
402,051
514,916
433,822
402,472
6,682,201
15,169
11,086
25,331
15,795
10,910
14,680
16,003
17,885
14,374
14,480
6,746
Health
Strength
Event
Charity
World
14,664
17,215
23,872
15,947
13,463
246,586
Sports
Love
Culture
Society
Summary
i
j
482 N.-V. Nguyen et al.
Please be reminded that the classi¯cation of webpages on two websites for the
same domain is not the same. Thus, the data in this dataset does not have the same
distribution as that of VLSP 2018 on the same domain.
4.2. Experimental settings
Initial settings: We set the dimension of word embedding and pre¯x feature size as
100. Since the dataset is relatively large, we select the batch size of 40 and the unit
number in the LSTM layer is 100. The number of CNN ¯lters is 30, and CNN window
size is 3. We set the parameters of the library by default for the parameters of
the optimization function Adam, i.e. learningrate ¼ 0:01; ꢁ1 ¼ 0:9; ꢁ2 ¼ 0:999;
ꢂ ¼ 10ꢀ7. The threshold ꢀ for ¯ltering reliable pre¯xes is set to 2.
Cross-domain experiments: Each domain dataset is chosen as the testing dataset
and the combination of 9 other domain datasets is chosen as the training dataset and
we obtained 10 results. This evaluation method desires to have a training model for
e®ective use in di®erent domains, thus saving the e®ort of manually assigning labels.
In-domain experiments: The training and testing datasets are from di®erent
domains. When a domain is selected for testing, the other domains are used as the
training datasets. Consequently, we have ten experiments in corresponding 10
domains. In addition, in order to evaluate the improvement over the model when
solving the new task after taking advantage of the knowledge learned from the old
tasks, we compare the Deep LML model with two models which do not use the
knowledge of the old tasks.
For evaluation, we used BiLSTMþCRF and CRFs as the baselines.
4.3. Experimental results and analyses
To evaluate the e®ect of our di®erent proposals, we carried out multiple experi-
mental scenarios.
4.3.1. Deep LML only experiments
This scenario investigates the advantages of Deep LML over the baselines. Table 5
shows the detailed experimental results of the proposed model. We found that for
both experimental strategies, the results of the BiLSTM+CRF model are better than
CRFs, and Deep LML model results are higher than the other two models.
Cross-domain result: In Table 5, the name of the domain X is the test set, the
other domains are used as the training set. Deep LML (1 run) means we ran this
algorithm once and recorded the results. Looking at the result table, we can see that
the Deep LML is better than the other two models with an average F1 scoring of
67,84%, while the conventional CRFs model is only 62.53% and that score in the
model Bi-LSTMþCRF that did not use prior knowledge just reached 66.63%.
Since the Deep LML has random characteristics, each run will produce di®erent
results. We ran it 10 times and selected the best, the results are shown in Table 6.
Improving Named Entity Recognition in Vietnamese Texts 483
Table 5. Cross-domain results (1 run).
CRF
Bi-LSTM+CRF
Deep LML (1 run)
Domain
P (%) R (%)
F1 (%) P (%) R(%) F1 (%)
P (%)
R (%)
F1 (%)
Life
75.71
64.00
70.83
60.18
74.38
83.47
50.00
62.62
62.53
82.38
68.61
67.05
53.96
63.42
62.80
64.53
75.78
56.55
37.47
49.55
68.57
59.97
70.36
55.73
66.27
57.89
67.12
78.92
59.08
42.54
53.17
74.23
62.53
65.82
63.86
78.36
66.47
69.63
83.73
67.02
39.39
61.18
75.47
67.09
67.89
70.00
76.14
53.74
69.38
79.05
62.20
49.79
62.75
73.97
66.49
66.84
66.79
77.23
59.43
69.50
81.32
64.52
43.99
61.95
74.71
66.63
52.33
70.72
82.44
51.33
72.75
90.96
71.53
56.62
61.38
65.32
67.54
71.10
72.62
83.92
45.54
69.89
85.77
67.91
62.45
61.38
64.62
68.52
60.28
71.65
83.17
48.26
71.29
88.29
69.67
59.39
61.38
64.97
67.84
Entertainment
Education
Sci/Tech
Economy
Law
World
Sports
Culture
Society
Average
Table 6. Cross-domain results (the best of 10 runs).
Deep LML (1 run)
Deep LML (best of 10 runs)
Domain
P (%) R (%) F1 (%)
P (%)
R (%)
F1 (%)
Life
52.33
70.72
82.44
51.33
72.75
90.96
71.53
56.62
61.38
65.32
67.54
71.10
72.62
83.92
45.54
69.89
85.77
67.91
62.45
61.38
64.62
68.52
60.28
71.65
83.17
48.26
71.29
88.29
69.67
59.39
61.38
64.97
67.84
70.41
68.38
80.12
67.63
68.76
84.91
66.11
43.93
61.95
77.02
68.92
77.50
68.68
75.69
53.92
72.88
87.10
63.41
61.43
63.79
80.52
70.49
73.80
68.53
77.84
60.00
70.76
85.99
64.73
51.23
62.85
78.73
69.45
Entertainment
Education
Sci/Tech
Economy
Law
World
Sports
Culture
Society
Average
In-domain: In this scenario, the same domain of training and testing data is used
for experiments. To evaluate the stability of the model, we created several datasets
by removing one domain (from both training and testing), i.e. only nine domains are
used for each experiment. The results of the experiments are shown in Table 7, where
the name of the domain, in each row, means that this domain is excluded from both
the training and testing datasets.
We found that Deep LML gave better results than CRFs and Bi-LSTM+CRF
with a F1 measurement of 70.51%. However, the results are not signi¯cantly higher.
Nevertheless, this result is reasonable because most of the labeled data in the training
phase are in the same domain of the evaluation phase. This indicates that in-domain
datasets have less noise, thus the performance of all the algorithms is improved.
Since, in-domain is not the main purpose of Deep LML, which focuses on
exploiting the knowledge from di®erent past domains, in the next subsections we
only focus on cross-domain experiments.
484 N.-V. Nguyen et al.
Table 7. In-domain results.
Bi-LSTM+CRF
CRF
Deep LML (1 run)
Domain
P (%) R (%) F1 (%)
P (%)
R (%) F1 (%) P (%) R (%) F1 (%)
Life
70.64
71.46
69.82
70.74
70.78
69.06
72.03
72.18
73.70
69.34
70.98
63.88
63.84
66.36
64.58
61.75
61.21
63.61
66.15
66.17
62.19
63.97
66.34
66.79
67.23
66.70
65.02
64.30
66.84
68.56
69.07
64.73
66.56
65.69
66.57
68.96
68.41
68.20
66.47
69.39
70.48
70.35
67.05
68.16
70.95
69.25
68.28
71.80
70.99
69.07
71.29
75.20
70.55
71.34
70.87
68.22
67.89
68.62
70.07
69.57
67.75
70.33
72.77
70.45
69.13
69.48
70.02
69.53
69.53
70.30
68.07
66.85
68.97
72.75
70.19
67.06
69.33
71.67
69.34
69.81
71.09
72.36
71.73
72.30
74.48
74.23
72.64
71.97
70.84
69.44
69.67
70.69
70.15
69.20
70.60
73.60
72.16
69.74
70.61
Entertainment
Education
Sci/Tech
Economy
Law
World
Sports
Culture
Society
Average
Due to the fact that the running time of Deep LML is long, all experiments in the
next subsections were run only once.
4.3.2. Deep LML with unlabeled data enhancement
After the unlabeled data enhancement step, the number of sentences in each domain
is reduced from 10% to 25%. In this experiment, we used the same parameters as
those in Sec. 5.3.1 i.e. the initial settings. The results in Table 8 proved the e®ec-
tiveness of the unlabeled data preprocessing. The F1 of 69.92% is 2.69% better than
that of single run Deep LML and also better than Deep LML with 10 runs.
4.3.3. Deep LML with both data preprocessing and parameter optimization
In this experiment, we applied both unlabeled data preprocessing and parameter
optimization, i.e. the parameters in Table 1. The results are given in Table 9.
Table 8. Cross-domain with unlabeled data preprocessing results.
Deep LML (1 run) Deep LML (best of 10 runs) Deep LML with data preprocessing
Domain
Life
P (%) R (%) F1 (%) P (%) R (%) F1 (%)
P (%)
R (%)
F1 (%)
52.33 71.10 60.28
70.41
68.38
80.12
67.63
68.76
84.91
66.11
43.93
61.95
77.02
68.92
77.50
68.68
75.69
53.92
72.88
87.10
63.41
61.43
63.79
80.52
70.49
73.80
68.53
77.84
60.00
70.76
85.99
64.73
51.23
62.85
78.73
69.45
54,92
68,22
79,84
51,97
74,06
90,96
74,50
61,49
67,27
64,29
68,75
76,11
76,81
93,52
47,57
72,73
84,13
70,33
63,83
66,11
65,41
71,65
63.80
72.26
86.14
49.68
73.39
87.41
72.36
62.64
66.68
64.85
69.92
Entertainment 70.72 72.62 71.65
Education
Sci/Tech
Economy
Law
82.44 83.92 83.17
51.33 45.54 48.26
72.75 69.89 71.29
90.96 85.77 88.29
71.53 67.91 69.67
56.62 62.45 59.39
61.38 61.38 61.38
65.32 64.62 64.97
67.54 68.52 67.84
World
Sports
Culture
Society
Average
Improving Named Entity Recognition in Vietnamese Texts 485
Table 9. Cross-domain with unlabeled data preprocessing and parameter optimization.
Deep LML
(1 run)
Deep LML with
data preprocessing
Deep LML w/data pre
& parameter optimization
Domain
P (%) R (%) F1 (%) P (%)
R (%)
F1 (%) P (%) R (%) F1 (%)
Life
52.33
70.72
82.44
51.33
72.75
90.96
71.53
56.62
61.38
65.32
67.54
71.10
72.62
83.92
45.54
69.89
85.77
67.91
62.45
61.38
64.62
68.52
60.28
71.65
83.17
48.26
71.29
88.29
69.67
59.39
61.38
64.97
67.84
54,92
68,22
79,84
51,97
74,06
90,96
74,50
61,49
67,27
64,29
68,75
76,11
76,81
93,52
47,57
72,73
84,13
70,33
63,83
66,11
65,41
71,65
63.80
72.26
86.14
49.68
73.39
87.41
72.36
62.64
66.68
64.85
69.92
60.65
73.15
83.74
53.26
72.23
89.65
72.31
58.76
68.01
64.29
69.60
70.88
78.29
86.26
49.27
75.25
90.59
69.25
63.86
67.11
65.41
71.62
65.37
75.63
84.98
51.19
73.71
90.12
70.75
61.21
67.56
64.85
70.53
Entertainment
Education
Sci/Tech
Economy
Law
World
Sports
Culture
Society
Average
In comparison with the initial (1 run) Deep LML, the combination of data en-
hancement with parameter adjustment helps to increase the performance of the NER
by 2.67%. This is evidence of our working in right direction.
5. Related Work
There are three tasks on Vietnamese NER in VLSP 2018, the Fifth International
Workshop on Vietnamese Language and Speech Processing in conjunction with
Conference on Computational Linguistics and Intelligent Text Processing (CICLing)
2018. All the three tasks focus on extracting four entity types: PER, LOC, ORG, and
MICS in the whole domain dataset of VLSP 2018.
Dong17 introduces three experimental systems for Vietnamese NER: a CRF-based
system (CRF-based approach), a 1-layer LSTM (LSTM-based approach for each
type of entity), and a 3-layer LSTM (joint model approach). For F1 measure, the
CRF-based system is 61%, the 1-layer LSTM is 54%, and the 3-layer LSTM is 52%.
Luong and Phan18 propose a system combined Bidirectional Long Short-Term
Memory and Conditional Random Field. First, the FastText algorithm is applied for
pre-training word embedding, then a BiLSTM is used to build a part of word
embedding from the characters. Second, new word embedding is fed to BiLSTM and
CRF to predict the NER tags. This system achieves an overall F1 measure of 74%
(Level 1) and 68% (Nested) on the standard VLSP NER dataset.
Pham19 proposes a feature-based NER model that combines word, word-shape
features, Brown-cluster-based features, and word-embedding-based features. The
NER model focuses on extracting three levels of entities: (i) Level-1 entities are those
that do not contain other entities inside, (i) Level-2 are those that contain only level-
1 entities inside, and (iii) Level-3 are those that contain at least one level-2 entity and
may contain some level-1 entities. The system achieved an overall F1 measure of
73.5% at all levels.
486 N.-V. Nguyen et al.
6. Conclusions and Future Work
This paper proposed a lifelong learning model using a combination of Bi-LSTM and
CRFs (DeepLML–NER model). The model retains pre¯x pattern knowledge gained
from the previous tasks and utilizes the knowledge to improve the performance of the
new task.
We also conducted the experiments on 10 domain datasets from the VLSP 2018
dataset. Experimental results have been shown to be satisfactory with the average
cross-domain and in-domain F1 measures of 69.45% and 70.51%, respectively. The
results show that lifelong learning methods can improve the performance of the
model based on prior knowledge.
Four entity types, i.e. PER, ORG, LOC, and MISC, do not have speci¯c char-
acteristics on the 10 domains, so other entity types should be considered in the
future. Moreover, other kinds of knowledge and solutions for ¯nding knowledge from
previous NER tasks and using the knowledge for future NER tasks should be studied.
Appendix A. The List of Sample Pre¯xes
The sample list of pre¯xes resulted from the Algorithm 1 is given in Table A.1.
Table A.1. Sample pre¯xes.
References
1. J. P. C. Chiu and E. Nichols, Named entity recognition with bidirectional LSTM-CNNs,
Transactions of the Association for Computational Linguistics, Association for Compu-
tational Linguistics 4 (2018) 357–370.
2. Z. Huang, W. Xu and K. Yu, Bidirectional LSTM-CRF models for sequence tagging,
Computing Research Repository (CoRR), abs/1508.01991, 2015.
3. G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami and C. Dyer, Neural archi-
tectures for named entity recognition, Proceedings of the 2016 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language
Technologies, Association for Computational Linguistics, pp. 260–270, abs/1603.01360,
2016.
4. X. Ma and E. H. Hovy, End-to-end sequence labeling via bi-directional LSTM-CNNS-
CRFs, Computing Research Repository (CoRR), abs/1603.01354, 2016.
Improving Named Entity Recognition in Vietnamese Texts 487
5. A. Bendale and T. E. Boult, Towards open world recognition, IEEE Conf. Computer
Vision and Pattern Recognition (CVPR), 2015, pp. 1893–1902.
6. Z. Chen and B. Liu, Lifelong Machine Learning, 2nd edn., Synthesis Lectures on Arti¯cial
Intelligence and Machine Learning (Morgan & Claypool Publishers, 2018).
7. D. L. Silver, Q. Yang and L. Li, Lifelong machine learning systems beyond learning
algorithms, AAAI Spring Symp. Lifelong Machine Learning 2013, Association for the
Advancement of Arti¯cial Intelligence, 2013.
8. S. Thrun and T. M. Mitchell, Lifelong robot learning, Robotics and Autonomous Systems,
Vol. 15 (Elsevier, 1995), pp. 25–46.
9. G. I. Parisi, J. Tani, C. Weber and S. Wermter, Lifelong learning of human actions
with deep neural network self-organization, Neural Networks, Vol. 96 (Elsevier, 2017),
pp. 137–149.
10. A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu,
R. Pascanu and R. Hadsell, Progressive neural networks, Computing Research Repository
(CoRR), abs/1606.04671, 2016.
11. J. D. La®erty, A. McCallum and F. C. N. Pereira, Conditional random ¯elds: Probabi-
listic models for segmenting and labeling sequence data, Proceedings of the Eighteenth
International Conference on Machine Learning (Morgan Kaufmann Publishers Inc.,
2001), pp. 282–289.
12. S. Hochreiter and J. Schmidhuber, Long short-term memory, Journal of Neural Com-
putation 9(8) (1997) 1735–1780.
13. M. Schuster and K. K. Paliwal, Bidirectional recurrent neural networks, IEEE Trans-
actions on Signal Processing 45(11) (1997) 2673–2681.
14. L. A. Ramshaw and M. Marcus, Text chunking using transformation-based learning,
Third Workshop on Very Large Corpora (ACL Anthology, 1995).
15. T.-H. Pham, X.-K. Pham, T.-A. Nguyen and P. Le-Hong, NNVLP: A neural network-
based Vietnamese language processing toolkit, Computing Research Repository (CoRR)
arXiv:1708.07241, 2017.
16. L. Shu, H. Xu and B. Liu, Lifelong learning CRF for supervised aspect extraction, Pro-
ceedings of the 55th Annual Meeting of the Association for Computational Linguistics,
Vol. 2, Association for Computational Linguistics, abs/1705.00251, 2017, pp. 148–154.
17. N. T. Dong, An investigation of Vietnamese nested entity recognition models, VLSP 2018
Workshop, 2018, pp. 14–16.
18. V.-T. Luong and L. K. Phan, Zaner: Vietnamese named entity recognition, VLSP 2018
Workshop, 2018, pp. 10–13.
19. Q. N. M. Pham, A feature-based model for nested named-entity recognition, VLSP 2018
Workshop, 2018, pp. 5–9.
17 trang
15680
Free
Bạn đang xem tài liệu "Improving named entity recognition in Vietnamese texts by a character-level deep lifelong learning model", để tải tài liệu gốc về máy hãy click vào nút Download ở trên
File đính kèm:
- improving_named_entity_recognition_in_vietnamese_texts_by_a.pdf
