TAILIEUCHUNG - Báo cáo khoa học: "An Iterative Algorithm to Build Chinese Language Models"

We present an iterative procedure to build a Chinese language model (LM). We segment Chinese text into words based on a word-based Chinese language model. However, the construction of a Chinese LM itself requires word boundaries. To get out of the chicken-and-egg problem, we propose an iterative procedure that alternates two operations: segmenting text into words and building an LM. Starting with an initial segmented corpus and an LM based upon it, we use a Viterbi-liek algorithm to segment another set of data. Then, we build an LM based on the second set and use the resulting LM to. | An Iterative Algorithm to Build Chinese Language Models Xiaoqiang Luo Center for Language and Speech Processing The Johns Hopkins University 3400 N. Charles St. Baltimore MD21218 USA Abstract We present an iterative procedure to build a Chinese language model LM . We segment Chinese text into words based on a word-based Chinese language model. However the construction of a Chinese LM itself requires word boundaries. To get out of the chicken-and-egg problem we propose an iterative procedure that alternates two operations segmenting text into words and building an LM. Starting with an initial segmented corpus and an LM based upon it we use a Viterbi-liek algorithm to segment another set of data. Then we build an LM based on the second set and use the resulting LM to segment again the first corpus. The alternating procedure provides a self-organized way for the segmenter to detect automatically unseen words and correct segmentation errors. Our preliminary experiment shows that the alternating procedure not only improves the accuracy of our segmentation but discovers unseen words surprisingly well. The resulting word-based LM has a perplexity of 188 for a general Chinese corpus. 1 Introduction In statistical speech recognition Bahl et al. 1983 it is necessary to build a language model LM for assigning probabilities to hypothesized sentences. The LM is usually built by collecting statistics of words over a large set of text data. While doing so is straightforward for English it is not trivial to collect statistics for Chinese words since word boundaries are not marked in written Chinese text. Chinese is a morphosyllabic language DeFrancis 1984 in that almost all Chinese characters represent a single syllable and most Chinese characters are also morphemes. Since a word can be multi-syllabic it is generally non-trivial to segment a Chinese sentence into words Wu and Tseng 1993 . Since segmentation is Salim Roukos IBM T. J. Watson Research Center Yorktown .

• Báo cáo khoa học
• An Iterative Algorithm
• Build Chinese Language Models
• Xiaoqiang Luo
• báo cáo khoa học
• mô hình ngôn ngữ
• xử lý ngôn ngữ tự nhiên
• An iterative greedy algorithm for sparsity constrained optimization
• Iterative greedy algorithm
• Sparsity constrained optimization
• Namely gradient matching pursuit
• Iterative Viterbi Algorithm
• K Best Sequential Decoding
• Zhiheng Huang
• báo cáo ngôn ngữ
• ngôn ngữ tự nhiên
• BMC Bioinformatics
• Iterative signature algorithm
• N6 methyladenosine
• Local functional blocks
• MeRIP Seq data
• Electron tomography
• Compressed sensing
• Matching pursuit
• Iterative reconstruction reprojection
• Theory of elasticity
• Incompressible materials
• Finite element method
• The Uzawa algorithm
• Iterative methods
• Systems of linear algebraic equations
• báo cáo hóa học
• công trình nghiên cứu về hóa học
• tài liệu về hóa học
• cách trình bày báo cáo
• tài liệu báo cáo nghiên cứu khoa học
• kiến thức toán học
• báo cáo toán học
• các báo cáo toán học hay
• Secondary structure prediction
• Hierarchical folding
• Minimum free energy
• RNA secondary structures
• Split feasibility problem
• Variational inequality
• Hillbert spaces
• Nonexpansive mapping
• Fixed point