TAILIEUCHUNG - Báo cáo khoa học: "Subword-based Tagging for Confidence-dependent Chinese Word Segmentation"

We proposed a subword-based tagging for Chinese word segmentation to improve the existing character-based tagging. The subword-based tagging was implemented using the maximum entropy (MaxEnt) and the conditional random fields (CRF) methods. We found that the proposed subword-based tagging outperformed the character-based tagging in all comparative experiments. In addition, we proposed a confidence measure approach to combine the results of a dictionary-based and a subword-tagging-based segmentation. . | Subword-based Tagging for Confidence-dependent Chinese Word Segmentation Ruiqiang Zhang1 2 and Genichiro Kikui and Eiichiro Sumita1 2 1National Institute of Information and Communications Technology 2ATR Spoken Language Communication Research Laboratories 2-2-2 Hikaridai Seiika-cho Soraku-gun Kyoto 619-0288 Japan @ Abstract We proposed a subword-based tagging for Chinese word segmentation to improve the existing character-based tagging. The subword-based tagging was implemented using the maximum entropy MaxEnt and the conditional random fields CRF methods. We found that the proposed subword-based tagging outperformed the character-based tagging in all comparative experiments. In addition we proposed a confidence measure approach to combine the results of a dictionary-based and a subword-tagging-based segmentation. This approach can produce an ideal tradeoff between the in-vocaulary rate and out-of-vocabulary rate. Our techniques were evaluated using the test data from Sighan Bakeoff 2005. We achieved higher F-scores than the best results in three of the four corpora PKU CITYU and MSR . 1 Introduction Many approaches have been proposed in Chinese word segmentation in the past decades. Segmentation performance has been improved significantly from the earliest maximal match dictionary-based approaches to HMM-based Zhang et al. 2003 approaches and recent state-of-the-art machine learning approaches such as maximum entropy MaxEnt Xue and Shen 2003 support vector machine Now the second author is affiliated with NTT. SVM Kudo and Matsumoto 2001 conditional random fields CRF Peng and McCallum 2004 and minimum error rate training Gao et al. 2004 . By analyzing the top results in the first and second Bakeoffs Sproat and Emerson 2003 and Emerson 2005 we found the top results were produced by direct or indirect use of so-called IOB tagging which converts the problem of word segmentation into one of character tagging so .

• Báo cáo khoa học
• Subword based Tagging
• Confidence dependent Chinese Word Segmentation
• Ruiqiang Zhang
• báo cáo khoa học
• mô hình ngôn ngữ
• xử lý ngôn ngữ tự nhiên
• Bài giảng Xử lý ngôn ngữ tự nhiên
• Ngôn ngữ tự nhiên
• Kỹ thuật lập trình
• Xử lý nhập nhằng ngữ nghĩa
• Xử lý ngữ nghĩa
• Xử lý ngôn ngữ
• Natural Language Processing
• Ngôn ngữ lập trình
• Nhập nhằng cấu trúc
• Tri thức về ngôn ngữ
• Mô hình n gram
• Dịch máy
• Phương pháp dịch máy
• Hiểu ngôn ngữ
• Phân tích ngữ nghĩa
• Phân loại tin tự động
• Biểu diễn vị từ
• Thuộc tính về sự kiện
• Thuộc tính về sự kiện
• Mô hình ngôn ngữ Google Book N grams
• Mô hình ngôn ngữ KenLM
• Phương pháp làm mịn
• Tách từ tiếng Việt
• Gán nhãn từ loại
• Phân tích cú pháp
• Nghĩa từ vựng
• Phân giải nhập nhằng từ
• Phân loại văn bản
• Lexical chain
• Phản hồi thông tin
• Bài toán PTCP
• Cấu trúc ngữ pháp
• Phân tích cú pháp xác suất
• CKY kết hợp xác suất
• Văn phạm phi ngữ cảnh xác suất
• Hình thái học