TAILIEUCHUNG - Báo cáo khoa học: "A Part of Speech Estimation Method for Japanese Unknown Words using a Statistical Model of Morphology and Context"

We present a statistical model of Japanese unknown words consisting of a set of length and spelling models classified by the character types that constitute a word. The point is quite simple: different character sets should be treated differently and the changes between character types are very important because Japanese script has both ideograms like Chinese (kanji) and phonograms like English (katakana). Both word segmentation accuracy and part of speech tagging accuracy are improved by the proposed model. . | A Part of Speech Estimation Method for Japanese Unknown Words using a Statistical Model of Morphology and Context Masaaki NAGATA NTT Cyber Space Laboratories 1-1 Hikari-no-oka Yokosuka-Shi Kanagawa 239-0847 Japan Abstract We present a statistical model of Japanese unknown words consisting of a set of length and spelling models classified by the character types that constitute a word. The point is quite simple different character sets should be treated differently and the changes between character types are very important because Japanese script has both ideograms like Chinese kanji and phonograms like English katakana . Both word segmentation accuracy and part of speech tagging accuracy are improved by the proposed model. The model can achieve tagging accuracy if unknown words are correctly segmented. 1 Introduction In Japanese around 95 word segmentation accuracy is reported by using a word-based language model and the Viterbi-like dynamic programming procedures Nagata 1994 Yamamoto 1996 Takeuchi and Matsumoto 1997 Haruno and Matsumoto 1997 . About the same accuracy is reported in Chinese by statistical methods Sproat et al. 1996 . But there has been relatively little improvement in recent years because most of the remaining errors are due to unknown words. There are two approaches to solve this problem to increase the coverage of the dictionary Fung and Wu 1994 Chang et al. 1995 Mori and Nagao 1996 and to design a better model for unknown words Nagata 1996 Sproat et al. 1996 . We take the latter approach. To improve word segmentation accuracy Nagata 1996 used a single general purpose unknown word model while Sproat et al. 1996 used a set of specific word models such as for plurals personal names and transliterated foreign words. The goal of our research is to assign a correct part of speech to unknown word as well as identifying it correctly. In this paper we present a novel statistical model for Japanese unknown words. It .

• Báo cáo khoa học
• A Part of Speech Estimation Method
• Japanese Unknown Words
• a Statistical Model of Morphology and Context
• báo cáo khoa học
• mô hình ngôn ngữ
• xử lý ngôn ngữ tự nhiên
• Bài giảng Xử lý ngôn ngữ tự nhiên
• Ngôn ngữ tự nhiên
• Kỹ thuật lập trình
• Xử lý nhập nhằng ngữ nghĩa
• Xử lý ngữ nghĩa
• Xử lý ngôn ngữ
• Natural Language Processing
• Ngôn ngữ lập trình
• Nhập nhằng cấu trúc
• Tri thức về ngôn ngữ
• Mô hình n gram
• Dịch máy
• Phương pháp dịch máy
• Hiểu ngôn ngữ
• Phân tích ngữ nghĩa
• Phân loại tin tự động
• Biểu diễn vị từ
• Thuộc tính về sự kiện
• Thuộc tính về sự kiện
• Mô hình ngôn ngữ Google Book N grams
• Mô hình ngôn ngữ KenLM
• Phương pháp làm mịn
• Tách từ tiếng Việt
• Gán nhãn từ loại
• Phân tích cú pháp
• Nghĩa từ vựng
• Phân giải nhập nhằng từ
• Phân loại văn bản
• Lexical chain
• Phản hồi thông tin
• Bài toán PTCP
• Cấu trúc ngữ pháp
• Phân tích cú pháp xác suất
• CKY kết hợp xác suất
• Văn phạm phi ngữ cảnh xác suất
• Hình thái học