TAILIEUCHUNG - Báo cáo khoa học: "Automatic training of lemmatization rules that handle morphological changes in pre-, in- and suffixes alike"

We propose a method to automatically train lemmatization rules that handle prefix, infix and suffix changes to generate the lemma from the full form of a word. We explain how the lemmatization rules are created and how the lemmatizer works. We trained this lemmatizer on Danish, Dutch, English, German, Greek, Icelandic, Norwegian, Polish, Slovene and Swedish full form-lemma pairs respectively. We obtained significant improvements of 24 percent for Polish, percent for Dutch, percent for English, percent for German and percent for Swedish compared to plain suffix lemmatization using a suffix-only lemmatizer. Icelandic deteriorated with . | Automatic training of lemmatization rules that handle morphological changes in pre- in- and suffixes alike Bart Jongejan CST-University of Copenhagen Njalsgade 140-142 2300 Kobenhavn S Denmark bartj@ Hercules Dalianisf Ị fDSV KTH - Stockholm University Forum 100 164 40 Kista Sweden ỊEuroling AB SiteSeeker Igeldammsgatan 22c 112 49 Stockholm Sweden hercules@ Abstract We propose a method to automatically train lemmatization rules that handle prefix infix and suffix changes to generate the lemma from the full form of a word. We explain how the lemmatization rules are created and how the lemmatizer works. We trained this lemmatizer on Danish Dutch English German Greek Icelandic Norwegian Polish Slovene and Swedish full form-lemma pairs respectively. We obtained significant improvements of 24 percent for Polish percent for Dutch percent for English percent for German and percent for Swedish compared to plain suffix lemmatization using a suffix-only lem-matizer. Icelandic deteriorated with percent. We also made an observation regarding the number of produced lemmatization rules as a function of the number of training pairs. 1 Introduction Lemmatizers and stemmers are valuable human language technology tools to improve precision and recall in an information retrieval setting. For example stemming and lemmatization make it possible to match a query in one morphological form with a word in a document in another morphological form. Lemmatizers can also be used in lexicography to find new words in text material including the words frequency of use. Other applications are creation of index lists for book indexes as well as key word lists Lemmatization is the process of reducing a word to its base form normally the dictionary look-up form lemma of the word. A trivial way to do this is by dictionary look-up. More advanced systems use hand crafted or automatically generated transformation rules that look at the surface form of the word and .

• Báo cáo khoa học
• Automatic training of lemmatization rules
• handle morphological changes
• suffixes alike
• báo cáo khoa học
• báo cáo ngôn ngữ
• xử lý ngôn ngữ tự nhiên
• Bài giảng Xử lý ngôn ngữ tự nhiên
• Ngôn ngữ tự nhiên
• Kỹ thuật lập trình
• Xử lý nhập nhằng ngữ nghĩa
• Xử lý ngữ nghĩa
• Xử lý ngôn ngữ
• Natural Language Processing
• Ngôn ngữ lập trình
• Nhập nhằng cấu trúc
• Tri thức về ngôn ngữ
• Mô hình ngôn ngữ
• Mô hình n gram
• Dịch máy
• Phương pháp dịch máy
• Hiểu ngôn ngữ
• Phân tích ngữ nghĩa
• Phân loại tin tự động
• Biểu diễn vị từ
• Thuộc tính về sự kiện
• Thuộc tính về sự kiện
• Mô hình ngôn ngữ Google Book N grams
• Mô hình ngôn ngữ KenLM
• Phương pháp làm mịn
• Tách từ tiếng Việt
• Gán nhãn từ loại
• Phân tích cú pháp
• Nghĩa từ vựng
• Phân giải nhập nhằng từ
• Phân loại văn bản
• Lexical chain
• Phản hồi thông tin
• Bài toán PTCP
• Cấu trúc ngữ pháp
• Phân tích cú pháp xác suất
• CKY kết hợp xác suất
• Văn phạm phi ngữ cảnh xác suất
• Hình thái học