TAILIEUCHUNG - Báo cáo khoa học: "Error Mining for Wide-Coverage Grammar Engineering"

Parsing systems which rely on hand-coded linguistic descriptions can only perform adequately in as far as these descriptions are correct and complete. The paper describes an error mining technique to discover problems in hand-coded linguistic descriptions for parsing such as grammars and lexicons. By analysing parse results for very large unannotated corpora, the technique discovers missing, incorrect or incomplete linguistic descriptions. The technique uses the frequency of n-grams of words for arbitrary values of n. It is shown how a new combination of suffix arrays and perfect hash finite automata allows an efficient implementation. . | Error Mining for Wide-Coverage Grammar Engineering Gertjan van Noord Alfa-informatica University of Groningen POBox716 9700 AS Groningen The Netherlands vannoord@ Abstract Parsing systems which rely on hand-coded linguistic descriptions can only perform adequately in as far as these descriptions are correct and complete. The paper describes an error mining technique to discover problems in hand-coded linguistic descriptions for parsing such as grammars and lexicons. By analysing parse results for very large unannotated corpora the technique discovers missing incorrect or incomplete linguistic descriptions. The technique uses the frequency of n-grams of words for arbitrary values of n. It is shown how a new combination of suffix arrays and perfect hash finite automata allows an efficient implementation. 1 Introduction As we all know hand-crafted linguistic descriptions such as wide-coverage grammars and large scale dictionaries contain mistakes and are incomplete. In the context of parsing people often construct sets of example sentences that the system should be able to parse correctly. If a sentence cannot be parsed it is a clear sign that something is wrong. This technique only works in as far as the problems that might occur have been anticipated. More recently tree-banks have become available and we can apply the parser to the sentences of the tree-bank and compare the resulting parse trees with the gold standard. Such techniques are limited however because treebanks are relatively small. This is a serious problem because the distribution of words is Zipfian there are very many words that occur very infrequently and the same appears to hold for syntactic constructions. In this paper an error mining technique is described which is very effective at automatically discovering systematic mistakes in a parser by using very large but unannotated corpora. The idea is very simple. We run the parser on a large set of sentences and then analyze those sentences

• Báo cáo khoa học
• Error Mining
• Wide Coverage Grammar Engineering
• Gertjan van Noord
• báo cáo khoa học
• mô hình ngôn ngữ
• xử lý ngôn ngữ tự nhiên
• Dependency Trees
• Long Papers
• báo cáo ngôn ngữ
• ngôn ngữ tự nhiên
• parsing results
• Benoît Sagot
• BMC Genomics
• Error correction
• Next generation sequencing
• Z score
• Machine induced errors
• Gamma distribution
• Bài giảng Xử lý ngôn ngữ tự nhiên
• Kỹ thuật lập trình
• Xử lý nhập nhằng ngữ nghĩa
• Xử lý ngữ nghĩa
• Xử lý ngôn ngữ
• Natural Language Processing
• Ngôn ngữ lập trình
• Nhập nhằng cấu trúc
• Tri thức về ngôn ngữ
• Mô hình n gram
• Dịch máy
• Phương pháp dịch máy
• Hiểu ngôn ngữ
• Phân tích ngữ nghĩa
• Phân loại tin tự động
• Biểu diễn vị từ
• Thuộc tính về sự kiện
• Thuộc tính về sự kiện
• Mô hình ngôn ngữ Google Book N grams
• Mô hình ngôn ngữ KenLM
• Phương pháp làm mịn
• Tách từ tiếng Việt
• Gán nhãn từ loại
• Phân tích cú pháp
• Nghĩa từ vựng
• Phân giải nhập nhằng từ
• Phân loại văn bản
• Lexical chain
• Phản hồi thông tin
• Bài toán PTCP
• Cấu trúc ngữ pháp
• Phân tích cú pháp xác suất
• CKY kết hợp xác suất
• Văn phạm phi ngữ cảnh xác suất
• Hình thái học