TAILIEUCHUNG - Báo cáo khoa học: "ALIGNING SENTENCES IN PARALLEL CORPORA"

In this paper we describe a statistical technique for aligning sentences with their translations in two parallel corpora. In addition to certain anchor points that are available in our , the only information about the sentences that we use for calculating alignments is the number of tokens that they contain. Because we make no use of the lexical details of the sentence, the alignment computation is fast and therefore practical for application to very large collections of text. | ALIGNING SENTENCES IN PARALLEL CORPORA Peter F. Brown Jennifer c. Lai and Robert L. Mercer IBM Thomas J. Watson Research Center . Box 704 Yorktown Heights NY 10598 ABSTRACT In this paper we describe a statistical technique for aligning sentences with their translations in two parallel corpora. In addition to certain anchor points that are available in our data the only information about the sentences that we use for calculating alignments is the number of tokens that they contain. Because we make no use of the lexical details of the sentence the alignment computation is fast and therefore practical for application to very large collections of text. We have used this technique to align several million sentences in the English-French Hansard corpora and have achieved an accuracy in excess of 99 in a random selected set of 1000 sentence pairs that we checked by hand. We show that even without the benefit of anchor points the correlation between the lengths of aligned sentences is strong enough that we should expect to achieve an accuracy of between 96 and 97 . Thus the technique may be applicable to a wider variety of texts than we have yet tried. INTRODUCTION Recent work by Brown et al. Brown et al. 1988 Brown et al. 1990 has quickened anew the long dormant idea OÍ using statistical techniques to carry out machine translation from one natural language to another. The lynchpin of their approach is a large collection of pairs of sentences that are mutual translations. Beyond providing grist to the statistical mill such pairs of sentences are valuable to researchers in bilingual lexicography Kla-vans and Tzoukermann 1990 Warwick and Russell 1990 and may be useful in other approaches to machine translation Sadler. 1989 . In this paper we consider the problem of extracting from parallel French and English corpora pairs sentences that are translations of one another. The task is not trivial because at times a. single sentence in one language is translated as two or more

• Báo cáo khoa học
• ALIGNING SENTENCES
• PARALLEL CORPORA
• Peter F
• Brown
• báo cáo khoa học
• mô hình ngôn ngữ
• xử lý ngôn ngữ tự nhiên
• Reliable Measures
• Aligning Japanese English
• News Articles and Sentences
• A PROGRAM FOR ALIGNING SENTENCES
• BILINGUAL CORPORA
• William A
• Gale Kenneth W
• Church
• LEXICAL INFORMATION
• Bài giảng Xử lý ngôn ngữ tự nhiên
• Ngôn ngữ tự nhiên
• Kỹ thuật lập trình
• Xử lý nhập nhằng ngữ nghĩa
• Xử lý ngữ nghĩa
• Xử lý ngôn ngữ
• Natural Language Processing
• Ngôn ngữ lập trình
• Nhập nhằng cấu trúc
• Tri thức về ngôn ngữ
• Mô hình n gram
• Dịch máy
• Phương pháp dịch máy
• Hiểu ngôn ngữ
• Phân tích ngữ nghĩa
• Phân loại tin tự động
• Biểu diễn vị từ
• Thuộc tính về sự kiện
• Thuộc tính về sự kiện
• Mô hình ngôn ngữ Google Book N grams
• Mô hình ngôn ngữ KenLM
• Phương pháp làm mịn
• Tách từ tiếng Việt
• Gán nhãn từ loại
• Phân tích cú pháp
• Nghĩa từ vựng
• Phân giải nhập nhằng từ
• Phân loại văn bản
• Lexical chain
• Phản hồi thông tin
• Bài toán PTCP
• Cấu trúc ngữ pháp
• Phân tích cú pháp xác suất
• CKY kết hợp xác suất
• Văn phạm phi ngữ cảnh xác suất
• Hình thái học