TAILIEUCHUNG - Báo cáo khoa học: "Stochastic Iterative Alignment for Machine Translation Evaluation"

A number of metrics for automatic evaluation of machine translation have been proposed in recent years, with some metrics focusing on measuring the adequacy of MT output, and other metrics focusing on fluency. Adequacy-oriented metrics such as BLEU measure n-gram overlap of MT outputs and their references, but do not represent sentence-level information. In contrast, fluency-oriented metrics such as ROUGE-W compute longest common subsequences, but ignore words not aligned by the LCS. | Stochastic Iterative Alignment for Machine Translation Evaluation Ding Liu and Daniel Gildea Department of Computer Science University of Rochester Rochester NY 14627 Abstract A number of metrics for automatic evaluation of machine translation have been proposed in recent years with some metrics focusing on measuring the adequacy of MT output and other metrics focusing on fluency. Adequacy-oriented metrics such as BLEU measure n-gram overlap of MT outputs and their references but do not represent sentence-level information. In contrast fluency-oriented metrics such as ROUGE-W compute longest common subsequences but ignore words not aligned by the LCS. We propose a metric based on stochastic iterative string alignment SIA which aims to combine the strengths of both approaches. We compare SIA with existing metrics and find that it outperforms them in overall evaluation and works specially well in fluency evaluation. 1 Introduction Evaluation has long been a stumbling block in the development of machine translation systems due to the simple fact that there are many correct translations for a given sentence. Human evaluation of system output is costly in both time and money leading to the rise of automatic evaluation metrics in recent years. In the 2003 Johns Hopkins Workshop on Speech and Language Engineering experiments on MT evaluation showed that BLEU and NIST do not correlate well with human judgments at the sentence level even when they correlate well over large test sets Blatz et al. 2003 . Liu and Gildea 2005 also pointed out that due to the limited references for every MT output using the overlapping ratio of n-grams longer than 2 did not improve sentence level evaluation performance of BLEU. The problem leads to an even worse result in BLEU S fluency evaluation which is supposed to rely on the long ngrams. In order to improve sentence-level evaluation performance several metrics have been proposed including ROUGE-W ROUGE-S Lin and Och 2004 and METEOR Banerjee

• Báo cáo khoa học
• Stochastic Iterative Alignment
• Machine Translation Evaluation
• Ding Liu and Daniel Gildea
• báo cáo khoa học
• mô hình ngôn ngữ
• xử lý ngôn ngữ tự nhiên
• Bài giảng Xử lý ngôn ngữ tự nhiên
• Ngôn ngữ tự nhiên
• Kỹ thuật lập trình
• Xử lý nhập nhằng ngữ nghĩa
• Xử lý ngữ nghĩa
• Xử lý ngôn ngữ
• Natural Language Processing
• Ngôn ngữ lập trình
• Nhập nhằng cấu trúc
• Tri thức về ngôn ngữ
• Mô hình n gram
• Dịch máy
• Phương pháp dịch máy
• Hiểu ngôn ngữ
• Phân tích ngữ nghĩa
• Phân loại tin tự động
• Biểu diễn vị từ
• Thuộc tính về sự kiện
• Thuộc tính về sự kiện
• Mô hình ngôn ngữ Google Book N grams
• Mô hình ngôn ngữ KenLM
• Phương pháp làm mịn
• Tách từ tiếng Việt
• Gán nhãn từ loại
• Phân tích cú pháp
• Nghĩa từ vựng
• Phân giải nhập nhằng từ
• Phân loại văn bản
• Lexical chain
• Phản hồi thông tin
• Bài toán PTCP
• Cấu trúc ngữ pháp
• Phân tích cú pháp xác suất
• CKY kết hợp xác suất
• Văn phạm phi ngữ cảnh xác suất
• Hình thái học