Đang chuẩn bị nút TẢI XUỐNG, xin hãy chờ
Tải xuống
Many machine translation (MT) evaluation metrics have been shown to correlate better with human judgment than BLEU. In principle, tuning on these metrics should yield better systems than tuning on BLEU. However, due to issues such as speed, requirements for linguistic resources, and optimization difficulty, they have not been widely adopted for tuning. | PORT a Precision-Order-Recall MT Evaluation Metric for Tuning Boxing Chen Roland Kuhn and Samuel Larkin National Research Council Canada 283 Alexandre-Taché Boulevard Gatineau Quebec Canada J8X 3X7 Boxing.Chen Roland.Kuhn Samuel.Larkin @nrc.ca Abstract Many machine translation MT evaluation metrics have been shown to correlate better with human judgment than BLEU. In principle tuning on these metrics should yield better systems than tuning on BLEU. However due to issues such as speed requirements for linguistic resources and optimization difficulty they have not been widely adopted for tuning. This paper presents PORT1 a new MT evaluation metric which combines precision recall and an ordering metric and which is primarily designed for tuning MT systems. PORT does not require external resources and is quick to compute. It has a better correlation with human judgment than BLEU. We compare PORT-tuned MT systems to BLEU-tuned baselines in five experimental conditions involving four language pairs. PORT tuning achieves consistently better performance than BLEU tuning according to four automated metrics including BLEU and to human evaluation in comparisons of outputs from 300 source sentences human judges preferred the PORT-tuned output 45.3 of the time vs. 32.7 BLEU tuning preferences and 22.0 ties . 1 Introduction Automatic evaluation metrics for machine translation MT quality are a key part of building statistical MT SMT systems. They play two 1 PORT Precision-Order-Recall Tunable metric. 930 roles to allow rapid though sometimes inaccurate comparisons between different systems or between different versions of the same system and to perform tuning of parameter values during system training. The latter has become important since the invention of minimum error rate training MERT Och 2003 and related tuning methods. These methods perform repeated decoding runs with different system parameter values which are tuned to optimize the value of the evaluation metric over a .