Đang chuẩn bị nút TẢI XUỐNG, xin hãy chờ
Tải xuống
In this paper we propose a new method for evaluating systems that extract temporal information from text. It uses temporal closure1 to reward relations that are equivalent but distinct. Our metric measures the overall performance of systems with a single score, making comparison between different systems straightforward. Our approach is easy to implement, intuitive, accurate, scalable and computationally inexpensive. | Temporal Evaluation Naushad UzZaman and James F. Allen Computer Science Department University of Rochester Rochester NY USA naushad james @cs.rochester.edu Abstract In this paper we propose a new method for evaluating systems that extract temporal information from text. It uses temporal closure1 to reward relations that are equivalent but distinct. Our metric measures the overall performance of systems with a single score making comparison between different systems straightforward. Our approach is easy to implement intuitive accurate scalable and computationally inexpensive. 1 Introduction The recent emergence of language processing applications like question answering information extraction and document summarization has motivated the need for temporally-aware systems. This along with the availability of the temporal annotation scheme TimeML Pustejovsky et al. 2003 a temporally annotated corpus TimeBank Pustejovsky et al. 2003 and the temporal evaluation challenges TempEval-1 Verhagen et al. 2007 and TempEval-2 Pustejovsky and Verhagen 2010 has led to an explosion of research on temporal information processing TIP . Prior evaluation methods TempEval-1 2 for different TIP subtasks have borrowed precision and recall measures from the information retrieval community. This has two problems First systems express temporal relations in different yet equivalent ways. Consider a scenario where the 1 Temporal closure is a reasoning mechanism that derives new implied temporal relations i.e. makes implicit temporal relations explicit. For example if we know A before B B before C then using temporal closure we can derive A before C. Allen 1983 demonstrates the closure table for 13 Allen interval relations. 351 reference annotation contains e1 e2 and e2 e3 and the system identifies the relation e1 e3. The traditional evaluation metric will fail to identify e1 e3 as a correct relation which is a logical consequence of the reference annotation. Second traditional evaluations tell
