TAILIEUCHUNG - Báo cáo khoa học: "Human Evaluation of a German Surface Realisation Ranker"

In this paper we present a human-based evaluation of surface realisation alternatives. We examine the relative rankings of naturally occurring corpus sentences and automatically generated strings chosen by statistical models (language model, loglinear model), as well as the naturalness of the strings chosen by the log-linear model. We also investigate to what extent preceding context has an effect on choice. We show that native speakers do accept quite some variation in word order, but there are also clearly factors that make certain realisation alternatives more natural. . | Human Evaluation of a German Surface Realisation Ranker Aoife Cahill Institut fur Maschinelle Sprachverarbeitung IMS University of Stuttgart 70174 Stuttgart Germany Martin Forst Palo Alto Research Center 3333 Coyote Hill Road Palo Alto CA 94304 USA mforst@ Abstract In this paper we present a human-based evaluation of surface realisation alternatives. We examine the relative rankings of naturally occurring corpus sentences and automatically generated strings chosen by statistical models language model log-linear model as well as the naturalness of the strings chosen by the log-linear model. We also investigate to what extent preceding context has an effect on choice. We show that native speakers do accept quite some variation in word order but there are also clearly factors that make certain realisation alternatives more natural. 1 Introduction An important component of research on surface realisation the task of generating strings for a given abstract representation is evaluation especially if we want to be able to compare across systems. There is consensus that exact match with respect to an actually observed corpus sentence is too strict a metric and that BLEU score measured against corpus sentences can only give a rough impression of the quality of the system output. It is unclear however what kind of metric would be most suitable for the evaluation of string realisations so that as a result there have been a range of automatic metrics applied including inter alia exact match string edit distance NIST SSA BLEU NIST ROUGE generation string accuracy generation tree accuracy word accuracy Bangalore et al. 2000 Callaway 2003 Nakanishi et al. 2005 Velldal and Oepen 2006 Belz and Reiter 2006 . It is not always clear how appropriate these metrics are especially at the level of individual sentences. Using automatic evaluation metrics cannot be avoided but ideally a metric for the evaluation of realisation rankers would rank .

• Báo cáo khoa học
• Human Evaluation
• German Surface Realisation Ranker
• Long Papers
• scientific reports
• model language
• process natural language
• Lecture Human Computer interaction
• Bài giảng Tương tác Người Máy tính
• Human Computer interaction
• Introduction to evaluation
• Evaluation paradigms
• Evaluation techniques
• Framework to guide evaluation process
• The evaluation addresses
• Evaluation paradigm
• MT Evaluation
• Human like
• Human Acceptable
• Enrique Amig
• báo cáo khoa học
• mô hình ngôn ngữ
• xử lý ngôn ngữ tự nhiên
• Humanities and social science
• Human intangible asset evaluation
• The master goldsmith’s figure
• Master goldsmith
• Jewelry sector
• Human development
• Economic projects
• The indicators
• Economic evaluation of some projects funded
• Human development in fayoum governorate
• Heuristic evaluation
• Web navigation design
• Department store navigation
• Navigation embodies
• Global navigation
• Trunk test
• Bài giảng Tương tác người máy
• Evaluating Designs
• Assessing Dialog System
• Simulation Evaluation
• Measures Using Human Judges
• báo cáo ngôn ngữ
• Correlation between ROUGE
• Extractive Meeting Summaries
• Correlating Human
• Automatic Evaluation
• German Surface Realiser
• A study on the morphology
• Sequential evaluation
• Human embryos
• IVF patients
• Morphological correlations
• Comparing Automatic
• NLG Systems
• International Journal of Computer Science and Telecommunications
• Quasi steady flow dynamics study
• Human aortic valve
• An evaluation with numerical techniques
• Optimize the prosthesis aortic design
• Iimpingement takes place
• Journal of computer science and information technology
• Tace elements in eneolithic
• Late medieval human bones
• Two archaeological sites in tuscany
• Evaluation of diagenetic processes
• Diet and exposure to heavy metals
• Conceptual framework
• Software interface
• Strategic nature
• Developing an interactive product
• Observing users
• Dirty evaluation techniques
• Data collection techniques
• Bone marrow
• Mesenchymal stem cell
• Evaluation of biocompatibility
• Tissue engineering
• Biological behavior of human MSCs
• Market Evaluation
• the Firm
• thesis
• internship reports
• project course
• graduate thesis
• Multimodal user interfaces
• Experimental evaluation
• Pursuit tracking
• Human factors
• Multimodal feedback
• BMC Bioinformatics
• Word similarity
• Intrinsic evaluation
• Downstream tasks
• Human similarity judgements
• Journal of Chemistry
• Bismuth film electrode (BiFE)
• Linear sweep cathodic stripping voltammetry
• Electrochemical evaluation
• Cationic surfactant
• Second generation antihistamine
• The hospitality industry
• Human resources
• Evaluating performance
• Making the evaluation
• Performance review process