TAILIEUCHUNG - Báo cáo khoa học: "Assessing the Effect of Inconsistent Assessors on Summarization Evaluation"

National Institute of Standards and Technology We investigate the consistency of human assessors involved in summarization evaluation to understand its effect on system ranking and automatic evaluation techniques. Using Text Analysis Conference data, we measure annotator consistency based on human scoring of summaries for Responsiveness, Readability, and Pyramid scoring. | Assessing the Effect of Inconsistent Assessors on Summarization Evaluation Karolina Owczarzak National Institute of Standards and Technology Gaithersburg MD 20899 Peter A. Rankel University of Maryland College Park Maryland rankel@ Hoa Trang Dang National Institute of Standards and Technology Gaithersburg MD 20899 John M. Conroy IDA Center for Computing Sciences Bowie Maryland conroy@ Abstract We investigate the consistency of human assessors involved in summarization evaluation to understand its effect on system ranking and automatic evaluation techniques. Using Text Analysis Conference data we measure annotator consistency based on human scoring of summaries for Responsiveness Readability and Pyramid scoring. We identify inconsistencies in the data and measure to what extent these inconsistencies affect the ranking of automatic summarization systems. Finally we examine the stability of automatic metrics ROUGE and CLASsY with respect to the inconsistent assessments. 1 Introduction Automatic summarization of documents is a research area that unfortunately depends on human feedback. Although attempts have been made at automating the evaluation of summaries none is so good as to remove the need for human assessors. Human judgment of summaries however is notper-fect either. We investigate two ways of measuring evaluation consistency in order to see what effect it has on summarization evaluation and training of automatic evaluation metrics. 2 Assessor consistency In the Text Analysis Conference TAC Summarization track participants are allowed to submit more than one run usually two and this option is often used to test different settings or versions of the same summarization system. In cases when the system versions are not too divergent they sometimes produce identical summaries for a given topic. Summaries are randomized within each topic before they are evaluated so the identical copies are usually

• Báo cáo khoa học
• Assessing the Effect
• Inconsistent Assessors
• Summarization Evaluation
• báo cáo khoa học
• báo cáo ngôn ngữ
• ngôn ngữ tự nhiên
• trình bày báo cáo
• tài liệu báo cáo khoa học
• báo cáo y học
• kiến thức y học
• nghiên cứu y học
• BMC Musculoskeletal Disorders
• Sedentary behaviour
• Physical activity
• Knee arthroplasty
• Objectively measured
• Self efficacy
• Perceived behavioral control
• Educational level
• Entrepreneurial intention
• Polish students
• Master's thesis of Engineering
• Master's thesis
• Anionic surfactant
• Non ionic surfactant
• Sequencing batch reactors
• The effect of grit on organizational commitment mediating
• Employability on organizational commitment mediating by job involvement
• Organizational commitment mediating
• The private higher education
• Waiting time
• Waiting environment
• Mind engagement
• Delay information
• Banking services
• Geotextile mulch
• Yield and physico chemical qualities
• Physico chemical qualities of litchi
• New technical approach
• Moisture conservation
• Organizational cultural values
• Employee culture fit
• Saudi banking sector
• Strong organizational culture
• Employees engagement on Performance excellence
• Thermal requirement
• Heat use efficiency
• Agro climatic indices
• Rainfed rice crop at Bhubaneswar
• Rainfed rice crop
• Heat transfer mechanism
• Gray correlation
• Gas production rate
• Wellhead fluid temperature
• Tubing wall
• Tubing insulation