TAILIEUCHUNG - Báo cáo khoa học: "Evaluation challenges in large-scale document summarization"

We present a large-scale meta evaluation of eight evaluation measures for both single-document and multi-document summarizers. To this end we built a corpus consisting of (a) 100 Million automatic summaries using six summarizers and baselines at ten summary lengths in both English and Chinese, (b) more than 10,000 manual abstracts and extracts, and (c) 200 Million automatic document and summary retrievals using 20 queries. | Evaluation challenges in large-scale document summarization Dragomir R. Radev U. of Michigan radev@ Wai Lam Chinese U. of Hong Kong wlam@ Arda Celebi USC ISI ardax@ Simone Teufel U. of Cambridge John Blitzer U. of Pennsylvania blitzer@ Danyu Liu U. of Alabama liudy@ Horacio Saggion U. of Sheffield Hong Qi U. of Michigan hqi@ Elliott Drabek Johns Hopkins U. edrabek@ Abstract We present a large-scale meta evaluation of eight evaluation measures for both single-document and multi-document summarizers. To this end we built a corpus consisting of a 100 Million automatic summaries using six summarizers and baselines at ten summary lengths in both English and Chinese b more than 10 000 manual abstracts and extracts and c 200 Million automatic document and summary retrievals using 20 queries. We present both qualitative and quantitative results showing the strengths and drawbacks of all evaluation methods and how they rank the different summarizers. 1 Introduction Automatic document summarization is a field that has seen increasing attention from the NLP community in recent years. In part this is because summarization incorporates many important aspects of both natural language understanding and natural language generation. In part it is because effective automatic summarization would be useful in a variety of areas. Unfortunately evaluating automatic summarization in a standard and inexpensive way is a difficult task Mani et al. 2001 . Traditional large-scale evaluations are either too simplistic using measures like precision recall and percent agreement which 1 don t take chance agreement into account and 2 don t account for the fact that human judges don t agree which sentences should be in a summary or too expensive an approach using manual judgements can scale up to a few hundred summaries but not to tens or hundreds of thousands . In this .

• Báo cáo khoa học
• Evaluation challenges
• large scale document summarization
• Dragomir R
• Radev
• báo cáo khoa học
• mô hình ngôn ngữ
• xử lý ngôn ngữ tự nhiên
• trình bày báo cáo
• tài liệu báo cáo khoa học
• báo cáo y học
• kiến thức y học
• nghiên cứu y học
• Biomarker Discovery
• Experimental Design
• Toxicogenomic Approaches
• Risk Assessment
• Scientific Challenges
• Evaluation of Exposure
• Inverse Matrix Calculation
• Database Development
• Cluster Based Method
• Evaluation
• Trucks Weighing
• Control Stations
• Newborns
• Benefits and Challenges
• Pediatric Infectious
• Fungal Infections
• Patients of Paediatric Age
• Infantile Chronic
• SOA End to End Security
• Security Challenges in SOA
• Prototype Development
• Transition to Cloud Computing
• Demo Evaluation of the Proposed Solution
• Taint Analysis Subsystem
• Environmental challenges
• Environmental risk perception
• Government environmental regulation implementation behavior
• Public participation
• Structural equation model
• International Conference on Emerging Challenges
• Stochastic frontier analysis
• Earnings management
• Frontier markets
• Performance evaluation
• Banking industry
• BMC Veterinary Research
• Stress measurement
• Stress behaviour
• Hypothalamus pituitary adrenal axis
• Arginine vasopressin
• Physiological pathways
• Bài giảng Xử lý ngôn ngữ tự nhiên
• Ngôn ngữ tự nhiên
• Kỹ thuật lập trình
• Xử lý nhập nhằng ngữ nghĩa
• Xử lý ngữ nghĩa
• Xử lý ngôn ngữ
• Natural Language Processing
• Ngôn ngữ lập trình
• Nhập nhằng cấu trúc
• Tri thức về ngôn ngữ
• Mô hình n gram
• Dịch máy
• Phương pháp dịch máy
• Hiểu ngôn ngữ
• Phân tích ngữ nghĩa
• Phân loại tin tự động
• Biểu diễn vị từ
• Thuộc tính về sự kiện
• Thuộc tính về sự kiện
• Mô hình ngôn ngữ Google Book N grams
• Mô hình ngôn ngữ KenLM
• Phương pháp làm mịn
• Tách từ tiếng Việt
• Gán nhãn từ loại
• Phân tích cú pháp
• Nghĩa từ vựng
• Phân giải nhập nhằng từ
• Phân loại văn bản
• Lexical chain
• Phản hồi thông tin
• Bài toán PTCP
• Cấu trúc ngữ pháp
• Phân tích cú pháp xác suất
• CKY kết hợp xác suất
• Văn phạm phi ngữ cảnh xác suất
• Hình thái học