TAILIEUCHUNG - Báo cáo khoa học: "A Comparison of Document, Sentence, and Term Event Spaces"

The trend in information retrieval systems is from document to sub-document retrieval, such as sentences in a summarization system and words or phrases in question-answering system. Despite this trend, systems continue to model language at a document level using the inverse document frequency (IDF). In this paper, we compare and contrast IDF with inverse sentence frequency (ISF) and inverse term frequency (ITF). A direct comparison reveals that all language models are highly correlated; however, the average ISF and ITF values are and higher than IDF. . | A Comparison of Document Sentence and Term Event Spaces Catherine Blake School of Information and Library Science University of North Carolina at Chapel Hill North Carolina NC 27599-3360 cablake@ Abstract The trend in information retrieval systems is from document to sub-document retrieval such as sentences in a summarization system and words or phrases in question-answering system. Despite this trend systems continue to model language at a document level using the inverse document frequency IDF . In this paper we compare and contrast IDF with inverse sentence frequency ISF and inverse term frequency ITF . A direct comparison reveals that all language models are highly correlated however the average ISF and ITF values are and higher than IDF. All language models appeared to follow a power law distribution with a slope coefficient of for documents and for sentences and terms. We conclude with an analysis of IDF stability with respect to random journal and section partitions of the 100 830 full-text scientific articles in our experimental corpus. 1 Introduction The vector based information retrieval model identifies relevant documents by comparing query terms with terms from a document corpus. The most common corpus weighting scheme is the term frequency TF x inverse document frequency IDF where TF is the number of times a term appears in a document and IDF reflects the distribution of terms within the corpus Salton and Buckley 1988 . Ideally the system should assign the highest weights to terms with the most discriminative power. One component of the corpus weight is the language model used. The most common language model is the Inverse Document Frequency IDF which considers the distribution of terms between documents see equation 1 . IDF has played a central role in retrieval systems since it was first introduced more than thirty years ago Sparck Jones 1972 . IDF ti log2 N -log2 ni 1 1 N is the total number of corpus documents ni is .

• Báo cáo khoa học
• A Comparison of Document
• Sentence
• Term Event Spaces
• báo cáo khoa học
• mô hình ngôn ngữ
• xử lý ngôn ngữ tự nhiên
• Typical flexible rubber
• flaps around belt to minimize gaps
• Various types of prevalent
• water spraying arrangements – a comparison
• Layout of wet dust suppression
• system or stone crushers in USA
• Matrix Based
• Operatorial Approach
• Differential
• Integral Problems
• Comparison of Methodologies
• Analysis
• Bài giảng Xử lý ngôn ngữ tự nhiên
• Ngôn ngữ tự nhiên
• Kỹ thuật lập trình
• Xử lý nhập nhằng ngữ nghĩa
• Xử lý ngữ nghĩa
• Xử lý ngôn ngữ
• Natural Language Processing
• Ngôn ngữ lập trình
• Nhập nhằng cấu trúc
• Tri thức về ngôn ngữ
• Mô hình n gram
• Dịch máy
• Phương pháp dịch máy
• Hiểu ngôn ngữ
• Phân tích ngữ nghĩa
• Phân loại tin tự động
• Biểu diễn vị từ
• Thuộc tính về sự kiện
• Thuộc tính về sự kiện
• Mô hình ngôn ngữ Google Book N grams
• Mô hình ngôn ngữ KenLM
• Phương pháp làm mịn
• Tách từ tiếng Việt
• Gán nhãn từ loại
• Phân tích cú pháp
• Nghĩa từ vựng
• Phân giải nhập nhằng từ
• Phân loại văn bản
• Lexical chain
• Phản hồi thông tin
• Bài toán PTCP
• Cấu trúc ngữ pháp
• Phân tích cú pháp xác suất
• CKY kết hợp xác suất
• Văn phạm phi ngữ cảnh xác suất
• Hình thái học