TAILIEUCHUNG - Báo cáo khoa học: " Entropy Rate Constancy in Text"

We present a constancy rate principle governing language generation. We show that this principle implies that local measures of entropy (ignoring context) should increase with the sentence number. We demonstrate that this is indeed the case by measuring entropy in three different ways. We also show that this effect has both lexical (which words are used) and non-lexical (how the words are used) causes. | Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics ACL Philadelphia July 2002 pp. 199-206. Entropy Rate Constancy in Text Dmitriy Genzel and Eugene Charniak Brown Laboratory for Linguistic Information Processing Department of Computer Science Brown University Providence RI USA 02912 dg ec @ Abstract We present a constancy rate principle governing language generation. We show that this principle implies that local measures of entropy ignoring context should increase with the sentence number. We demonstrate that this is indeed the case by measuring entropy in three different ways. We also show that this effect has both lexical which words are used and non-lexical how the words are used causes. 1 Introduction It is well-known from Information Theory that the most efficient way to send information through noisy channels is at a constant rate. If humans try to communicate in the most efficient way then they must obey this principle. The communication medium we examine in this paper is text and we present some evidence that this principle holds here. Entropy is a measure of information first proposed by Shannon 1948 . Informally entropy of a random variable is proportional to the difficulty of correctly guessing the value of this variable when the distribution is known . Entropy is the highest when all values are equally probable and is lowest equal to 0 when one of the choices has probability of 1 . deterministically known in advance. In this paper we are concerned with entropy of English as exhibited through written text though these results can easily be extended to speech as well. The random variable we deal with is therefore a unit of text a word for our purposes1 that a random person who has produced all the previous words in the text stream is likely to produce next. We have as many random variables as we have words in a text. The distributions of these variables are obviously different and depend on all previous

• Báo cáo khoa học
• Entropy Rate Constancy in Text
• Dmitriy Genzel
• Eugene Charniak Brown
• báo cáo khoa học
• mô hình ngôn ngữ
• xử lý ngôn ngữ tự nhiên
• Bài giảng Xử lý ngôn ngữ tự nhiên
• Ngôn ngữ tự nhiên
• Kỹ thuật lập trình
• Xử lý nhập nhằng ngữ nghĩa
• Xử lý ngữ nghĩa
• Xử lý ngôn ngữ
• Natural Language Processing
• Ngôn ngữ lập trình
• Nhập nhằng cấu trúc
• Tri thức về ngôn ngữ
• Mô hình n gram
• Dịch máy
• Phương pháp dịch máy
• Hiểu ngôn ngữ
• Phân tích ngữ nghĩa
• Phân loại tin tự động
• Biểu diễn vị từ
• Thuộc tính về sự kiện
• Thuộc tính về sự kiện
• Mô hình ngôn ngữ Google Book N grams
• Mô hình ngôn ngữ KenLM
• Phương pháp làm mịn
• Tách từ tiếng Việt
• Gán nhãn từ loại
• Phân tích cú pháp
• Nghĩa từ vựng
• Phân giải nhập nhằng từ
• Phân loại văn bản
• Lexical chain
• Phản hồi thông tin
• Bài toán PTCP
• Cấu trúc ngữ pháp
• Phân tích cú pháp xác suất
• CKY kết hợp xác suất
• Văn phạm phi ngữ cảnh xác suất
• Hình thái học