TAILIEUCHUNG - Báo cáo khoa học: "A Comparison and Semi-Quantitative Analysis of Words and Character-Bigrams as Features in Chinese Text Categorization"

Words and character-bigrams are both used as features in Chinese text processing tasks, but no systematic comparison or analysis of their values as features for Chinese text categorization has been reported heretofore. We carry out here a full performance comparison between them by experiments on various document collections (including a manually word-segmented corpus as a golden standard), and a semi-quantitative analysis to elucidate the characteristics of their behavior; and try to provide some preliminary clue for feature term choice (in most cases, character-bigrams are better than words) and dimensionality setting in text categorization systems. . | A Comparison and Semi-Quantitative Analysis of Words and Character-Bigrams as Features in Chinese Text Categorization Jingyang Li Maosong Sun Xian Zhang National Lab. of Intelligent Technology Systems Department of Computer Sci. Tech. Tsinghua University Beijing 100084 China lijingyang@ sms@ kevinn9@ Abstract Words and character-bigrams are both used as features in Chinese text processing tasks but no systematic comparison or analysis of their values as features for Chinese text categorization has been reported heretofore. We carry out here a full performance comparison between them by experiments on various document collections including a manually word-segmented corpus as a golden standard and a semi-quantitative analysis to elucidate the characteristics of their behavior and try to provide some preliminary clue for feature term choice in most cases character-bigrams are better than words and dimensionality setting in text categorization systems. 1 Introduction1 Because of the popularity of the Vector Space Model VSM in text information processing document indexing term extraction acts as a pre-requisite step in most text information processing tasks such as Information Retrieval Baeza-Yates and Ribeiro-Neto 1999 and Text Categorization Sebastiani 2002 . It is empirically known that the indexing scheme is a nontrivial complication to system performance especially for some Asian languages in which there are no explicit word margins and even no natural semantic unit. Concretely in Chinese Text Categorization tasks the two most important index ing units feature terms are word and characterbigram so the problem is which kind of terms2 should be chosen as the feature terms words or character-bigrams To obtain an all-sided idea about feature choice beforehand we review here the possible feature variants or options . First at the word level we can do stemming do stop-word pruning include POS Part of Speech information etc. Second term .

• Báo cáo khoa học
• A Comparison and Semi Quantitative Analysis
• Words and Character Bigrams as Features
• Chinese Text Categorization
• báo cáo khoa học
• mô hình ngôn ngữ
• xử lý ngôn ngữ tự nhiên
• Bài giảng Xử lý ngôn ngữ tự nhiên
• Ngôn ngữ tự nhiên
• Kỹ thuật lập trình
• Xử lý nhập nhằng ngữ nghĩa
• Xử lý ngữ nghĩa
• Xử lý ngôn ngữ
• Natural Language Processing
• Ngôn ngữ lập trình
• Nhập nhằng cấu trúc
• Tri thức về ngôn ngữ
• Mô hình n gram
• Dịch máy
• Phương pháp dịch máy
• Hiểu ngôn ngữ
• Phân tích ngữ nghĩa
• Phân loại tin tự động
• Biểu diễn vị từ
• Thuộc tính về sự kiện
• Thuộc tính về sự kiện
• Mô hình ngôn ngữ Google Book N grams
• Mô hình ngôn ngữ KenLM
• Phương pháp làm mịn
• Tách từ tiếng Việt
• Gán nhãn từ loại
• Phân tích cú pháp
• Nghĩa từ vựng
• Phân giải nhập nhằng từ
• Phân loại văn bản
• Lexical chain
• Phản hồi thông tin
• Bài toán PTCP
• Cấu trúc ngữ pháp
• Phân tích cú pháp xác suất
• CKY kết hợp xác suất
• Văn phạm phi ngữ cảnh xác suất
• Hình thái học