Đang chuẩn bị nút TẢI XUỐNG, xin hãy chờ
Tải xuống
This paper describes the personalized normalization of a multilingual chat system that supports chatting in user defined short-forms or abbreviations. One of the major challenges for multilingual chat realized through machine translation technology is the normalization of non-standard, self-created short-forms in the chat message to standard words before translation. | Personalized Normalization for a Multilingual Chat System Ai Ti Aw and Lian Hau Lee Human Language Technology Institute for Infocomm Research 1 Fusionopolis Way 21-01 Connexis Singapore 138632 aaiti@i2r.a-star. edu.sg Abstract This paper describes the personalized normalization of a multilingual chat system that supports chatting in user defined short-forms or abbreviations. One of the major challenges for multilingual chat realized through machine translation technology is the normalization of non-standard self-created short-forms in the chat message to standard words before translation. Due to the lack of training data and the variations of short-forms used among different social communities it is hard to normalize and translate chat messages if user uses vocabularies outside the training data and create short-forms freely. We develop a personalized chat normalizer for English and integrate it with a multilingual chat system allowing user to create and use personalized short-forms in multilingual chat. 1 Introduction Processing user-generated textual content on social media and networking usually encounters challenges due to the language used by the online community. Though some jargons of the online language has made their way into the standard dictionary a large portion of the abbreviations slang and context specific terms are still uncommon and only understood within the user community. Consequently content analysis or translation techniques developed for a more formal genre like news or even conversations cannot apply directly and effectively to the social media content. In recent years there are many works Aw et al. 2006 Cook et al. 2009 Han et al. 2011 on text normalization to preprocess user generated content such as tweets and short messages before further processing. The approaches include supervised or unsupervised methods based on morphological and phonetic variations. However most of the multilingual chat systems on the Internet have not yet .
