TAILIEUCHUNG - Báo cáo khoa học: "An Efficient Indexer for Large N-Gram Corpora"

Hakan Ceylan Department of Computer Science University of North Texas Denton, TX 76203 hakan@ Rada Mihalcea Department of Computer Science University of North Texas Denton, TX 76203 rada@ Abstract We introduce a new publicly available tool that implements efficient indexing and retrieval of large N-gram datasets, such as the Web1T 5-gram corpus. Our tool indexes the entire Web1T dataset with an index size of only 100 MB and performs a retrieval of any N-gram with a single disk access. With an increased index size of 420 MB and duplicate data, it also allows users to issue wild card queries provided that the wild. | An Efficient Indexer for Large N-Gram Corpora Hakan Ceylan Department of Computer Science University of North Texas Denton TX 76203 hakan@ Rada Mihalcea Department of Computer Science University of North Texas Denton TX 76203 rada@ Abstract We introduce a new publicly available tool that implements efficient indexing and retrieval of large N-gram datasets such as the Web1T 5-gram corpus. Our tool indexes the entire Web1T dataset with an index size of only 100 MB and performs a retrieval of any N-gram with a single disk access. With an increased index size of 420 MB and duplicate data it also allows users to issue wild card queries provided that the wild cards in the query are contiguous. Furthermore we also implement some of the smoothing algorithms that are designed specifically for large datasets and are shown to yield better language models than the traditional ones on the Web1T 5-gram corpus Yuret 2008 . We demonstrate the effectiveness of our tool and the smoothing algorithms on the English Lexical Substitution task by a simple implementation that gives considerable improvement over a basic language model. 1 Introduction The goal of statistical language modeling is to capture the properties of a language through a probability distribution so that the probabilities of word sequences can be estimated. Since the probability distribution is built from a corpus of the language by computing the frequencies of the N-grams found in the corpus the data sparsity is always an issue with the language models. Hence as it is the case with many statistical models used in Natural Language Processing NLP the models give a much better performance with larger data sets. However the large data sets such as the Web1T 5-Gram corpus of Brants and Franz 2006 present 103 a major challenge. The language models built from these sets cannot fit in memory hence efficient accessing of the N-gram frequencies becomes an issue. Trivial methods such as linear or binary search

• Báo cáo khoa học
• báo cáo khoa học
• báo cáo ngôn ngữ
• ngôn ngữ tự nhiên