TAILIEUCHUNG - Báo cáo khoa học: "A Hierarchical Bayesian Language Model based on Pitman-Yor Processes"

We propose a new hierarchical Bayesian n-gram model of natural languages. Our model makes use of a generalization of the commonly used Dirichlet distributions called Pitman-Yor processes which produce power-law distributions more closely resembling those in natural languages. We show that an approximation to the hierarchical Pitman-Yor language model recovers the exact formulation of interpolated Kneser-Ney, one of the best smoothing methods for n-gram language models. Experiments verify that our model gives cross entropy results superior to interpolated Kneser-Ney and comparable to modified Kneser-Ney. . | A Hierarchical Bayesian Language Model based on Pitman-Yor Processes Yee Whye Teh School of Computing National University of Singapore 3 Science Drive 2 Singapore 117543. tehyw@ Abstract We propose a new hierarchical Bayesian n-gram model of natural languages. Our model makes use of a generalization of the commonly used Dirichlet distributions called Pitman-Yor processes which produce power-law distributions more closely resembling those in natural languages. We show that an approximation to the hierarchical Pitman-Yor language model recovers the exact formulation of interpolated Kneser-Ney one of the best smoothing methods for n-gram language models. Experiments verify that our model gives cross entropy results superior to interpolated Kneser-Ney and comparable to modified Kneser-Ney. 1 Introduction Probabilistic language models are used extensively in a variety of linguistic applications including speech recognition handwriting recognition optical character recognition and machine translation. Most language models fall into the class of n-gram models which approximate the distribution over sentences using the conditional distribution of each word given a context consisting of only the previous n 1 words T P sentence JJ P word. wordi-n i 1 i 1 with n 3 trigram models being typical. Even for such a modest value of n the number of parameters is still tremendous due to the large vocabulary size. As a result direct maximum-likelihood parameter fitting severely overfits to the training data and smoothing methods are indispensible for proper training of n-gram models. A large number of smoothing methods have been proposed in the literature see Chen and Goodman 1998 Goodman 2001 Rosenfeld 2000 for good overviews . Most methods take a rather ad hoc approach where n-gram probabilities for various values of n are combined together using either interpolation or back-off schemes. Though some of these methods are intuitively appealing the main justification has

• Báo cáo khoa học
• A Hierarchical Bayesian
• Language Model
• Pitman Yor Processes
• báo cáo khoa học
• mô hình ngôn ngữ
• xử lý ngôn ngữ tự nhiên
• Hierarchical Bayesian
• Language Modelling
• the Linguistically Informed
• scientific reports
• model language
• process natural language
• Public health
• Integrated nested laplace approximation
• Latent gaussian model
• Bayesian hierarchical model
• Identifying risk factors
• BMC Bioinformatics
• Hierarchical latent variable models
• Misclassification errors
• Disease loci
• Human physiology
• Cox survival models
• Grouped predictors
• Hierarchical modeling
• Spike and slab prior
• Data integration
• Dirichlet multinomial
• Microbiome data
• Variable selection
• Bayesian networks
• Genetic regulatory networks
• Hierarchical bayesian modelling
• Biological interactions
• BMC Genomics
• HLA genotyping
• Next generation sequencing
• Whole genome sequencing
• Whole exome sequencing
• Cancer immunogenomics
• Markov chain Monte Carlo
• Genome Biology
• Alternative splicing
• Differential splicing
• Differential transcript usage
• Bayesian hierarchical modelling
• Non negative matrix factorization
• Bayesian hierarchical modeling
• Disease processes
• Analyze clinical data
• báo cáo hóa học
• công trình nghiên cứu về hóa học
• tài liệu về hóa học
• cách trình bày báo cáo
• RNA Seq
• Longitudinal data
• Correlated data
• Allelic imbalance
• Hierarchical generalized linear mixed model
• High throughput sequencing experiments
• Single nucleotide polymorphism
• Bayesian hierarchical models
• Flow cytometry
• Model based clustering
• Healthy individuals
• Differential expression
• Differential alternative splicing
• RNA seq experiments
• DNA methylation
• Bayesian analysis
• 5 methylcytosine oxidation
• Bisulfite sequencing
• Single cell RNA seq
• Cellular heterogeneity
• High throughput sequencing
• Powerful statistical models
• Meaningful biological signal
• Advantage illustrated
• Drosophila dataset