Alejandro Figuera Smoothing Methods for LM in IR

Smoothing Methods for LM in IR : Smoothing Methods for LM in IR Alejandro Figueroa


Outline : Outline The linguistic phenomena behind the retrieval of documents. Language Modeling Approach. Smoothing methods. Overview. Methods. Parameters setting. Interpolation vs. Back-off. Comparison of methods. Combination of methods. Personal outlook and conclusions.


The Linguistic Phenomena behind IR. : The Linguistic Phenomena behind IR. „Reducing Information Variation on Texts“ (Agata Savary and Christian Jacquemin). Work on our QA Group – DFKI.


Information Variation : Information Variation The problem: simply keyword matching is not enough to retrieve the best documents for a query. For example: „When was Albert Einstein born?„ The nobel prize of physics Albert Einstein was born in 1879 in Ulm, Germany. Born: 14 March 1879 in Ulm, Württemberg, Germany. Physics nobel prize Albert Einstein was born at Ulm, in Württemberg, Germany, on March 14, 1879. Died 18 Apr 1955 (born 14 Mar 1879) German-American physicist. The same information can be found in several ways:


Information Variation : Information Variation Kinds of variation: Graphic: "14 March 1879“ and "14 Mar 1879“. Morphological:” Physics nobel prize“ Syntactical: “German-American physicist“ Semantic:"Albert Einstein was born at Ulm“ and "German-American physicist“. Appropriateness: Precision. Economy.


Language Modeling Approach : Language Modeling Approach „A Study of smoothing methods for Language Models applied to Information Retrieval“ (Chengxiang Zhai and John Lafferty)


Language Modeling : Language Modeling The probability that a query Q was generated by a probabilistic model based on a document. Uni-gram model: P(q|d)?0


Language Modeling : Language Modeling Smoothing methods makes use of two probabilites for the model Pu(w|d) and Ps(w|d).


Language Modeling : Language Modeling carried out over the matched terms. Longer documents => less smoothing, longer documents => greater penalty!!.


Smoothing Methods : Smoothing Methods


Overview : Overview The problem: Adjust the MLE to compensate data sparseness. The role of smoothing is: LM more accurate. Explain the non-informative words in the query. Goal of the work: How sensitive is retrieval performance to the smoothing of a document LM? How should be the model and the parameters chosen?


Overview : Overview The unsmoothed model is the MLE:


Overview : Overview Smoothing: tackles the effect of statistical variability in small training sets. Discounting: the relative frequencies of seen events are discounted; the gained probability mass is then distributed over the unseen words.


Smoothing Methods : Smoothing Methods Based on the Good-turing idea: Estimate the probabilities of new events by taking the counts of singleton events, dividing it by the total number of events (0,1).


GooD-Turing Idea : GooD-Turing Idea The probability of a term with freq. tf is given by: Nd = Total number of terms occurred in d. Number of terms with frequency tf in a document. Expected value of Ntf. Total number of terms occurred in d.


Smoothing Methods : Smoothing Methods Jelinek-mercer method: involves a linear interpolation of the ML model with the collection model.


Smoothing Methods : Smoothing Methods Absolute discounting: decrease the probability of seen words by substracting a constant from their counts.


Smoothing Methods : Smoothing Methods Bayesian smoothing using Dirichlet priors: A multinomial distribution, for which the conjugate prior for bayesian analysis is the dirichlet distribution: The idea is to adjust the probabilities according to the query.


Summary: Smoothing Methods : Summary: Smoothing Methods


Parameters Setting : Parameters Setting 5 databases from TREC: Financial Times on disk 4. FBIS on disk 5. Los Angeles on disk 5. Disk 4 and disk 5 minus Congressional Record. The TREC8 web data. Queries: Topics 351-400 (TREC 7 ad-hoc task). Topics 401-450 (TREC 8 ad hoc web task).


Parameters Setting : Parameters Setting Number: 384