Overview of Question Answering: Overview of Question Answering Svetlana Stenchikova “QA with Semantic Roles” - Internship project at AT&T with the mentors: Dilek Hakkani-Tür and Gokhan Tur


Presentation Plan: Presentation Plan Overview of Question Answering QA Baseline Approach TREC competition


QA is different from Search: QA is different from Search


Example: Where did Maya people live?: Example: Where did Maya people live? the Maya people lived in: a large cultural area that archaeologists call Mesoamerica south-central Mexico the region that is now Eastern and Southern Mexico, Guatemala, Belize, El Salvador and Western Honduras small scattered villages of pole and thatch houses QA System:


QA Scenarios: QA Scenarios Limited Domain – Fixed set of documents Information is more reliable but sparse Data preprocessing is feasible Unlimited domain – Web Greater redundancy, but lower reliability On-demand data processing TREC competition


Slide6: QA System Architecture


Related Work: Related Work Insight(Russia): extensive use of patterns START (MIT): first QA system on the web. Uses Knowledge Annotation Technique: annotating resources with metadata Use specific data source for different queries


Slide8: Related Work AskMSR: count 1-, 2-, 3-grams on the snippet outputs, finds the most frequent item. Use answer tiling to generate longer answers Insight(Russia) START (MIT)


Slide9: Related Work AskMSR Insight(Russia) LCC: Extract Logical Form from questions and answers use ontology to “paraphrase” use logic provers for answer verification Search for answer in a paragraph instead of a sentence START (MIT).


Presentation Plan: Presentation Plan Overview of Question Answering QA Baseline Approach TREC competition


Slide11: QA System Architecture Who won the Nobel prize in literature in 1988? “won the Nobel prize in literature in 1988” http://www.nytimes.com/books/99/04/18/specials/rushdie-rally.html http://www.newyorker.com/printables/fact/040712fa_fact1 … Naguib Mahfouz won the nobel prize in literature in 1988 Egyptian writer won the nobel prize in literature in 1988 A distinguished Egyptian writer Naguib Mahfouz won the nobel prize in literature in 1988 Naguib Mahfouz Egyptian writer 2/3 1/3


Candidate Sentences – Actual Scenario: Candidate Sentences – Actual Scenario The veritable terrorism of which he is a target is unjustifiable , indefensible , " wrote Naguib Mahfouz , the Egyptian who won the Nobel Prize in Literature in 1988 Mahfouz , who won the Nobel Prize in Literature in 1988 , is known for his intimate stories of Egyptian life …


Slide13: Phrase Extraction and Search Methods Search: find candidate documents Exact Phrase Conjunction of sub-phrases (inexact phrase)


Slide14: Phrase Extraction and Search: Exact Phrase If a question contains an auxiliary verb (is/was/were/are/did/do/does): drop all words between the 'wh' word and the auxiliary match the tense of the predicate with the auxiliary verb. When did Vesuvius last erupt? ~ “Vesuvius last erupted” What state does Martha Stewart live in ~ “Martha Stewart lives in”


Slide15: Phrase Extraction and Search : Inexact Phrase SRL-parse the question, identifying the argument phrases: Who invented the electric guitar? [ARG0 Who] [TARGET invented] [ARG1 the electric guitar] Search for “invented” AND “the electric guitar” Searching: Use Google search API


Slide16: Sentence Extraction Methods Sentence Extraction: finding candidate sentences Exact phrase Conjunction of phrases Predicate


Evaluation of Search and Sentence Extraction: Evaluation of Search and Sentence Extraction Accuracy: % of candidate sentences containing a correct answer Coverage: % of the queries containing at least one correct answer in the set of candidate sentences


Slide18: SRL can improve the result ONLY when a correct answer is present in candidate sentences. Future work: Improve Search/extraction Using synonyms for search and extractions Back-off on prepositional clauses Future Work on Sentence Extraction


Baseline Answer Extraction: Baseline Answer Extraction Single template: Answer appears on the left of the right side of the search phrase Question: Who invented electric guitar? Phrase: “invented the electric guitar” Pattern: […] invented the electric guitar Question: When was Nostradamus born? Phrase: “Nostradamus was born” Pattern : Nostradamus was born in […]


SRL Architecture: SRL Architecture Q: Who created the comic strip Garfield? Predicate: created Argument type: ARG0 SRL Parse: [ARG0 Who] [TARGET created ] [ARG1 the comic strip Garfield] Task: Find ARG0 of the TARGET created Semantic Role module Semantic Role module


Slide21: SRL Architecture Q: Who created the comic strip Garfield? Predicate: created Argument type: ARG0 SRL Candidate Sentence: Garfield is [ARG1 a popular comic strip] [TARGET created ] [ARG0 by Jim Davis] featuring the cat Garfield ... Extracted Answers: by Jim Davis Semantic Role module Semantic Role module


Automatic Evaluation: Mean Reciprocal Ranking (MRR) and precision of correct answer: Automatic Evaluation: Mean Reciprocal Ranking (MRR) and precision of correct answer 0.24 => 0.35 is a statistically significant improvement MRR: question gets score = 1/N where N is the index of the first correct answer Use 5 top answers Precision(% questions with first answer being correct)


Measure of Conciseness and Irrelevant Information in Correct Answers: Measure of Conciseness and Irrelevant Information in Correct Answers * This experiment was done manually


Presentation Plan: Presentation Plan Overview of Question Answering QA Baseline Approach TREC competition


TREC competition: TREC competition TREC competitions QA track ~ Since 1999 Set of factoid/definition/context questions. Data Set 3 GB


Slide26: TREC history


TREC 2004 - 2006 main task: TREC 2004 - 2006 main task Define a topic target for a set of questions: Person George Bush Thing Shiite Organization Amway Event return of Hong Kong to Chinese sovereignty (Event was introduced in 2005) Question Types: List Factoid Other The answer to the "Other" question is to be interesting information about the target that is not covered by the preceding questions in the series.


TREC Main Task Question Example: TREC Main Task Question Example target text="Shiite“ FACTOID    Who was the first Imam of the Shiite sect of Islam?  FACTOID  Where is his tomb? FACTOID    What was this person's relationship to the Prophet Mohammad? FACTOID    Who was the third Imam of Shiite Muslims? FACTOID  When did he die? FACTOID  What portion of Muslims are Shiite? LIST What Shiite leaders were killed in Pakistan? OTHER    Other


Scoring : Scoring 1/3*factoid-score + 1/3*list-score + 1/3*other-score. Factoid answers are judged as: "incorrect", "unsupported", "non-exact", "locally correct", or "globally correct" factoid-score= % "globally correct"


Factoid and List Answer Scoring: Factoid and List Answer Scoring Factoid: count an answer as correct if the answer string consists of exactly the right answer (and supported by the document). Report accuracy - % of correct answers List: An individual instance is interpreted as for factoid questions IR = # instances judged correct & distinct/|final answer set| IP = # instances judged correct & distinct/# instances returned F = (2*IP*IR)/(IP+IR)


Other Question Scoring: Other Question Scoring Inspired by: Jimmy Lin and Dina Demner-Fushman. Will Pyramids Built of Nuggets Topple Over? the assessor creates a list of acceptable information nuggets from the union of the returned responses and the information discovered during question development Assessor marks “vital” and “Okay” nuggets present in the system’s answer Precision and recall are computed as a function of # VITAL and # OKAY nuggets in response


TREC 2005 “Factoid” Questions Evaluation: TREC 2005 “Factoid” Questions Evaluation


TREC 2005 “List” Questions Results: TREC 2005 “List” Questions Results


TREC 2005 “other” Questions Results: TREC 2005 “other” Questions Results


Complex Interactive QA Track: Complex Interactive QA Track Questions were created by 4 military analysts Evaluated the same way as “other” in Main QA task In 2005 competition scores range from .061 - .276


Complex Interactive QA Track: Complex Interactive QA Track What evidence is there for transport of [goods] from [entity] to [entity]? What [relationship] exist between [entity] and [entity]? where [relationship] is a element of {"financial relationships", "organizational ties", "familial ties", "common interests"} What influence/effect do(es) [entity] have on/in [entity]? What is the position of [entity] with respect to [issue]? Is there evidence to support the involvement of [entity] in [event/entity]?


Question Example: Question Example   What evidence is there for transport of [drugs] from [Bonaire] to [the United States]? The analyst would like to know of efforts made to discourage narco traffickers from using Bonaire as a transit point for drugs to the United States. Specifically, the analyst would like to know of any efforts by local authorities as well as the international community.  


References: References http://www.umiacs.umd.edu/~jimmylin/ciqa/guidelines.html Overview of the TREC 2005 Question Answering Track E.M. Voorhees, H.T. Dang, NIST http://trec.nist.gov/act_part/tracks/qa/qa.06.guidelines.html


THANK YOU: THANK YOU