MEI NAACL Presentation

Mandarin-English Information (MEI):Investigating Translingual Speech RetrievalJohns Hopkins University Summer Workshop 2000: 

Mandarin-English Information (MEI): Investigating Translingual Speech Retrieval Johns Hopkins University Summer Workshop 2000 Presented at the ANLP-NAACL 2000 Embedded Machine Translation Systems Workshop The MEI Team

MEI Team: 

MEI Team Senior Members Students Helen Meng Chinese University of Hong Kong Erika Grams Advanced Analytic Tools Sanjeev Khudanpur Johns Hopkins University Gina-Anne Levow University of Maryland Douglas Oard University of Maryland Patrick Schone US Department of Defense Hsin-Min Wang Academia Sinica, Taiwan Berlin Chen National Taiwan University Wai-Kit Lo Chinese University of Hong Kong Karen Tang Princeton University Jianqiang Wang University of Maryland

Outline: 

Outline Audio indexing MEI Project overview Research challenges System architecture Collaboration opportunities

Motivation: 

Motivation Speech retrieval applications are emerging e.g., http://speechbot.research.compaq.com source: www.real.com, Feb 2000 Internet-accessible Radio and Television Stations

The Big Picture: 

The Big Picture Speech to Speech Translation Translingual Audio Browsing Translingual Audio Search English Query English Audio Select Examine MEI

Related Work: 

Related Work TREC Spoken Document Retrieval Close coupling of recognition and retrieval TREC Cross-Language Retrieval Close coupling of translation and retrieval TDT-3 Topic Tracking Coupling recognition, translation and retrieval Using speech recognition transcripts

The MEI Project: 

The MEI Project Closely couple recognition and translation For the purpose of retrieval Using English examples, find Mandarin audio

Research Challenges: 

Research Challenges Multi-scale audio indexing Multiple feature sets capture more information Multi-scale translation Lexicon and pronunciation are complementary Multi-scale retrieval Combination of evidence can add robustness

Multi-scale Mandarin Audio Indexing: 

Multi-scale Mandarin Audio Indexing

Multi-scale Translation: 

Word-scale Dictionary-based [Levow & Oard 00] Parallel corpora [Nie 99] Comparable corpora [Fung 98] Subword-scale [Knight & Graehl 97] Cross-language phonetic mapping /bei2 ai4 er3 lan2/ Kosovo (/ke1-sou3-wo4/, /ke1-sou3-fo2/, /ke1-sou3-fu1/, /ke1-sou3-fu2/) Multi-scale Translation

Cross-Language Phonetic Mapping: 

Cross-Language Phonetic Mapping Syllabify English spelling e.g. Jiang Zemin, Shandong Province Map English pronunciation to Mandarin Convert phonemes to pinyin e.g. /k ow s ax v ow/ to /ke1-suo3-wo4/ Plan to investigate alternative techniques Rule-based Statistical mapping

Multi-scale Retrieval: 

Multi-scale Retrieval Word-scale exploits lexical knowledge Enhances precision Subwords can achieve complete coverage Enhances recall Combination of evidence may be best If a good merging strategy can be found

Multi-scale Retrieval Techniques: 

Multi-scale Retrieval Techniques Subword-scale Syllable lattice matching [Chen, Wang & Lee 00] Overlapping syllable n-grams [Meng et al. 99] Syllable confusion matrix [Meng et al. 99] Word-scale Structured queries [Pirkola 98] Structured translation [Sperer & Oard 00]

Merging Strategies: 

Merging Strategies Loose coupling Separate retrieval runs Merge ranked lists [Voorhees 95] Tight coupling [Ng 00] Unified indexing of words and subwords Single ranked list

Robust Retrieval: 

Robust Retrieval Multiple causes Speech recognition errors Translation ambiguity Transliteration ambiguity Possible solutions Weighted n-best indexing [Levow & Oard 00] Syllable lattice indexing [Chen, Wang & Lee 00] Syllable confusion expansion [Meng et al. 99] Structured queries [Pirkola 98] Document expansion [Levow & Oard 00]

System Architecture Overview: 

System Architecture Overview Word Translation English Example Phonetic Transcription Known Terms Retrieval System Words Syllable n-grams Mandarin Documents Translation Lexicon Corpus Statistics Eval Code Average Precision Relevance Judgments Syllable n-gram Generation

The TDT Collections: 

The TDT Collections 41 Hours VOA Mandarin Audio 59 Topics 121 Hours Voice of America (VOA) Mandarin Audio APW+NYT English Associated Press (APW) New York Times (NYT) English Newswire 20 Topics Development Test (TDT-2) Evaluation (TDT-3) Mar 98 Oct 98 Dec 98 Jun 98 Jan 98 Four stories per topic in each language Each reporting on some aspect of one event Story Boundaries Known Condition

MEI Project Schedule: 

MEI Project Schedule Six Weeks at Hopkins: Dec Feb Jun Apr Aug Workshop 00 Planning Meeting First Team Planning Meeting Second Team Planning Meeting

Things We Need: 

Things We Need Ideas To sharpen our focus Connections To build a community of interest Resources To build on what others have done

For More Information: 

For More Information MEI Project http://www.glue.umd.edu/~meiweb Translingual Retrieval http://www.clis.umd.edu/dlrg/clir Speech Retrieval http://www.clis.umd.edu/dlrg/speech Hopkins Summer Workshop Series http://www.clsp.jhu.edu/workshops

Backup Slides: 

Backup Slides

Background: Chinese: 

Background: Chinese Many dialects (e.g., Mandarin and Cantonese) differences in phonetics, vocabularies, syntax… Syllable-based language ~400 base syllables, 4 lexical tones + light tone Syllable structure (CG)V(X) (CG): onset, optional, consonant+medial glide V: nuclear vowel X: coda, glide / alveolar nasal / velar nasal ~ 21 initials, 39 finals

Background: Chinese (cont): 

Background: Chinese (cont) Characters (written) -> syllables (spoken) Degenerate mapping /hang2/, /hang4/, /heng2/ or /xing2/ /fu4 shu4/ (LDC’s CALLHOME lexicon) Tokenization / Segmentation /zhe4 yi1 wan3 hui4 ru2 chang2 ju3 xing2/

Translating the English Example: 

Translating the English Example English Example Documents (words) Named entities and unknown words Words found in translation dictionary Translation Transliteration Mandarin Queries (words and syllable n-grams) Indexed Mandarin Audio (words and syllable n-grams) Information Retrieval Engine Ranked List

Detailed Query Processing (1): 

White-space separated text with named entity tags. Stopping Stemming Phrase Extraction List of translatable words and phrases Terms that have Mandarin translations Named entities terms with no Mandarin translations Detailed Query Processing (1)

Detailed Query Processing (2): 

term translation English-Mandarin translation lexicon Named entity parsing Named entity parsing rules for transliteration phonetic expansion English pronunciation lexicon term translation phonetic expansion xxxxx xx xxx xxx zzzzzzz ww xx ww z rrrrrr sss ttttt tt ww www eh n t ih t iy t eh k s t Bag of Mandarin terms Bag of English phone sequences Terms that have Mandarin translations Terms with no Mandarin translations Named entities Northern Ireland Detailed Query Processing (2)

Detailed Query Processing (3): 

English phone strings to Mandarin syllables Mandarin syllabification rules ASR Insertion prone words ASR substitution/deletion prone words Retain this term? sa sb sc sd sc se s1 s2 s2 s5 s1 s2 s3 Mandarin pronunciation lexicon Bag of Mandarin terms Syllabic expansion yes xxxxx xxx zzzzzzz xx ww z rrrrrr sss ttttt tt ww www Bag of English phone sequences Two bags of Mandarin syllable sequences Smaller bag of Mandarin terms To trash (or downweight) no Detailed Query Processing (3)

Detailed Query Processing (4): 

Syllable n-gram generation Syllable n-gram generation s0 s1 s1 s2 s2 s3 s3 s4 s3 s5 s5 s6 s5 s7 s7 s7 sa sb sb sc sc sd sc se s1 s2 s2 s3 s2 s5 s5 s1 Syllable n-gram generation xxxxx xxx zzzzzzz xx ww z rrrrrr sss ttttt tt ww www Mandarin pronunciation lexicon syllabic expansion Mandarin syllable sequences from likely recognition errors Mandarin syllable sequences from unknown words Bag of Mandarin lexical terms Three Bags of Mandarin syllable n-grams from different sources Bag of high-confidence Mandarin terms Detailed Query Processing (4)

Translating the Mandarin Audio: 

Translating the Mandarin Audio English Example Documents (words) Indexed Mandarin Audio (words) Translation Information Retrieval Engine Ranked List Documents in English