2006 11 08 slides

Opinion Retrieval forEnglish Document Collections: 

Opinion Retrieval for English Document Collections Yejun Wu & Doug Oard College of Information Studies & UMIACS University of Maryland, College Park UMIACS CLIP Internal Talk November 8, 2006

Outline: 

Outline TREC 2006 Blog Pilot Track Dissertation proposal (omitted) 2

TREC 2006 Blog Track: 

TREC 2006 Blog Track Opinion Retrieval Task: -- locating blog posts that express an opinion about a given target (posting + comments) -- target: NE, product name, concept, event. -- NIST judges, 50 topics Open task: other aspects of blogs for 2007 -- temporal; gender/location classification … 3

Blog Collection: 

Blog Collection U. Glasgow: 12/2005 - 2/2006 Size: 23GB compressed -- RSS feeds (8GB) -- permalinks (11GB)  retrieval units -- homepages (4GB) Distributed in March, received in May Ran experiments in June and July. 4

Slide5: 

www.liveJournal.com BLOG06-20051206-028-0017251333 Example Permalink doc #Home Site Map Technical Support Previous Next valerie ( [info] xnerveagentx) wrote, @ 2005-09-23 08:34:00 Previous Entry Add to memories! Next Entry the last public entry. "It's okay to hate him if it makes things easier for you. Don't worry about hurting him, because was he really worried about hurting you? He abandoned you after he promised you he wouldn't. You are doing the best you can, and if that's not good enough for someone, then you have to ask yourself if he's really worth it.“ "You have to decide how long you're going to let this affect you. How long are you going to keep punishing yourself for something that's not entirely your fault?" - my mom Some days I feel okay, but some days I feel hopeless. Yesterday was obviously the latter. I try not to think about it too much, but that's hard. I can't comprehend why you'd completely abandon someone you claim to love. He's not here for me anymore at all. …… valerie (Post a new comment) [dot.gif] [info] vagynafondue 2005-09-23 01:41 pm UTC (link) Your mom is a smart woman! You're lucky to have her in your life. My mom was never any help with stuff like that. ……I'm so sorry you're going through all this:( (Reply to this) ____________________________________________ LiveJournal Log in now. (Create account, or use OpenID) Username: _______________ Password: __________ Login? forget your login? [_] remember me Search: _______________ Category: [Site & Username] Navigate: * Welcome + Login + Create an Account * Search 5

Slide6: 

BLOG06-20051224-029-0001622821.cln Notify Blogger about objectionable content. What does this mean? Blogger Get your own blog Flag Blog Next blog rabbit + crow blog It's like anything. Tuesday, December 20, 2005 Blue Planet "...Whoa!...Wow!...WOW!...Holy shit!...WOW!!!..." (my wife and I watching the first episode of Sir David Attenborough's "The Blue Planet" tonight) posted by Neal Romanek at 9:18 PM - permalink 0 Comments: Comment? About Me My Photo Name:Neal Romanek Location:Los Angeles View my complete profile NealRomanek.com The Shop Other Writing Bio Photos Contact The Previous Posts * I like Bush (example) * Smoking is harmless (example) Archives SUBSCRIBE to the Rabbit + Crow Blog! WEEKLEY POALE In our last Weeklie Poll, we asked which was your favorite Ceratopsian. The winner, amid spiky competition, was, of course... ...STYRACOSAURUS. THIS WEEK... If you found you could no longer walk, which mode of ambulation would you instead adopt? (_) Bustling (_) Charging (_) Creeping (_) Tip-Toeing (_) All of the above in various combinations (_) None of the above. I would adopt a stony stillness. buy the new shirt questions? suggestions? confessions? ask@nealromanek.com Listed on BlogShares blog search directory Blogarama - The Blog Directory advanced web statistics

Cleaning Permalink Docs: 

Cleaning Permalink Docs Layout w/ sites Top 5 sites: livejournal.com spaces.com xanga.com blogs.msdn.com 6-allthe.info 10% permalinks Check 100 docs Livejounal: Remove: -- Beneath “Log in now.” - B/t “Page Summary” and date pattern “Nov 24th, 2005” - B/t “Previous Posts:” and * -B/t “Blogs I read:” and * -B/t “Documents I liked:” and * - short line w/ *, _+, # o, @ …… 6

Ideas to Test: 

Ideas to Test Paragraphs vs. fixed sized passages Demoting non-opinionated paragraphs Query formulation using title vs. title + desc. 8

Slide9: 

Permalink Docs “Cleaned” Docs Passages Paragraphs Top 1000 Pars Ranked List Par0001 Par0002 Par0003 Par0004 Par0005 Indri Index Indri Index cleaning query Compute SO of words … Wilson & Wiebe’s lexicon Demotion by 2 or 3 times If <0.15 normalized 0.21 0.12 0.44 0.56 0.32 1 2 3 4 5 16773 lemmas (-3 ~ -0.05) negative, (0.05 ~ 5) positive 8221 lemmas 0.2 1.0 lemmatized Method 9 Topic Relevance Assessment Opinion Relevance Assessment Par/Pas aggregated back to docs for evaluation

Segment into Paragraphs & Passages: 

Segment into Paragraphs & Passages Paragraph: (1) Identify candidate paragraphs by merging lines. ……! . ? “ ) …… - > * + o nums … (2) Merge a current candidate paragraph into previous if -- Merged paragraph < 50 tokens. -- Current par >= 50, previous < 30 … …… Passage: Window = 50 words, Overlap = 10 words. TextTiling? Sub-topic passages. What if topic & attitude target do not match? 10 (a bullet) title

Slide11: 

the online home of Sara Zarr Wednesday, December 21, 2005 "if you think this is big, wait til you see the toast" I guess it's safe to tell the secret now, since there is no chance G-man will read my blog while he is flying home from a little trip. Alan and I went tree shopping after work on Monday (I'd been watching the prices, and Smith's had slashed trees to $14.98 for a 6-7 footer) and loaded two giant trees into his pickup (everyone should have a friend with a truck). We got somewhat filthy as it had rained and snowed and the trees were nice and fresh and sappy. After wrastling the tree through the front door, we got it on the stand and in place and damn but it takes up nearly the whole living room! It smells unbelievable. Anyhoo, last night I pulled out the decorations and bedecked the tree and the house while watching Uncle Buck (a movie which must be watched once every 28-36 months). Normally, the tree stuff is something I'd want to do with G., but when he left on Sunday our household was in high stress and chaos; I think it will be nice for him to come home to Christmas in progress without actually having to do any of the work himself. I wanted to leave on the tree lights and have Christmas music playing to enhance his experience of coming home, but I got paranoid about setting the house on fire and I couldn't find any of the Christmas CDs, so the sight and smell of the tree by day will have to be enough. BLOG06-20051224-060-0007909376.cln Segmenting into paragraphs/passages. 1.Merge lines by checking punctuations. 2. Merge candidate paragraphs 11

Slide12: 

the stories of a girl the online home of Sara Zarr Tuesday, December 20, 2005 Today is a day I want to stay in bed with books and tea. No phone, no e-mail, no TV, no outside world. Only reading and napping and dreaming. Maybe next week. Can you keep a secret? Probably not. So never mind. I've yet to begin my Christmas shopping or cards. Somehow I don't think cards are going to happen this year, so don't take it personally if your mailbox goes Sara-less. Yeah, I know that's the same song and dance I give every year. posted by Sara Z. @ 1:17 PM 2 comments 2 Comments: At 9:15 PM, Danny said... "Today is a day I want to stay in bed with books and tea." Hey, just about every day I wish I could mix records in bed. Actually, right now I would just like to mix more records (and then get money afterwards.) :) At 1:36 PM, slc urban princess said... um, i can keep a secret but dang good. BLOG06-20051224-060-0007929048.cln Segmenting into paragraphs/passages At time sb said 12

Segment into Paragraphs & Passages: 

Segment into Paragraphs & Passages Paragraph: (1) Identify candidate paragraphs by merging lines. ……! . ? “ ) …… - > * + o nums … (2) Merge a current candidate paragraph into previous if -- Merged paragraph < 50 tokens. -- Current par >= 50, previous < 30 … …… Passage: Window = 50 words, Overlap = 10 words. TextTiling? Sub-topic passages. Topic & attitude target do not match? 13 (a bullet) title

Slide14: 

Ranked List lemmas Indri Paragraph index Morpha PMIs Remove: stop words (lemmas) Not in dictionary by “spell” Wilson & Wiebe’s lexicon 16773 lemmas SO of lemmas By title query Compute SO of Words 14 Turney & Littman (Altavista) SO-PMI(w) = PMI(w, {positive paradigms}) – PMI(w, {negative paradigms}) {good, nice, excellent, positive, fortunate, correct, superior} {bad, nasty, poor, negative, unfortunate, incorrect, inferior} Strength of association N: total num of docs indexed DF<=40

Slide15: 

Positivity Not judged:-1 Irrelevant: 0 Relevant, Not opinionated: 1 Low High Low High Negativity Neutral X Relevant, Mixed: 3 NIST Relevance Assessment 15 Relevant, Positive: 4 Relevant, Negative: 2 • 6 levels Opinion relevance: topic not have to be the target, but an opinion about the target present Our results at two levels: -- topic relevant level (>=1 vs. <1); -- opinion relevant level (>=2 vs. <2)

Slide16: 

ParTitDesDef vs. ParTiDesDmt2: not sig. at 0.05 vs. ParTiDesDef3: marginally sig. p=0.05 vs. ParTiDef (PasTiDesDef): sig. p<0.05 Wilcoxon signed-rank test for paired samples on MAP ParTitDesDef vs. ParTiDesDmt2 (or ParTiDesDef3): not sig. at 0.05 vs. ParTiDef (or PasTiDesDef): sig. at p<0.05 16

Slide17: 

Opinion Relevance. Median of 57 runs; outperform 39 out of 50 topics Topic by topic effect of proportional demotion by 2 times

Conclusion: 

Conclusion Paragraph > Passage (topic & opinion relevance) Query formulation with title + desc. > title Demotion: neither help nor harm; good/bad for some topics 18

Future Work: 

Future Work Currently: Select words w/ DF<=40 for computing SO. Paragraph detection flawed; passage size small. Threshold: normalized attitude <0.15 (intensity?) Computation of SO of words (large corpus) Aggregation of attitude (1 point vs. 0.2 point) Future: Tuning 19