naacl10 v5

Unsupervised Modeling of Twitter Conversations : 

Unsupervised Modeling of Twitter Conversations Alan Ritter (UW) Colin Cherry (NRC) Bill Dolan (MSR)

Twitter : 

Twitter Most of Twitter looks like this: I want a cuban sandwich extra bad!! About 10-20% are replies I 'm going to the beach this weekend! Woo! And I'll be there until Tuesday. Life is good. Enjoy the beach! Hope you have great weather! thank you 

Gathering Conversations : 

Gathering Conversations Twitter Public API Public Timeline 20 randomly selected posts per minute Use to get random sample of twitter users Query to get all their posts Follow any that are replies to collect conversations No need for disentanglement [Elsner & Charniak 2008]

Conversation Length(number of Tweets) : 

Conversation Length(number of Tweets)

Modeling Latent Structure:Dialogue Acts : 

Modeling Latent Structure:Dialogue Acts I 'm going to the beach this weekend! Woo! And I'll be there until Tuesday. Life is good. Enjoy the beach! Hope you have great weather! thank you  Status Comment Thanks

Dialogue Acts:Many Useful Applications : 

Dialogue Acts:Many Useful Applications Conversation Agents [Wilks 2006] Dialogue Systems [Allen et al. 2007] Dialogue Summarization [Murray et al. 2006] Flirtation Detection [Ranganath et al. 2009] …

Traditional Approaches : 

Traditional Approaches Gather Corpus of Conversations focus on speech data Telephone Conversations - [Jurafsky et. al. 1997] Meetings - [Dhillon et. al. 2004] [Carletta et. al. 2006] Annotation Guidelines Manual Labeling Expensive

Dialogue Acts for Internet Conversations : 

Dialogue Acts for Internet Conversations Lots of Variety Email [Cohen et. al. 2004] Internet Forums [Jeong et. al. 2009] IRC Facebook Twitter … More on the horizon? Tags from speech data not always appropriate Includes: Backchannel, distruption, floorgrabber Missing: Meeting request, Status post, etc…

Our Contributions : 

Our Contributions Unsupervised Tagging of Dialogue Acts First application to open domain Dialogue Modeling on Twitter Potential for new language technology applications Lots of data available Release a dataset of Twitter conversations Collected over several months http://research.microsoft.com/en-us/downloads/8f8d5323-0732-4ba0-8c6d-a5304967cc3f/default.aspx

Modeling Latent Structure:Dialogue Acts : 

I 'm going to the beach this weekend! Woo! And I'll be there until Tuesday. Life is good. Enjoy the beach! Hope you have great weather! thank you  Status Comment Thanks Modeling Latent Structure:Dialogue Acts

Discourse constraints : 

Discourse constraints I 'm going to the beach this weekend! Woo! And I'll be there until Tuesday. Life is good. Enjoy the beach! Hope you have great weather! thank you  Status Comment Thanks

Words indicate dialogue act : 

Words indicate dialogue act I 'm going to the beach this weekend! Woo! And I'll be there until Tuesday. Life is good. Enjoy the beach! Hope you have great weather! thank you  Status Comment Thanks

Conversation Specific Topic Words : 

Conversation Specific Topic Words I 'm going to the beach this weekend! Woo! And I'll be there until Tuesday. Life is good. Enjoy the beach! Hope you have great weather! thank you  Status Comment Thanks

Content Modeling [Barzilay & Lee 2004] : 

Content Modeling [Barzilay & Lee 2004] Summarization Model order of events in news articles Very specific topics: e.g. Earthquakes Model Sentence Level HMM states emit whole sentences Learn parameters with EM Where, when Rictor scale Damage

Adapt CM to Unsupervised DA Tagging : 

Adapt CM to Unsupervised DA Tagging

Content Models:Pros/Cons : 

Content Models:Pros/Cons Pros: Models Transitions Between Dialogue Acts Unsupervised Cons: Topic Clusters Barzilay & Lee masked out named entities to focus on general (not specific) events This is more difficult on Twitter

Problem: Strong Topic Clusters : 

Problem: Strong Topic Clusters We want: Not:

Goal: separate content/dialogue words : 

Goal: separate content/dialogue words Dialogue act words Conversation specific words LDA-style topic model Each word is generated from 1 of 3 sources: General English Conversation Topic Specific Vocabulary Dialogue Act specific Vocabulary Similar to: [Daume III and Marcu, 2006] [Haghighi & Vanderwende 2009]

Conversation Model : 

Conversation Model

Conversation+Topic Model : 

Conversation+Topic Model

Inference : 

Inference Collapsed Gibbs sampling Sample each hidden variable conditioned on assignment of all others Integrate out parameters But, lots of hyperparameters to set Act transition multinomial Act emission multinomial Doc-specific multinomal English multinomial Source distribution multinomial Slice Sampling Hyperparameters [Neal 2003]

Probability Estimation : 

Probability Estimation Problem: Need to evaluate the probability of a conversation Integrate out hidden variables Use as a language model Chibb-style Estimator [Wallach et. al 2009] [Murray & Salakhutdinov 2009]

Chibb Style Estimator [Wallach et. al 2009] [Murray & Salakhutdinov 2009] : 

Chibb Style Estimator [Wallach et. al 2009] [Murray & Salakhutdinov 2009] Just Need to estimate:

Qualitative Evaluation : 

Qualitative Evaluation Trained on 10,000 Twitter conversations of length 3 to 6 tweets

Conversation+Topic modelDialogue act transitions : 

Conversation+Topic modelDialogue act transitions

Slide 26: 

Status

Slide 27: 

Question

Slide 28: 

Question to Followers

Slide 29: 

Reference Broadcast

Slide 30: 

Reaction

Evaluation : 

Evaluation How do we know which works best? How well can we predict sentence order? Generate all permutations of a conversation Compute probability of each How similar is highest ranked to original order? Measure permutation similarity with Kendall Tau Counts number of swaps needed to get desired order

Experiments – Conversation Ordering : 

Experiments – Conversation Ordering

Experiments – Conversation Ordering : 

Experiments – Conversation Ordering

Experiments – Conversation Ordering : 

Experiments – Conversation Ordering Content words help predict sentence order -Adjacent sentences contain similar content words

Conclusions : 

Conclusions Presented a corpus of Twitter Conversations http://research.microsoft.com/en-us/downloads/8f8d5323-0732-4ba0-8c6d-a5304967cc3f/default.aspx Conversations on Twitter seem to have some common structure Strong topic clusters are a problem for open-domain unsupervised DA tagging Presented an approach to address this Gibbs Sampling/Full Bayesian inference seems to outperform EM on Conversation Ordering