logging in or signing up TextMining 06 Barbara Download Post to : URL : Related Presentations : Share Add to Flag Embed Email Send to Blogs and Networks Add to Channel Uploaded from authorPOINTLite Insert YouTube videos in PowerPont slides with aS Desktop Copy embed code: (To copy code, click on the text box) Embed: URL: Thumbnail: WordPress Embed Customize Embed The presentation is successfully added In Your Favorites. Views: 267 Category: News & Reports.. License: All Rights Reserved Like it (0) Dislike it (0) Added: October 03, 2007 This Presentation is Public Favorites: 0 Presentation Description No description available. Comments Posting comment... By: manawyal (21 month(s) ago) can i plz get this ppt "manawya.haque@yahoo.com" Saving..... Post Reply Close Saving..... Edit Comment Close Premium member Presentation Transcript Text Mining: Finding Nuggets in Mountains of Textual Data: Text Mining: Finding Nuggets in Mountains of Textual Data Jochen Dijrre, Peter Gerstl, Roland Seiffert Presented by Drew DeHaasOutline: Outline Motivation Methodology Feature Extraction Clustering and Categorizing Some Applications Conclusion & Exam QuestionsMotivation: Motivation A large portion of a company’s data is unstructured or semi-structured Letters Emails Phone recordings Contracts Technical documents Patents Web pages ArticlesMotivation: Motivation Rapid processing of large document collections Speed! Automation of tasks Objective analysisTypical Applications: Typical Applications Summarizing documents Discovering/monitoring relations among people, places, organizations, etc Organizing documents by content Indexing for search and retrieval Retrieving documents by contentOutline: Outline Motivation Methodology Feature Extraction Clustering and Categorizing Some Applications Conclusion & Exam QuestionsMethodology: Challenges: Methodology: Challenges Information is in unstructured textual form Natural language interpretation is difficult & complex task! (not fully possible) Text mining deals with huge collections of documents Methodology: Two Aspects: Methodology: Two Aspects Knowledge Discovery Extraction of codified information Mining proper; determining some structure Information Distillation Analysis of feature distribution Two Text Mining Approaches: Two Text Mining Approaches Extraction Extraction of codified information from single document Analysis Analysis of the features to detect patterns, trends, etc, over whole collections of documentsComparison with Data Mining: Comparison with Data Mining Data mining Identify data set(s) Select features manually Prepare data Analyze distribution Text mining Identify documents Extract features Select features (automatically) Prepare data Analyze distributionIBM Intelligent Miner for Text: IBM Intelligent Miner for Text IBM introduced product in 1998 SDK with: Feature extraction, clustering, categorization, and more Traditional components (search engine, etc) No longer available? The rest of the paper describes text mining methodology of Intelligent Miner.Outline: Outline Motivation Methodology Feature Extraction Clustering and Categorizing Some Applications Conclusion & Exam QuestionsFeature Extraction: Feature Extraction Recognize and classify “significant” vocabulary items from the text Categories of vocabulary Proper names Multiword terms Abbreviations Relations Other useful things: numerical forms of numbers, percentages, money, etcCanonical Form Examples: Canonical Form Examples Normalize numbers, money Four = 4, five-hundred dollar = $500 Conversion of date to normal form Morphological variants Drive, drove, driven = drive Proper names and other forms Mr. Johnson, Bob Johnson, The author = Bob Johnson Feature Extraction Approach: Feature Extraction Approach Linguistically motivated heuristics Pattern matching Limited lexical information (part-of-speech) Avoid analyzing with too much depth Does not use too much lexical information No in-depth syntactic or semantic analysisFeature Extraction Example: Feature Extraction Example Disambiguating Proper Names (Nominator Program) Apply heuristics to strings, instead of interpreting semantics The unit of context for extraction is a document. The heuristics represent English naming conventionsAdvantages to IBM’s approach: Advantages to IBM’s approach Processing is very fast (helps when dealing with huge amounts of data) Heuristics work reasonably well Generally applicable to any domainOutline: Outline Motivation Methodology Feature Extraction Clustering and Categorizing Some Applications Conclusion & Exam QuestionsClustering: Clustering Fully automatic process Documents are grouped according to similarity of their feature vectors Each cluster is labeled by a listing of the common terms/keywords Good for getting an overview of a document collectionTwo Clustering Engines: Two Clustering Engines Hierarchical clustering Orders the clusters into a tree reflecting various levels of similarity Binary relational clustering Flat clustering Relationships of different strengths between clusters, reflecting similarityClustering Model: Clustering ModelCategorization: Categorization Assigns documents to preexisting categories Classes of documents are defined by providing a set of sample documents. Training phase produces “categorization schema” Documents can be assigned to more than one category If confidence is low, document is set aside for human interventionCategorization Model: Categorization ModelOutline: Outline Motivation Methodology Feature Extraction Clustering and Categorizing Some Applications Conclusion & Exam QuestionsApplications: Applications Customer Relationship Management application provided by IBM Intelligent Miner for Text called “Customer Relationship Intelligence” “Help companies better understand what their customers want and what they think about the company itself”Customer Intelligence Process: Customer Intelligence Process Take as input body of communications with customer Cluster the documents to identify issues Characterize the clusters to identify the conditions for problems Assign new messages to appropriate clustersCustomer Intelligence Usage: Customer Intelligence Usage Knowledge Discovery Clustering used to create a structure that can be interpreted Information Distillation Refinement and extension of clustering results Interpreting the results Tuning of the clustering process Selecting meaningful clustersOutline: Outline Motivation Methodology Feature Extraction Clustering and Categorizing Some Applications Conclusion & Exam QuestionsConclusion: Conclusion This paper introduced text mining and how it differs from data mining proper. Focused on the tasks of feature extraction and clustering/categorization Presented an overview of the tools/methods of IBM’s Intelligent Miner for Text Exam Question #1: Exam Question #1 Name an example of each of the two main classes of applications of text mining. Knowledge Discovery: Discovering a common customer complaint in a large collection of documents containing customer feedback. Information Distillation: Filtering future comments into pre-defined categoriesExam Question #2: Exam Question #2 How does the procedure for text mining differ from the procedure for data mining? Adds feature extraction phase Infeasible for humans to select features manually The feature vectors are, in general, highly dimensional and sparseExam Question #3: Exam Question #3 In the Nominator program of IBM’s Intelligent Miner for Text, an objective of the design is to enable rapid extraction of names from large amounts of text. How does this decision affect the ability of the program to interpret the semantics of text? Does not perform in-depth syntactic or semantic analysis of the text; the results are fast but only heuristic with regards to actual semantics of the text.Questions?: Questions? You do not have the permission to view this presentation. In order to view it, please contact the author of the presentation.
TextMining 06 Barbara Download Post to : URL : Related Presentations : Share Add to Flag Embed Email Send to Blogs and Networks Add to Channel Uploaded from authorPOINTLite Insert YouTube videos in PowerPont slides with aS Desktop Copy embed code: (To copy code, click on the text box) Embed: URL: Thumbnail: WordPress Embed Customize Embed The presentation is successfully added In Your Favorites. Views: 267 Category: News & Reports.. License: All Rights Reserved Like it (0) Dislike it (0) Added: October 03, 2007 This Presentation is Public Favorites: 0 Presentation Description No description available. Comments Posting comment... By: manawyal (21 month(s) ago) can i plz get this ppt "manawya.haque@yahoo.com" Saving..... Post Reply Close Saving..... Edit Comment Close Premium member Presentation Transcript Text Mining: Finding Nuggets in Mountains of Textual Data: Text Mining: Finding Nuggets in Mountains of Textual Data Jochen Dijrre, Peter Gerstl, Roland Seiffert Presented by Drew DeHaasOutline: Outline Motivation Methodology Feature Extraction Clustering and Categorizing Some Applications Conclusion & Exam QuestionsMotivation: Motivation A large portion of a company’s data is unstructured or semi-structured Letters Emails Phone recordings Contracts Technical documents Patents Web pages ArticlesMotivation: Motivation Rapid processing of large document collections Speed! Automation of tasks Objective analysisTypical Applications: Typical Applications Summarizing documents Discovering/monitoring relations among people, places, organizations, etc Organizing documents by content Indexing for search and retrieval Retrieving documents by contentOutline: Outline Motivation Methodology Feature Extraction Clustering and Categorizing Some Applications Conclusion & Exam QuestionsMethodology: Challenges: Methodology: Challenges Information is in unstructured textual form Natural language interpretation is difficult & complex task! (not fully possible) Text mining deals with huge collections of documents Methodology: Two Aspects: Methodology: Two Aspects Knowledge Discovery Extraction of codified information Mining proper; determining some structure Information Distillation Analysis of feature distribution Two Text Mining Approaches: Two Text Mining Approaches Extraction Extraction of codified information from single document Analysis Analysis of the features to detect patterns, trends, etc, over whole collections of documentsComparison with Data Mining: Comparison with Data Mining Data mining Identify data set(s) Select features manually Prepare data Analyze distribution Text mining Identify documents Extract features Select features (automatically) Prepare data Analyze distributionIBM Intelligent Miner for Text: IBM Intelligent Miner for Text IBM introduced product in 1998 SDK with: Feature extraction, clustering, categorization, and more Traditional components (search engine, etc) No longer available? The rest of the paper describes text mining methodology of Intelligent Miner.Outline: Outline Motivation Methodology Feature Extraction Clustering and Categorizing Some Applications Conclusion & Exam QuestionsFeature Extraction: Feature Extraction Recognize and classify “significant” vocabulary items from the text Categories of vocabulary Proper names Multiword terms Abbreviations Relations Other useful things: numerical forms of numbers, percentages, money, etcCanonical Form Examples: Canonical Form Examples Normalize numbers, money Four = 4, five-hundred dollar = $500 Conversion of date to normal form Morphological variants Drive, drove, driven = drive Proper names and other forms Mr. Johnson, Bob Johnson, The author = Bob Johnson Feature Extraction Approach: Feature Extraction Approach Linguistically motivated heuristics Pattern matching Limited lexical information (part-of-speech) Avoid analyzing with too much depth Does not use too much lexical information No in-depth syntactic or semantic analysisFeature Extraction Example: Feature Extraction Example Disambiguating Proper Names (Nominator Program) Apply heuristics to strings, instead of interpreting semantics The unit of context for extraction is a document. The heuristics represent English naming conventionsAdvantages to IBM’s approach: Advantages to IBM’s approach Processing is very fast (helps when dealing with huge amounts of data) Heuristics work reasonably well Generally applicable to any domainOutline: Outline Motivation Methodology Feature Extraction Clustering and Categorizing Some Applications Conclusion & Exam QuestionsClustering: Clustering Fully automatic process Documents are grouped according to similarity of their feature vectors Each cluster is labeled by a listing of the common terms/keywords Good for getting an overview of a document collectionTwo Clustering Engines: Two Clustering Engines Hierarchical clustering Orders the clusters into a tree reflecting various levels of similarity Binary relational clustering Flat clustering Relationships of different strengths between clusters, reflecting similarityClustering Model: Clustering ModelCategorization: Categorization Assigns documents to preexisting categories Classes of documents are defined by providing a set of sample documents. Training phase produces “categorization schema” Documents can be assigned to more than one category If confidence is low, document is set aside for human interventionCategorization Model: Categorization ModelOutline: Outline Motivation Methodology Feature Extraction Clustering and Categorizing Some Applications Conclusion & Exam QuestionsApplications: Applications Customer Relationship Management application provided by IBM Intelligent Miner for Text called “Customer Relationship Intelligence” “Help companies better understand what their customers want and what they think about the company itself”Customer Intelligence Process: Customer Intelligence Process Take as input body of communications with customer Cluster the documents to identify issues Characterize the clusters to identify the conditions for problems Assign new messages to appropriate clustersCustomer Intelligence Usage: Customer Intelligence Usage Knowledge Discovery Clustering used to create a structure that can be interpreted Information Distillation Refinement and extension of clustering results Interpreting the results Tuning of the clustering process Selecting meaningful clustersOutline: Outline Motivation Methodology Feature Extraction Clustering and Categorizing Some Applications Conclusion & Exam QuestionsConclusion: Conclusion This paper introduced text mining and how it differs from data mining proper. Focused on the tasks of feature extraction and clustering/categorization Presented an overview of the tools/methods of IBM’s Intelligent Miner for Text Exam Question #1: Exam Question #1 Name an example of each of the two main classes of applications of text mining. Knowledge Discovery: Discovering a common customer complaint in a large collection of documents containing customer feedback. Information Distillation: Filtering future comments into pre-defined categoriesExam Question #2: Exam Question #2 How does the procedure for text mining differ from the procedure for data mining? Adds feature extraction phase Infeasible for humans to select features manually The feature vectors are, in general, highly dimensional and sparseExam Question #3: Exam Question #3 In the Nominator program of IBM’s Intelligent Miner for Text, an objective of the design is to enable rapid extraction of names from large amounts of text. How does this decision affect the ability of the program to interpret the semantics of text? Does not perform in-depth syntactic or semantic analysis of the text; the results are fast but only heuristic with regards to actual semantics of the text.Questions?: Questions?