Lecture3

Introduction to Biometrics: 

Introduction to Biometrics Dr. Bhavani Thuraisingham The University of Texas at Dallas Lecture #3 Information Management and Data Mining August 29, 2005

Objective of the Unit: 

Objective of the Unit This unit gives an overview of various information management technologies. In addition some details of data mining will also be given.

Outline of the Unit: 

Outline of the Unit What is Information Management? Some Information Management Technologies Information management Applications Data Mining

Revisiting the DM/IM/KM Framework: 

Revisiting the DM/IM/KM Framework Information Management Database Systems Information Retrieval Data Warehouse Security Distributed and Heterogeneous Database Security Object Database Security Privacy Secure Knowledge Digital Forensics Web Sensor Database Security Relational Database Security Inference Problem Secure Information Management Technologies Data Mining And Security Database Security Dependable Information Management Knowledge Management Information and Computer Security Biometrics Secure Digital Libraries Information Management Database Systems Information Retrieval Data Warehouse Security Distributed/ Federated Data Security Object/Multimedia Security Privacy Secure Knowledge Digital Forensics Web Sensor Database Security Relational Database Security Inference Problem Secure Information Management Technologies Data Mining And Security Database Security Dependable Information Management Knowledge Management Information and Computer Security Biometrics Secure Semantic Web Information Management Database Systems Information Retrieval Data Warehouse Security Distributed and Heterogeneous Database Security Object Database Security Privacy Secure Knowledge Digital Forensics Web Sensor Database Security Relational Database Security Inference Problem Secure Information Management Technologies Data Mining And Security Database Security Dependable Information Management Knowledge Management Information and Computer Security Biometrics Secure Digital Libraries Information Management Database Systems Information Retrieval Data Warehouse Security Distributed/ Federated Data Security Object/Multimedia Security Privacy Secure Knowledge Digital Forensics Web Sensor Database Security Relational Database Security Inference Problem Secure Information Management Technologies Data Mining And Security Database Security Dependable Information Management Knowledge Management Information and Computer Security Biometrics Secure Semantic Web Information Management Database Systems Information Retrieval Data Warehouse Security Distributed and Heterogeneous Database Security Object Database Security Privacy Secure Knowledge Digital Forensics Web Sensor Database Security Relational Database Security Inference Problem Secure Information Management Technologies Data Mining And Security Database Security Dependable Information Management Knowledge Management Information and Computer Security Biometrics Secure Digital Libraries Data Management Technologies Heterogeneous Database Management Relational Database Systems Secure Database Systems Data Warehouse Systems Peer-to-Peer Information Management Multimedia Information System Knowledge Portals Knowledge Creation and Acquisition Knowledge Mining Web Sensor Information Management Data Mining Semantic Web Expert systems and Reasoning under uncertainty Knowledge Sharing Information Management Knowledge Manipulation Distributed Databases Object Database Knowledge Models Knowledge Representation Knowledge Management Technologies Information Management Technologies Each layer builds on the technologies of the lower layers

What is Information Management?: 

What is Information Management? Information management essentially analyzes the data and makes sense out of the data Several technologies have to work together for effective information management Data Warehousing: Extracting relevant data and putting this data into a repository for analysis Data Mining: Extracting information from the data previously unknown Multimedia: managing different media including text, images, video and audio Web: managing the databases and libraries on the web

Data Warehouse: 

Data Warehouse Oracle DBMS for Employees Sybase DBMS for Projects Informix DBMS for Medical Data Warehouse: Data correlating Employees With Medical Benefits and Projects Could be any DBMS; Usually based on the relational data model Users Query the Warehouse

Data Mining: 

Data Mining

Multimedia Information Management: 

Multimedia Information Management Video Source Scene Change Detection Speaker Change Detection Silence Detection Commercial Detection Key Frame Selection Story Segmentation Named Entity Tagging Broadcast News Editor (BNE) Broadcast News Navigator (BNN) Multimedia Database Management System Web-based Search/Browse by Program, Person, Location, ... Imagery Audio Closed Caption Text Segregate Video Streams Analyze and Store Video and Metadata Story GIST Theme Frame Classifier Closed Caption Preprocess Correlation Token Detection Broadcast Detection

Semantic Web: 

Semantic Web Some Challenges: Security and Privacy cut across all layers; Integration of Services; Composability Adapted from Tim Berners Lee’s description of the Semantic Web

Semantic Web Technologies: 

Semantic Web Technologies Web Database/Information Management Information retrieval and Digital Libraries XML, RDF and Ontologies Representation information Information Interoperability Integrating heterogeneous data and information sources Intelligent agents Agents for locating resources, managing resources, querying resources and understanding web pages Semantic Grids Integrating semantic web with grid computing technologies

Secure Data Sharing Across Coalitions: 

Secure Data Sharing Across Coalitions Export Data/Policy Component Data/Policy for Agency A Data/Policy for Coalition Export Data/Policy Component Data/Policy for Agency C Component Data/Policy for Agency B Export Data/Policy

Some Emerging Information Management Technologies: 

Some Emerging Information Management Technologies Visualization Visualization tools enable the user to better understand the information Peer-to-Peer Information Management Peers communicate with each other, share resources and carry out tasks Sensor and Wireless Information Management Autonomous sensors cooperating with one another, gathering data, fusing data and analyzing the data Integrating wireless technologies with semantic web technologies

Information Management for Applications: Examples: 

Information Management for Applications: Examples Decision Support E-Commerce Collaboration Training Knowledge Management Virtual Organizations and Dynamic Coalitions

Outline of Data Mining: 

Outline of Data Mining What is Data Mining Steps to Data Mining Need for Data Mining Example Applications Technologies for Data Mining Why Data Mining Now? Preparation for Data Mining Data Mining Tasks, Methodology, Techniques Commercial Developments Status, Challenges , and Directions Example Data Mining Technique

Data Mining: 

Data Mining

Steps to Data Mining: 

Steps to Data Mining Data Sources Integrate data sources Clean/ modify data sources Mine the data Examine Results/ Prune results Report final results

Need for Data Mining: 

Large amounts of current and historical data being stored As databases grow larger, decision-making from the data is not possible; need knowledge derived from the stored data Data for multiple data sources and multiple domains Medical, Financial, Military, etc. Need to analyze the data Support for planning (historical supply and demand trends) Yield management (scanning airline seat reservation data to maximize yield per seat) System performance (detect abnormal behavior in a system) Mature database analysis (clean up the data sources) Need for Data Mining

Example Applications: 

Example Applications Medical supplies company increases sales by targeting certain physicians in its advertising who are likely to buy the products A credit bureau limits losses by selecting candidates who are likely not to default on their payment An Intelligence agency determines abnormal behavior of its employees An investigation agency finds fraudulent behavior of some people

Integration of Multiple Technologies: 

Integration of Multiple Technologies Machine Learning Database Management Artificial Intelligence Statistics Data Mining Visualization Parallel Processing

Why Data Mining Now?: 

Why Data Mining Now? Large amounts of data is being produced Data is being organized Technologies are developing for database management, data warehousing, parallel processing, machine intelligent, etc. It is now possible to mine the data and get patterns and trends Interesting applications exist

Preparation for Data Mining: 

Preparation for Data Mining Getting the data into the right format Data warehousing Scrubbing and cleaning the data Some idea of application domain Determining the types of outcomes e.g., Clustering, classification Evaluation of tools Getting the staff trained in data mining

Some Types of Data Mining (Data Mining Tasks): 

Some Types of Data Mining (Data Mining Tasks) Classification – grouping records into meaningful subclasses e.g., Marketing organization has a list of people living in Manhattan all owning cars costing over 20K Sequence Detection John always buys groceries after going to the bank Data dependency analysis – identifying potentially interesting dependencies or relationships among data items If John, James, and Jane meet, Bill is also present Deviation detection – discovery of significant differences between an observation and some reference Anomalous instances Discrepancies between observed and expected values

Data Mining Methodology (or Approach): 

Data Mining Methodology (or Approach) Top-down Hypothesis testing Validate beliefs Bottom-up Discover patterns Directed Some idea what you want to get Undirected Start from fresh

Some Data Mining Techniques: 

Some Data Mining Techniques Market Basket analysis Decision Trees Neural networks Link Analysis Genetic Algorithms Automatic Cluster Detection Inductive logic programming

Commercial Developments in Data Mining: Some Products: 

Commercial Developments in Data Mining: Some Products WizSoft - WhizWhy Hugin - Hugin IBM - Intelligent Miner Red Brick - DataMind Neo Vista - Decision Series Reduct Systems - Datalogic/R IDIS - Information Discovery Lockheed Martin - Recon Nicesoft – Nicel SAS – Enterprise Miner

Current Status, Challenges and Directions: 

Current Status, Challenges and Directions Status Data Mining is now a technology Several prototypes and tools exist; Many or almost all of them work on relational databases Challenges Mining large quantities of data; Dealing with noise and uncertainty, reasoning with incomplete data Directions Mining multimedia and text databases, Web mining (structure, usage and content), Mining metadata, Real-time data mining

Example Data Mining Technique:What is Market Basket Analysis?: 

Example Data Mining Technique: What is Market Basket Analysis? Market basket analysis is a collection of techniques that will discover rules such as what items are purchased together It has roots in point of sale transactions; but has gone beyond this applications E.g., who travels together, who is seen with whom, etc. Market basket analysis is used as a starting point when transactions data is available and we are not sure of the patterns we are looking for Find items that are purchased together Essentially market basket analysis produces association rules

Example: 

Example Person Countries Visited John England, France James Germany, England, Switzerland William England, Austria Mary England, Austria, France Jane Switzerland, France Co-Occurrence Table England Switzerland Germany France Austria England 4 1 1 2 2 Switzerland 1 2 1 1 0 Germany 1 1 1 0 0 France 2 1 0 3 1 Austria 2 0 0 1 2

Example (Concluded): 

Example (Concluded) England and France / England and Austria are more likely to be traveled together than any other two countries Austria is never traveled together with Germany or Switzerland Germany is never traveled together with Austria or France Rule: If a person travels to France then he/she also travels to England Support for this rule is 2 out of 5 and that is 40% since 2 trips out of five support this rule Confidence for this rule is 66% since two out of three trips that contain France also contains England That is, if France then England rule has support 40% and confidence 66% Challenge: How to automatically generate the rules

Basic Process: 

Basic Process Choosing the right set of items Need to gather the right set of transaction data and the right level of detail, ensuring data quality Generating rules from the data Generate co-occurrence matrix for single items Generate co-occurrence matrix with 2 items and use this to find rules with 2 items Generate co-occurrence matrix with 3 items and use this to find rules with 3 items; etc. - - - Overcoming practical limits imposed by thousand of items Avoid combinatorial explosions

Association Rules: 

Association Rules Rules that find associations in data Example of a association rule is (x1, x2, x3}  x4 meaning that if x1, x2, and x3 are purchased x4 is also purchased Association rules have confidence values Strong rules are rules with confidence value above a threshold Challenge is to improve the algorithm E.g., Partition-based approach, sampling

Challenges and Directions: 

Challenges and Directions Performance improvements Applying techniques for web mining including web content mining, web structure mining and web usage mining Finding associations in text Associations between words in a document or multiple documents