Multimedia Signal Processing & Content-Based Image Retrieval : Multimedia Signal Processing & Content-Based Image Retrieval Anastasios N. Venetsanopoulos
University of Toronto
Contact: anv@dsp.toronto.edu
http://www.dsp.toronto.edu
http://www.ece.toronto.edu
OUTLINE : OUTLINE INTRODUCTION
MULTIMEDIA APPLICATIONS
IMPACT OF MULTIMEDIA
CONTENT-BASED IMAGE RETRIEVAL
(CBIR)
MPEG-7
RESEARCH ISSUES
INTRODUCTION-1 : INTRODUCTION-1
WHAT IS MULTIMEDIA?
WHAT IS MULTIMEDIA PROCESSING?
GOALS OF MULTIMEDIA PROCESSING
INTRODUCTION-2 : INTRODUCTION-2 DIFFICULT TO DEFINE
GENERALLY CONSISTS OF:
MULTIMEDIA DATA
INTERACTION SET
MULTIMEDIA DATA:
MULTI-SOURCE, MULTI-TYPE, MULTI-FORMAT
INTERACTION SET:
WITHOUT INTERACTIONS BETWEEN MULTIMEDIA COMPONENTS, MULTIMEDIA IS MERELY A COLLECTION OF DATA WHAT IS MULTIMEDIA?
INTRODUCTION-3 : INTRODUCTION-3 REAL OBJECTS
VIRTUAL OBJECTS
REAL SPEECH Mutimedia
Data
Components COMPLEX INTERACTIONS
BETWEEN COMPONENTS IN
THE SCENE MAKE VIRTUAL
COMPONENTS SEEM MORE
REALISTIC EXAMPLE: AUGMENTED REALITY CONFERENCE
INTRODUCTION-4 : INTRODUCTION-4
MULTIMEDIA PROCESSING
APPLY SIGNAL PROCESSING TOOLS TO MULTIMEDIA DATA TO ENABLE:
REPRESENTATION
INTERPRETATION
ENCODING
DECODING
WHAT IS MULTIMEDIA PROCESSING?
INTRODUCTION-5 : INTRODUCTION-5
EFFECTIVE & EFFICIENT
ACCESS
MANIPULATION
EXCHANGE
STORAGE
OF MULTIMEDIA CONTENT
GOALS OF MULTIMEDIA PROCESSING
CONTINUING… : CONTINUING… INTRODUCTION
MULTIMEDIA APPLICATIONS
IMPACT OF MULTIMEDIA
CONTENT-BASED IMAGE RETRIEVAL
(CBIR)
MPEG-7
RESEARCH ISSUES
MULTIMEDIA APPLICATIONS-1 : MULTIMEDIA APPLICATIONS-1 GPS NAVIGATION SCALABLE VIDEO
STREAMING
MULTIMEDIA APPLICATIONS-2 : MULTIMEDIA APPLICATIONS-2 E-COMMERCE TELEPRESENCE CELLULAR
MULTIMEDIA APPLICATIONS-3 : MULTIMEDIA APPLICATIONS-3 MORE SPECIFIC EXAMPLES
MULTIMEDIA APPLICATION GOALS
IMPROVE INTERPERSONAL COMMUNICATION
PROMOTE UNDERSTANDING OF IDEAS
ALLOW INTERACTIVITY WITH MEDIA
INCREASE ACCESSIBILITY TO DATA
MPEG-4, 7, 21
JPEG-2000
MP3 & PERCEPTUAL
CODING MULTIMEDIA STORAGE
VIDEO-ON-DEMAND
DIGITAL CINEMA
AUTHENTICATION
GOING ON… : GOING ON… INTRODUCTION
MULTIMEDIA APPLICATIONS
IMPACT OF MULTIMEDIA
CONTENT-BASED IMAGE RETRIEVAL
(CBIR)
MPEG-7
RESEARCH ISSUES
IMPACT OF MULTIMEDIA-4 : IMPACT OF MULTIMEDIA-4 WORLD INTERNET USAGE (July 23, 2005)
IMPACT OF MULTIMEDIA-2 : IMPACT OF MULTIMEDIA-2 USERS (S0CIETY) DEMAND
INCREASED MOBILITY
EASE-OF-USE
PERSONAL CUSTOMIZATION
DEVICE FLEXIBILITY
HIGH LEVEL OF COLLABORATION WITH PEERS
DEVICES MUTATE AND BECOME
MULTI-FUNCTIONAL, NOT SPECIALIZED
EFFORTLESSLY PORTABLE, NOT STATIONARY
UBIQUITOUSLY NETWORKED, NOT ISOLATED
IMPACT OF MULTIMEDIA-3 : MULTI-FUNCTIONAL DEVICES MUST
BROWSE INTERNET
ENTERTAIN
BE EASY-TO-USE
CUSTOMIZATION
PERSONALIZATION (THEMES, PREFERENCES)
NETWORKED
CAPABLE OF CONNECTING TO MANY DIFFERENT NETWORKS (INTERNET, P.O.T.S., LAN, CELLULAR, BLUETOOTH, 802.11b, GPS) FACILITATE MANY
TYPES OF WORKFLOW
MANAGE USER’S TIME
IMPACT OF MULTIMEDIA-3
IMPACT OF MULTIMEDIA-4 :
CONVERGENCE
TECHNOLOGIES WHICH WERE
TOTALLY UNRELATED 10 YEARS
AGO ARE NOW UNIFIED UNDER
THE CONCEPT OF MULTIMEDIA IMPACT OF MULTIMEDIA-4
IMPACT OF MULTIMEDIA-5 : EXAMPLE: CELLULAR PHONES IMPACT OF MULTIMEDIA-5 PRIMARY CONSUMER USE:
WIRELESS TELEPHONY CONVERGED USES
PERSONAL ORGANIZER
INTERNET BROWSER/EMAIL
ENTERTAINMENT (MP3, RADIO)
VIDEO/STILL CAMERA
PAGER/MESSAGING (SMS)
IMPACT OF MULTIMEDIA-6 : IMPACT OF MULTIMEDIA-6 DEMANDS
FUNCTIONALITY
CONSUMPTION OF MANY MEDIA TYPES
CONNECTIVITY
PORTABILITY, ETC.
RESULT
HIGHLY COMPLEX DEVICES
PUSH TOWARDS DENSE CIRCUITRY
MULTIMEDIA DEVICES BECOME UBIQUITOUS
DEVICES GENERATE MULTIMEDIA DATA (INCLUDING IMAGES, VIDEO, AUDIO) OVERALL
MOVING ALONG… : MOVING ALONG… INTRODUCTION
MULTIMEDIA APPLICATIONS
IMPACT OF MULTIMEDIA
CONTENT-BASED IMAGE RETRIEVAL
(CBIR)
MPEG-7
RESEARCH ISSUES
CBIR OVERVIEW : MOTIVATION & GOALS
WHAT IS CBIR?
CONTRIBUTING DISCIPLINES
APPLICATION SCENARIOS
SOME SPECIFIC ISSUES
TYPICAL CAPABILITIES CBIR OVERVIEW
CBIR MEDIA FLOODING :
EFFECTS & PROCESSING
RESULT: DIGITAL MEDIA FLOOD
HOW DO WE COPE, TRACK, ORGANIZE IT ALL? POLAROID FILED FOR BANKRUPTCY
HAS DIGITAL KILLED FILM? IF SO, WHY?
CHEAP & DENSE STORAGE
CBIR MEDIA FLOODING EXAMPLE: GENERAL PHOTOGRAPHY SNAPSHOT PREVIEWS
EASY SHARING VIA INTERNET
MEMORY REUSABLE
PRINTER TECHNOLOGY
CBIR MOTIVATION : DEVICE FUNCTION CONVERGENCE
DATA RAPIDLY GENERATED BY MANY DEVICES
INTERNET ACTS AS GLOBAL TRANSPORT
DATA CONSUMED BY DEVICES ON DEMAND
MULTIMEDIA DATA NEEDS TO BE
EFFICIENTLY STORED
INDEXED ACCURATELY
EASILY RETRIEVED CBIR MOTIVATION
CBIR IS… : CONTENT BASED IMAGE RETRIEVAL
PART OF MULTIMEDIA INDEXING
IMAGES (2-D SPACE-DEPENDENT SIGNALS)
VIDEO (TIME-VARYING IMAGE SET)
AUDIO (1-D TIME-DEPENDENT SIGNALS)
TEXT (e.g. BOOK INDEX, SEARCH ENGINES)
COMPUTER BASED
HIGHLY AUTOMATED
DIFFICULT TO DO PROPERLY CBIR IS…
CBIR SIMPLE EXAMPLE : FOR A GIVEN QUERY…
EXAMPLE IMAGE
ROUGH SKETCH
EXPLICIT DESCRIPTION CRITERIA
…RETURN ALL ‘SIMILAR’ IMAGES CBIR SIMPLE EXAMPLE RETRIEVAL
SYSTEM
CBIR QUERY TYPES : CBIR QUERY TYPES SKETCH EXAMPLE COLOR SHAPE TEXTURE MORE COMPLEX TYPES EXIST YET ABOVE ARE MOST FUNDAMENTAL & MOST REGULARLY USED
CBIR CONTRIBUTORS : COMBINES HIGH-TECH ELEMENTS
MULTIMEDIA/SIGNAL/IMAGE PROCESSING
COMPUTER VISION/PATTERN RECOGNITION
COMPUTER SCIENCES
(I.E. HUMAN-COMPUTER INTERACTION)
AND MORE TRADITIONAL CONCEPTS
PSYCHOLOGY/HUMAN PERCEPTION
INFORMATION SCIENCES (I.E. LIBRARY) CBIR CONTRIBUTORS
CBIR SCENARIOS : a a a GOVERNMENT (E.G. MUGSHOTS)
ENTERTAINMENT (FILM, TV) DESIGN/VISUAL ARTS INDUSTRY (LOGO MANAGEMENT) SOME CBIR APPLICATION AREAS
CBIR SCENARIOS MEDICAL IMAGING ART/CULTURAL HERITAGE
CBIR VERSUS TEXT : IMPORTANT QUESTION ARISES:
“WHY NOT SIMPLY INDEX USING TEXT?”
(YAHOO! HAS HAD SOME SUCCESS WITH THIS)
INTUITIVE, YET USING TEXT IS
SIMPLE BUT SIMPLISTIC
TIME CONSUMING – CAN’T AUTOMATE
HIGHLY SUBJECTIVE & USER-DEPENDENT
SUSCEPTIBLE TO TRANSLATION PROBLEMS
CBIR VERSUS TEXT
CBIR BASIC STRUCTURE : CBIR BASIC STRUCTURE FEATURE
EXTRACTION I N D E X SIMILARITY CALCULATION GENERATION
OF RESULTS USER
INTERFACE SIMILAR
RESULTS QUERY FEATURE
DESCRIPTIONS 3 BASIC FEATURES
COLOR, TEXTURE, SHAPE
MANY DESCRIPTORS
MPEG-7 IS ISO STANDARD
REALLY A DESIGN CHOICE
SIMILARITY
OPEN TO RESEARCH
LITTLE PERCEPTUAL CONSIDERATION
CBIR (DIS)SIMILARITY? :
ON WHAT BASIS ARE THEY SIMILAR?
COLOR CONTENT?
SHAPE CONTENT?
HIGH LEVEL IDEAS (‘MASKS’, ‘GENDER’)?
PERCEPTION IS ALWAYS AN ISSUE CONSIDER THREE IMAGES
CBIR (DIS)SIMILARITY? SIMILARITY IS NOT SO SIMPLE
CBIR SIMILARITY : CBIR SIMILARITY DOMAIN [0,1]
CAN BE CALCULATED MANY WAYS
GENERALIZED
MINKOWSKI
CANBERRA
PERCEPTUAL
MEASURE
CBIR TYPICAL ABILITIES : EFFECTIVE QUERIES IN
COLOR, TEXTURE, SHAPE
SIMPLE HYBRID QUERIES
DESCRIPTOR SUPERVECTORS
WEIGHTED AVERAGE OF (DIS)SIMILARITIES
RELEVANCE FEEDBACK
USER PLACED IN LOOP GIVES BETTER RESULTS
STATISTICAL APPROACHES
APPLY/ADJUST FEATURE WEIGHTS TO RELEVANT/IRRELEVANT ELEMENTS CBIR TYPICAL ABILITIES
CBIR SUMMARY : CBIR SUMMARY BORN FROM MULTIMEDIA FLOOD
TEXT TOO SIMPLE AND LABORIOUS
SYSTEMS WORK DECENTLY IN VITRO
QUERY BY SHAPE, COLOR, TEXTURE, EXAMPLE
SHORTCOMINGS
NEED RELEVANCE FEEDBACK & PERCEPTUAL
HYBRID QUERIES DIFFICULT TO CREATE
SEMANTIC GAP NEEDS TO BE BRIDGED
MPEG-7: IMPORTANT DEVELOPMENT
GOING FORWARD… : GOING FORWARD… INTRODUCTION
MULTIMEDIA APPLICATIONS
IMPACT OF MULTIMEDIA
CONTENT-BASED IMAGE RETRIEVAL
(CBIR)
MPEG-7
RESEARCH ISSUES
MPEG : MPEG MOTION PICTURES EXPERT GROUP
MPEG-1
MPEG-2
MPEG-4
MPEG-7: ISO/IEC 15938
MULTIMEDIA CONTENT DESCRIPTION INTERFACE
MPEG-21
MPEG-1 & MPEG-2 : MPEG-1 & MPEG-2 MPEG-1 (c. 1992)
BASIC VIDEO CODING USING DPCM & DCT
TARGET: CD-BASED VIDEO & MULTIMEDIA
USE I, B & P-FRAMES IN YUV SPACE
MPEG-2 (c. 1994)
SUPERSET OF MPEG-1
GOAL: DTV/DSS OR ATM TRANSPORT
MINIMUM OF NTSC/PAL QUALITY
MORE ERROR RESILIENT
SCALABLE – GRACEFUL DEGRADATION
MPEG-4 & MPEG-21 : MPEG-4 & MPEG-21 MPEG-4 (c. 1998)
TOOLS TO AUTHOR MULTIMEDIA CONTENT
TRAFFIC AWARE, ERROR RESILIENT
OBJECT-BASED CODING
VERY EFFICIENT FOR LOW BIT-RATES
MPEG-21 (STARTED JUNE 2000)
AN OPEN “MULTIMEDIA FRAMEWORK” IDEA
ADDRESSES DIGITAL RIGHTS MANAGEMENT
ENHANCED DELIVERY & ACCESS OF DATA FOR DEVICES ON HETEROGENEOUS NETWORKS
MPEG-7 NEW PARADIGM : MPEG-7 NEW PARADIGM UNLIKE MPEG-1, MPEG-2, & MPEG-4
DOESN’T REPRESENT CONTENT ITSELF
MPEG-7 ONLY DESCRIBES CONTENT
DIFFICULT CONCEPT FOR SOME TO GRASP
APPLICABLE TO
IMAGES
VIDEO
INDEPENDENT OF
STORAGE
ARCHITECTURE AUDIO & SPEECH
TEXT TRANSPORT
CODING
MPEG-7 HOW IT DIFFERS : MPEG-7 HOW IT DIFFERS MPEG-1
TAKES INPUT FRAMES AND REPRESENTS AS AN BINARY ENCODED VIDEO BITSTREAM
MPEG-7
TAKES VIDEO FRAMES (SAY MPEG-1 FORMAT) AND DESCRIBES CONTENTS OF EACH FRAME.
FRAME 1: COLOR CONTENT: 20% WHITE, 14% BLUE, SHAPES: BRIDGE, etc. FRAME 2: COLOR CONTENT: 20% WHITE, 15% BLUE, SHAPES: BRIDGE, etc. FRAME 3: COLOR CONTENT: 21% WHITE, 14% BLUE, SHAPES: BRIDGE, etc.
MPEG-7 SCOPE : MPEG-7 SCOPE
MPEG-7
SCOPE FEATURE
EXTRACTION
ALGORITHM CODING
SCHEME CONTENT
DESCRIPTION OTHER
ELEMENTS
. . . MULTIMEDIA DATA
MPEG-7 GOALS : MPEG-7 GOALS DESCRIBE MULTIMEDIA CONTENT
SET OF DESCRIPTORS (D)
RELATIONS BETWEEN DESCRIPTORS
SET OF DESCRIPTION SCHEMES (DS)
LANGUAGE DEFINING D’s & DS’s
DESCRIPTION DEFINITION LANGUAGE (DDL)
BASED ON XML (eXtensible Markup Language)
USED TO BUILD UP NEW D’s & DS’s
ENCODING OF D’s FOR EFFICIENCY
MPEG-7 SUMMARY-1 : MPEG-7 SUMMARY-1 STANDARDIZED DESCRIPTIONS
APPLIES TO ALL DIGITAL MEDIA
CBIR IS CASE FOR STILL IMAGES
DOES NOT REPRESENT DATA ITSELF
DESCRIBES WHAT DATA REPRESENTS
SETS THE BAR FOR SYSTEMS
MULTIMEDIA/IMAGE RETRIEVAL SYSTEMS NEED AT LEAST MPEG-7 CONFORMANCE
MPEG-7 SUMMARY-2 : MPEG-7 SUMMARY-2 DOES NOT ADDRESS
SIMILARITY
RELEVANCE FEEDBACK
FEATURE EXTRACTION
HYBRID QUERY GENERATION
ARCHIVE ORGANIZATION
THE ABOVE ISSUES HAVE BEEN PURPOSEFULLY LEFT OPEN FOR INNOVATION
FORGING AHEAD… : FORGING AHEAD… INTRODUCTION
MULTIMEDIA APPLICATIONS
IMPACT OF MULTIMEDIA
CONTENT-BASED IMAGE RETRIEVAL
(CBIR)
MPEG-7
RESEARCH ISSUES
RESEARCH ISSUES : SHORTCOMINGS OF CBIR SYSTEMS
ONGOING RESEARCH
RELEVANCE FEEDBACK
HYBRID QUERY GENERATION
DISTRIBUTED MULTIMEDIA INDEXING
OPEN RESEARCH AVENUES RESEARCH ISSUES
CBIR SHORTCOMINGS-1 : CBIR SHORTCOMINGS-1 COLOR
USUALLY GLOBAL
HIGH DIMENSIONALITY
GAMMA NONLINEARITIES CAUSE PROBLEMS
SHAPE
COMPLICATED & DIFFICULT
OCCLUSION ISSUES DURING EXTRACTION
TEXTURE
COMPLICATED & UNINTUITIVE
USER-SYSTEM RIFT FOR QUERY CREATION
CBIR SHORTCOMINGS-2 : CBIR SHORTCOMINGS-2 PERCEPTUAL ISSUES
SUBTLE DIFFERENCES BETWEEN VIEWERS
COLOR-BLIND USERS
SIMILARITY MEASURES
NEED TO BE TUNED TO DESCRIPTORS
e.g. EUCLIDEAN DISTANCE NOT APPLICABLE IN NON-EUCLIDEAN DESCRIPTION SPACE
RELEVANCE FEEDBACK
PERFORMED AT GLOBAL (IMAGE) LEVEL
NEED TO ADDRESS SPECIFIC IMAGE ELEMENTS
ONGOING RESEARCH-2 : ONGOING RESEARCH-2 ITERATIVE QUERY REFINEMENT
PLACE USER IN LOOP TO ITERATIVELY IMPROVE RETRIEVAL RATES
HIGH-DIMENSIONAL SPACE NEEDS PRUNING
EMPHASIZED FEATURE(S) MUST BE FOUND
TYPICAL APPROACHES
STATISTICAL METHODS
FEATURE WEIGHTING RELEVANCE FEEDBACK
ONGOING RESEARCH-2 : ONGOING RESEARCH-2 FEATURE SELECTIVE INTERFACE
WHY CHOOSE IMAGES ON WHOLE? REQUIRES PROCESSING/STATS TO FIND GOOD FEATURES
USER CAN EXPLICITLY INDICATE ELEMENTS OF IMAGE WHICH ARE GOOD: NO GUESSWORK RELEVANT COLOR RELEVANT SHAPE EXPLICIT FEATURES TO R.F. ENGINE RELEVANCE FEEDBACK
ONGOING RESEARCH-3 : ONGOING RESEARCH-3 TYPICALLY USED APPROACHES
BOOLEAN (AND, OR & NOT OPERATORS)
EUCLIDEAN (MINKOWSKI W/ r=1)
WEIGHTED AVERAGE (WA) i.e. SUPERVECTORS
DISADVANTAGES
EUCLIDEAN: FCN OF DESCRIPTORS – CHANGE DESCRIPTOR, DRASTICALLY ALTER MEASURE
WA: INFLEXIBLE FOR HIGH LEVEL QUERIES, SUPERVECTORS IMPOSE CERTAIN STRUCTURE
BOOLEAN: HARD LIMITED TO LOGIC FCNs
ALL LACK PERCEPTUAL CONSIDERATIONS SIMILARITY AGGREGATION/HYBRID QUERIES
ONGOING RESEARCH-4 : FUZZY AGGREGATION OF DECISIONS
USE MEMBERSHIP FUNCTION TO ‘FUZZIFY’ DISTANCES & GENERATE A ‘FUZZY DECISION’
EXPONENTIAL MODELS HUMAN PERCEPTION
ONGOING RESEARCH-4 SIMILARITY AGGREGATION/HYBRID QUERIES FUZZY
MEMBERSHIP
FUNCTION SIMILARITY DISTANCE d FUZZY DISTANCE DECISION m
ONGOING RESEARCH-5 : INDEXES USUALLY CENTRALIZED
ENTIRE SYSTEM FAILS IF COMPONENT FAILS
NO GRACEFUL PERFORMANCE DEGRADATION
HIGH DATA VOLUME = HIGH SYSTEM REQ’S
DISTRIBUTED INDEXES
SPREAD WORKLOAD OVER MANY SUBSYSTEMS
INCREASE REDUNDANCY
P2P SYSTEMS LACK CENTRALIZED ELEMENTS
P2P SYSTEMS RESEMBLE SOCIAL NETWORKS ONGOING RESEARCH-5 DISTRIBUTED MULTIMEDIA INDEXING
ONGOING RESEARCH-6 : SMALL WORLD INDEXING MODEL1
SOCIOLOGICAL PEER DESCRIPTIONS
WE ARE NOT BLIND TO WHO OUR PEERS ARE
PEOPLE KEEP MEMORY OF THEIR PEERS
WE ARE NOT BLIND TO HOW OUR PEERS ARE
WE REFER OTHERS TO OUR PEERS
EXAMPLE ONGOING RESEARCH-6 DISTRIBUTED MULTIMEDIA INDEXING [1] P. Androutsos, D. Androutsos and A. N. Venetsanopoulos, “A distributed fault-tolerant MPEG-7 retrieval scheme based on small world theory”, Distributed Media Technologies and Applications Special Issue of IEEE Transactions on Multimedia, under review.
ONGOING RESEARCH-7 : INDEX AND ARCHIVE BECOME ONE
SWIM DATA STORED IN ARCHIVE OBJECTS
EACH DATA OBJECT BEHAVES AS OWN AGENT
AGENTS ARE EFFECTIVE IN HIGHLY NETWORKED ENVIRONMENTS (SWIM)
RETRIEVALS
AGENT BASED RETRIEVAL
USE OF REFERRAL BASED TECHNIQUE SIMILAR TO ‘SIX DEGREES OF SEPARATION’
CURRENTLY PERFORMED WITH IMAGES ONGOING RESEARCH-7 DISTRIBUTED MULTIMEDIA INDEXING
ONGOING RESEARCH-8 : ONGOING RESEARCH-8 DISTRIBUTED MULTIMEDIA INDEXING2 [2] P. Androutsos, D. Androutsos and A. N. Venetsanopoulos, “Graceful image retrieval performance degradation using small world distributed indexing”, International Conference on Image Processing ICIP2005, Genoa, Italy.
RESEARCH AVENUES-1 : RESEARCH AVENUES-1 HYBRID QUERIES & AGGREGATION
WHAT DO WEIGHTS MEAN? HOW TO CHOOSE?
ALTERNATIVE AGGREGATIONS METHODS
ADAPTIVE SCHEMES USING REL. FEEDBACK
USER INTERFACE
BRIDGE SEMANTIC GAP BETWEEN USER’S IDEA, AND ABILITY TO EXPRESS AS A QUERY
ALTERNATIVE INTERFACES–ICONIC, SEMANTIC
RESEARCH AVENUES-2 : RESEARCH AVENUES-2 PERCEPTUAL ISSUES
EMPHASIS OF DOMINATING FEATURES
FEATURE MASKING
EMOTIONAL INDEXING/
ALL USERS DIFFERENT–CUSTOMIZED PROFILE
ARCHIVE DEPENDENCE
SYSTEMS USUALLY SPECIALIZED
ADAPTIVE INDEXING – MOST APPROPRIATE SYSTEM USED BASED ON PRELIMINARY SURVEY OF CANDIDATE DATABASE
RESEARCH AVENUES-3 : RESEARCH AVENUES-3 DISTRIBUTED INDEXING
DISTRIBUTED INDEXES & RETRIEVAL
INDEX SYNCHRONIZATION
RESULTS ORGANIZATION & RANKING
SWIM OVERHEAD ESTIMATION
EXTENSION OF SWIM TO OTHER DATA TYPES
INCORPORATE TEXT METHODS
TEXT-INDEXING USING LIMITED VOCABULARY
DON’T REJECT BUT USE INTELLIGENTLY
EXTEND TO MPEG-21 & METADATA
SUMMARY-1 : SUMMARY-1 MULTIMEDIA PROCESSING
RESULTS FROM MULTIMEDIA EXPLOSION
USERS DEMANDING MORE FROM DEVICES
DEVICES ARE CONVERGING
CONTENT BASED IMAGE RETRIEVAL
NECESSARY TO TRACK VISUAL SEA OF DATA
GOOD CAPABILITIES, BUT W/ SHORTCOMINGS
PERCEPTUAL/SUBJECTIVE ISSUES
RELEVANCE FEEDBACK
DISTRIBUTED CONCEPTS BECOMING CRITICAL
SUMMARY-2 : SUMMARY-2 MPEG-7
AIMED AT STANDARDIZING DESCRIPTIONS
RADICALLY DIFFERENT THAN PREVIOUS MPEGs
DDL IS AN EXTENSION OF XML SCHEMA
APPLICABLE TO ALL MULTIMEDIA DATA
ALWAYS MORE TO DO
MPEG-7 HAS LEFT MANY ISSUES OPEN
CBIR NEEDS TO ADDRESS USERS, PERCEPTION, HYBRID QUERIES, DISTRIBUTED SYSTEMS, ETC
VIBRANT RESEARCH COMMUNITY
THANK YOU : THANK YOU
IMPACT OF MULTIMEDIA : HIGH FLEXIBILITY RESULTS IN
RISE IN DATA GENERATION & STORAGE
INCREASE IN BANDWIDTH NEEDS
ONE TOOL DOING WORK OF MANY
MANY TYPES OF NETWORKS CAUSE
COMPLEX HARDWARE COMBINATIONS
ONE DEVICE CONNECTING TO ALL NETWORKS
SMALL, PORTABLE DEVICES
MINIATURIZATED WITH HUGE CAPABILITIES
ONE DEVICE REPLACES MANY IMPACT OF MULTIMEDIA
CBIR WHO’S WHO : CBIR WHO’S WHO
MPEG-7 D, DS, & DDL :
DEFINED
VIA DDL
DEFINED IN MPEG-7
STANDARD MPEG-7 D, DS, & DDL DDL D D DS DS D D DS D BUILDING MORE Ds & DSs USING THE DDL
MPEG-7 COMPONENTS : MPEG-7 COMPONENTS SYSTEMS
DDL
VISUAL
PRIMARY CONCERN FOR THIS PRESENTATION
AUDIO
MULTIMEDIA DESCRIPTION SCHEMES
EXPERIMENTATION MODEL (XM)
CONFORMANCE
MPEG-7 VISUAL COMPONENT : MPEG-7 VISUAL COMPONENT BASIC DESCRIPTORS
GRID LAYOUT
2D/3D VIEW
TIME SERIES
SPATIAL 2D COORDS
TEMPORAL INTERPOLATION
COLOR DESCRIPTORS
COLOR SPACE
COLOR QUANTIZATION
DOMINANT COLOR
SCALABLE COLOR
COLOR STRUCTURE
COLOR LAYOUT
GoF/GoP COLOR
OTHER
FACE RECOGNITION TEXTURE DESCRIPTORS
EDGE HISTOGRAM
HOMOGENEOUS TEXTURE
TEXTURE BROWSING
SHAPE DESCRIPTORS
REGION-BASED
CONTOUR-BASED
3D SHAPE
MOTION DESCRIPTORS
CAMERA MOTION
MOTION TRAJECTORY
PARAMETRIC MOTION
MOTION ACTIVITY
LOCALIZATION
SPATIO-TEMPORAL
REGION LOCATOR HIGHLIGHTED DESCRIPTORS USED BY UofT
ONGOING RESEARCH : FUZZY AGGREGATION OF DECISIONS
AGGREGATE DECISIONS USING LOGIC
USE COMPENSATIVE OPERATOR
PARAMETER g CONTROLS DEGREE OF ANDNESS (max) & ORNESS (min)
RESULT IS A SINGLE VALUE IN [0,1] INDICATING OVERALL IMAGE SIMILARITY ONGOING RESEARCH SIMILARITY AGGREGATION/HYBRID QUERIES