mmir challenges

Content-based Multimedia Information Retrieval: Challenges & Opportunities: 

Content-based Multimedia Information Retrieval: Challenges & Opportunities Stefan Rüger et al http://km.doc.ic.ac.uk

Content-based MM IR: 

Content-based MM IR Multimedia Information Retrieval aims, applications and a retrieval example Challenges semantic gap polysemy the “multi” in multimedia Video Search and Summarisation Music Retrieval Information Navigation

Need for Information Retrieval : 

Need for Information Retrieval Information is of no use unless you can actually access it.

Multimedia Information Retrieval: 

Multimedia Information Retrieval archive text, video, images, speech, music, combinations query text, stills, sketch, speech, humming, examples content-based present results browsing, summaries, story boards document clustering, cluster summaries utilise relevance feedback

Query-retrieval matrix: 

Query-retrieval matrix text video images speech music sketches multimedia text stills sketch speech sound humming examples query doc Example

Some applications: 

Some applications medicine get diagnosis of cases with similar scans law enforcement child pornography prosecution copyright infringement (music, videos, images) CCTV video retrieval (car park, public spaces) digital libraries searching, visualisation, summaries, browsing

Example: get me similar images!: 

Example: get me similar images! extract, eg, 50,000 primitive features provide positive image examples, generate negative examples at random Feature selection & learning ADA-Boost, K-NN, SVM, ... eg, compute separating hyper-plane and rank all images in database accordingly

Example: Jupiter video search: 

Example: Jupiter video search video segmentation: generate paragraphs identify key frame of video paragraph get Jupiter example images, eg, from web Google image search: treat video search as image search [with Marcus Pickering and David Sinclair, CVIR 2002]

Result list of video key frames: 

Result list of video key frames

Content-based MM IR: 

Content-based MM IR Multimedia Information Retrieval aims, applications and a retrieval example Challenges semantic gap polysemy the “multi” in multimedia Video Search and Summarisation Music Retrieval Information Navigation

The semantic gap: 

The semantic gap

Bridging the semantic gap: 

Bridging the semantic gap region segmentation + region classification (grass, water, ...) using simple models for complex concepts (grass+plates+people = barbeque)

Region segmentation: 

Region segmentation collaboration with AT&T Research, Cambridge

Region classifiers: 

Region classifiers visual categories grass, sky (blue), sky (cloudy), skin, trees, wood, water, sand, brick, snow, tarmac give regions a probability of membership Positive Examples Negative Examples Cluster Prune Cluster Nearest Neighbours Test region Probability Cluster

Example: grass classifier: 

Example: grass classifier

Modelling semantic concepts: 

Modelling semantic concepts outdoor town crowd sky grass skin tarmac Bayesian networks

Content-based MM IR: 

Content-based MM IR Multimedia Information Retrieval aims, applications and a retrieval example Challenges semantic gap polysemy the “multi” in multimedia Video Search and Summarisation Music Retrieval Information Navigation

Polysemy: 

Polysemy old Volkswagen colour contrast road signs outback

Relevance feedback: 

Relevance feedback system needs plasticity (parameters) images are quickly assessed and user can inform system explicitly or implicitly system needs to learn from user = change the parameters

Relevance feedback mechanism : 

Relevance feedback mechanism centre = query = ideal result results are displayed such that distance to centre is the dissimilarity to the query user indicates her/his idea of similarity by rearranging the displayed results system recomputes optimal parameters for this specific query automatically

Example: relevance feedback: 

Example: relevance feedback query initial result

User action: 

User action

After relevance feedback: 

After relevance feedback number of relevant images has doubled

GUI: 

GUI

User modelling: 

User modelling simulate users who click at most three images mean average precision increase - weight space movement: 15% - query change and weight change: 58% [with Daniel Heesch, ECIR 2003]

Content-based MM IR: 

Content-based MM IR Multimedia Information Retrieval aims, applications and a retrieval example Challenges semantic gap polysemy the “multi” in multimedia Video Search and Summarisation Music Retrieval Information Navigation

The “multi” of multimedia: 

The “multi” of multimedia high-level features words and phrases from text, speech recognition medium-level features face detector, regions classifiers, outdoor etc low-level features Fourier transforms, wavelet decomposition, texture histograms, colour histograms, shape primitives, filter primitives

Unified theoretical framework: 

Unified theoretical framework document network index time run time query network

Content-based MM IR: 

Content-based MM IR Multimedia Information Retrieval aims, applications and a retrieval example Challenges semantic gap polysemy the “multi” in multimedia Video Search and Summarisation Music Retrieval Information Navigation

System overview: 

System overview [with M Pickering, D Heesch, R O’Callaghan and D Bull, TREC 2002]

TREC 2002 evaluation: 10 best manual runs: 

TREC 2002 evaluation: 10 best manual runs [with M Pickering, D Heesch, R O’Callaghan and D Bull, TREC 2002]

VideoSummary: 

Video Summary story-level segmentation keyframe summary videotext summary full-text search named entities [with L Wong and M Pickering]

Content-based MM IR: 

Content-based MM IR Multimedia Information Retrieval aims, applications and a retrieval example Challenges semantic gap polysemy the “multi” in multimedia Video Search and Summarisation Music Retrieval Information Navigation

Polyphonic Music Indexing Technique: 

Polyphonic Music Indexing Technique n-grams encode music as text strings using pitch and onsets index text words with text search engine process query in the same way application: eg, Query by Humming [with Shyamala Doraisamy, ISMIR 2000, ISMIR 2001, ISMIR 2002]

Monophonic pitch n-gramming : 

Monophonic pitch n-gramming 0 +7 0 +2 0 -2 0 -2 0 Interval: Example: musical strings with interval-only representation [0 +7 0 +2] ZGZB [+7 0 +2 0] GZBZ [0 +2 0 -2] ZBZb

N-grams and polyphony: 

N-grams and polyphony Polyphony: index all monophonic combinations Encoded rhythm in similar way Performed well with known-item search Studied fault-tolerance

Content-based MM IR: 

Content-based MM IR Multimedia Information Retrieval aims, applications and a retrieval example Challenges semantic gap polysemy the “multi” in multimedia Video Search and Summarisation Music Retrieval Information Navigation

Presentation of search results: 

Presentation of search results ranked list adequate? [funded by NSF-EU: Cultural Heritage Language Technologies] [with D Heesch et al]

Vision: labelled clusters: 

Vision: labelled clusters suggest keywords refine query drill down/up

Keyword computation: 

Keyword computation example: search for “computer” related keywords: “hardware”, “software”, “IBM”, “Linux”, etc

Document representation: 

Document representation word histogram vectors (“bag of words”) cost dog drug hospital hunt impact mafia reform … vocabulary doc1 doc2 …

New document representation: 

New document representation use keywords only for returned documents low-dimensional vector (10-30 dim) efficient clustering no curse of dimensionality

Slide43: 

Sammon

Tree-Map: 

Tree-Map

Slide45: 

DendroVis

Slide46: 

Radial

Slide47: 

Radial

Conclusions: 

Conclusions Multimedia Information Retrieval Challenging research questions Draws on computer vision, audio processing, natural language analysis, unstructured document analysis, information retrieval, information visualisation, computer human interaction, artificial intelligence

Collaborations: 

Collaborations part of the High Performance Informatics area existing collaborations with Tufts’s Perseus Digital Library Imperial’s Newton Project AT&T Research, Cambridge ISE Dept of the Ben Gurion University, Israel EE Dept of Bristol University the Greenstone Digital Library, U of Waikato, NZ intended collaborations with Center for Intelligent Information Retrieval, Umass EIE Dept of Hong Kong Polytechnic University

Content-based Multimedia Information Retrieval: Challenges & Opportunities: 

Content-based Multimedia Information Retrieval: Challenges & Opportunities Stefan Rüger et al http://km.doc.ic.ac.uk

The semantic gap: 

The semantic gap

Rhythm encoding: 

Rhythm encoding we use ratios, not absolute values and onset time differences, not durations ri = (oi+2 - oi+1)/(oi+1 - oi) we quantise this number (use 21 letters) this is already invariant to tempo change

Keyword computation: 

Keyword computation potentially interesting for the user related to the returned documents able to discriminate the returned documents candidate keywords: medium document freq rank words with (h/d)  h log(|H|/h) h returned-document frequency d document frequency H returned-document set keywords: highly ranked candidates

Hierarchical clustering: 

Hierarchical clustering

Slide55: 

drill down DendroVis