07-MPEG-7

The MPEG-7 Standard - A Brief Tutorial -: 

The MPEG-7 Standard - A Brief Tutorial - Ali Tabatabai Sony US Research Laboratories February 27, 2001

Outline: 

2 Outline Objectives of the MPEG-7 Standard Main Elements of MPEG-7 Scope of MPEG-7 MPEG-7 Application Areas MPEG 7’s relation with other standards

PowerPoint Presentation: 

3 Why do we need MPEG-7 ? N e e d Content Management Fast & Accurate Access Personalized Content Production and Consumption Automation + Support for Advanced Query Visual Audio Sketch

MPEG: A Brief History (1): 

4 MPEG: A Brief History (1) MPEG: Moving Picture Experts Group ISO / IEC/JTC1/SC29/WG11 A Working Group of ISO/IEC in charge of the Development of Standards for Coded Representation of Digital Audio and Video Established in 1988

MPEG: A Brief History (2): 

5 MPEG: A Brief History (2) MPEG-1 : Interactive CD and MP3 11 / 1992 MPEG-2 : DTV, STB, DVD 11 / 1994 MPEG-4 : Web and Mobility ver1: 09 /1998 ver2: 11 /1999 MPEG-7 : ??? 08 / 2001 MPEG-21 : Multimedia Framework 11 / 2001

MPEG-7: What Is It ?: 

6 MPEG-7: What Is It ? THE MPEG 7 STANDARD Content Description of Various Audio Visual Information IS NOT a COMPRESSION Standard similar to MPEG-1/2/4 or their Extension IS NOT a STANDARD for FEATURE EXTRACTION/MATCHING Types of Audio Visual Information Audio, Speech Moving video, still pictures, graphics Information on how objects are combined in scenes

MPEG-7: Application Areas: 

MPEG-7: Application Areas Storage and retrieval of audiovisual databases ( image, film, radio archives ) Broadcast media selection ( radio, TV programs ) Surveillance ( traffic control, surface transportation, production chains ) E-commerce and Tele-shopping ( searching for clothes / patterns ) Remote sensing ( cartography, ecology, natural resources management ) Entertainment ( searching for a game, for a karaoke ) Cultural services ( museums, art galleries ) Journalism ( searching for events, persons ) Personalized news service on Internet ( push media filtering ) Intelligent multimedia presentations Educational applications Bio-medical applications

MPEG-7 Description Scope for AV Content : 

8 MPEG-7 Description Scope for AV Content Description Granularity Low-level High-level Form Access Classification Link Context

MPEG-7: Main Elements: 

9 MPEG-7: Main Elements Descriptors (D) syntax and semantics of each feature representation Description Schemes (DS) structure and semantics of the relationships between components Description Definition Language (DDL) creation of new DS’s modification/extension of existing DS’s

MPEG-7: Major Functionalities: 

10 MPEG-7: Major Functionalities Systems (ISO / IEC 15938 - 1) Description Definition Language (ISO / IEC 15938 - 2) Visual (ISO / IEC 15938 - 3) Audio (ISO / IEC 15938 - 4) Multimedia Description Schemes (ISO / IEC 15938 - 5) Reference Software (ISO / IEC 15938 - 6)

PowerPoint Presentation: 

11 DS1 DS2 DS2 DS3 D1 D3 D2 Systems 0001100 <Object> <Label/> <Definition/> . . </Object> DDL Instantiation MPEG-7: Main Elements (2)

MPEG-7: Systems: 

12 MPEG-7: Systems It defines tools to: manage and protect intellectual property synchronize between content and description provide for efficient storage and transport

MPEG-7: DDL and its Components : 

13 MPEG-7: DDL and its Components Description Definition Language: Creation of the Ds and DS’s: XML Schema & MPEG-7 Extensions Instantiation of XML XML Schema: Data types Simple and Complex types Elements, attributes Inheritance, Abstract types MPEG-7 extensions: Array and Matrix data type

MPEG-7: Audio: 

14 MPEG-7: Audio Sound Effects Music Instrument Timbre Spoken Content Melody Contour

MPEG-7: Visual (1): 

15 MPEG-7: Visual (1) Color quantization, dominant, scalable, color-structure, layout, GoF/GoP Texture Shape region-based, contour-based, 3D Motion camera motion, motion trajectory, parametric motion, motion activity

MPEG-7: Visual (2): 

16 MPEG-7: Visual (2) Localization spatio temporal Others face recognition

MPEG-7: Basic Visual Structures: 

17 MPEG-7: Basic Visual Structures Grid Layout 2D-3D Multiple View Time Series Spatial 2D Coordinates Temporal Interpolation

Low level Audio Visual descriptors: 

18 Low level Audio Visual descriptors Color Camera motion Motion activity Mosaic Color Motion trajectory Parametric motion Spatio-temporal shape Color Shape Position Texture Video segments Still regions Moving regions Audio segments Spoken content Spectral characterization Music: timbre, melody

MPEG-7: MMDS Basic Elements: 

19 MPEG-7: MMDS Basic Elements Datatype & structures Link & media localization Basic DSs Basic elements Schema tools Time, Duration, Medialocators Language Annotation,Person, Place Root, Top-level elements, Packages

MPEG 7: Content Management and Description: 

20 MPEG 7: Content Management and Description Content description Content management Creation & production Media Content Usage Conceptual aspects Structural aspects Title, Creator, Creation location & date, Purpose, Classification, Genre, Review, Parental guidance, etc. (Author generated) Format, Coding, Instances, Identification, Transcoding Hint, etc. (Several instances) Rights holder, Access rights, Usage Record, Financial aspects, etc. (Evolution) Datatype & structures Link & media localization Basic DSs Basic elements Schema tools Viewpoint of the structure: Segments Spatial / temporal structure Audio, video low-level Ds Elementary semantic information. Viewpoint of conceptual notions Events, objects, abstract concepts, and their relation

Example of Segment trees: 

21 Foreground Example of Segment trees SR1: · Creation, Usage meta information · Media description · Textual annotation · Color histogram, Texture SR2: · Shape · Color Histogram · Textual annotation SR6: · Color Histogram · Textual annotation SR5: · Shape · Textual annotation SR4: · Shape · Color Histogram · Textual annotation SR3: · Shape · Color Histogram · Textual annotation Background

PowerPoint Presentation: 

22 Segment Tree Shot1 Shot2 Shot3 Segment 1 Sub-segment 1 Sub-segment 2 Sub-segment 3 Sub-segment 4 segment 2 Segment 3 Segment 4 Segment 5 Segment 6 Segment 7 Semantic DS (Events) • Introduction • Summary • Program logo • Studio • Overview • News Presenter • News Items • International • Clinton Case • Pope in Cuba • National • Twins • Sports • Closing Time Axis

MPEG 7: Navigation and Access: 

23 MPEG 7: Navigation and Access Navigation & Access Summary Variation Content description Content management Creation & production Media Content Usage Conceptual aspects Structural aspects Datatype & structures Link & media localization Basic DSs Basic elements Schema tools Efficient support of: discovery, browsing, navigation, visualization Substitution of the original content Adaptation to terminal, network, or user preferences

MPEG 7: Hierarchical summary: 

24 MPEG 7: Hierarchical summary A-V Data Hierarchical Summary HighlightLevel HighlightLevel Highlight Segment Highlight Segment Highlight Segment Highlight Segment Highlight Segment Highlight Segment Highlight Segment

MPEG 7: Sequential summary: 

25 MPEG 7: Sequential summary Sequential Summary A-V Data Text Property Text Property Frame Property Frame Property Frame Property Sound Property Sound Property Sound Property

MPEG 7: Variation: 

26 MPEG 7: Variation Universal Multimedia Access Adapt delivery to network and terminal characteristics (QoS)

MPEG-7: Content Organization: 

27 MPEG-7: Content Organization Navigation & Access Summary Variation Analytic Model Content organization Content description Content management Creation & production Media Content Usage Conceptual aspects Structural aspects Description and organization of collection of documents Collection & Classification Datatype & structures Link & media localization Basic DSs Basic elements Schema tools

MPEG-7: Collection: 

28 MPEG-7: Collection

MPEG 7: User Interaction: 

29 MPEG 7: User Interaction Navigation & Access Summary Variation Analytic Model Content organization Content description Content management Creation & production Media Content Usage Conceptual aspects Structural aspects Collection & Classification User Interaction User preferences Datatype & structures Link & media localization Basic DSs Basic elements Schema tools User identification and preferences: Filtering, search and browsing

MPEG-7 Its Relation with other standards: 

30 MPEG-7 Its Relation with other standards AHG on “Metadata harmonization”: SMPTE: Metadata dictionary, KLV encoding Dublin Core Metadata Initiative European Broadcast Union AHG on TV AnyTime Application Large number of Liaisons: SMPTE Dublin Core W3C (XML Schema) etc.

MPEG-7: TimeLine - The Work Plan: 

Competition: Individual work Definition scope and r MPEG-7: TimeLine - The Work Plan Call for proposals 2001 2000 1999 1998 1996 Working draft Committee draft Draft international standard Final committee draft International standard Divergence Convergence

Conclusions on AV Content Description and MPEG-7: 

32 Conclusions on AV Content Description and MPEG-7 MPEG -7: AV content description for interoperable application Description Definition Language: XML Schema (flexibility) + Binary version (efficiency) Description Schemes: Library of description tools Covers a wide range of generic needs