case study1 1

案例研討一Video Data Management Systems: Metadata and Architecture : 案例研討一 Video Data Management Systems: Metadata and Architecture Chapter 9 of Multimedia Data Management: Using Metadata to Integrate and Apply Digital Media


課程目標 : 課程目標 以一Video Data Management System的設計為例，探討設計一Digital Library時應該注意的事項，以及詮釋資料在DL中所扮演的角色 Good understanding of digital media Typical applications of digital media Types of queries


課程內容 : 課程內容 Introduction Video Data Management System (VDMS) Application of Video Classification of Video Queries ViMOD: The Video Data Model


Introduction : Introduction Video data management system (VDMS) Storage of video on computer systems Content based retrieval Real-time synchronized delivery of video Content based retrieval Data modeling Automatic extraction of data models Query and retrieval mechanisms


Video Data Management System (VDMS) : Video Data Management System (VDMS)


What is a VDMS : What is a VDMS A software system which provides Content based access to video data Audiovisual content of video Semantic content of video Facilities Facilities provided by standard DBMS (insertion, deletion, schema definition…) User interface Predefined set of query classes and an associated query interface Tools for navigation and manipulation video data


Example Scenario: Sporting Event VDMS (I) : Example Scenario: Sporting Event VDMS (I) Purpose Postgame analysis Plan strategies for future games Analyze game strategies of opposing teams Scenario 1 Remember the OSU game from last fall? Retrieve

Example Scenario: Sporting Event VDMS (II) : Example Scenario: Sporting Event VDMS (II) Scenario 2 Didn’t OSU score a field goal in the 3rd quarter of the game? Locate The retrieved video is marked with the time points of all field goal attempts Scenario 3 Can we see a close up shot of this kick? Retrieve The database is searched for a close up shot and the video is cued if the search is successful


Example Scenario: Sporting Event VDMS (III) : Example Scenario: Sporting Event VDMS (III) Scenario 4 Let’s look at the track of the kicker’s foot Tracking Mode. Using the interface, a bounding box is placed around the kicker’s foot to indicate the object to be tracked. The system tracks the kicker’s foot through the shot, and displays a track of the foot Scenario 5 Let’s see other kickers with similar kicks in last years NCAA football Similarity Search. Compare the kickers’ tracks for attempts. Ranked set


Content of Video : Content of Video


Content of Video : Content of Video Semantic content Message of information conveyed Audiovisual content Video clips and audio signals Distinction: Amount of contextual information and knowledge required to extract contents


Semantic Content : Semantic Content Content extraction Need background knowledge Complex, manually Example Emotion, Classification Similar to manage textual information Access: Finer grain scenes, shot


Audiovisual Content : Audiovisual Content Content extraction No Need background knowledge (Semi-)automatically Example Object recognition, object tracking over time, temporal events recognition, word and sentence recognition, unusual sound events Camera and object motion, color and texture properties, audio properties


Application of Video : Application of Video


Feature Films : Feature Films Film viewer List films with Title=X, Actors=Y, Directors=Z,… List films with Genre=Western Film critics Find scene where Actor=X & Emotion=cry Find shot with camera=stationary, Lens actions=Zoom in Find scene with Special Effect=Morphing Film Database Managers Number of rentals for Title=X, Actor=Y Average number of movies per customer per week


News Video : News Video News Browser Retrieve hockey events occurred between 1994 and 1995 Retrieve results of 1992 elections News Producers and Reporters News reuse Nomination of a new presidential candidate Highlight the person’s life beginning from birth


Sporting Event Videos : Sporting Event Videos Casual Viewer Locating game videos (like film viewers) Sports Coaches, Trainers Coaching teams, analyzing player performance, game strategies Example Queries


Classification of Video Queries : Classification of Video Queries


Content Type : Content Type Semantic Query Require high level semantic recognition and interpretation of the video content Require metadata generated manually Find scene with Actor=X & Emotion=Crying Audiovisual Query Require metadata generated automatically or semi-automatically Find shot with Camera=Stationary, Lens Actions=Zoom in


Matching Required : Matching Required Exact match query Find scene with Actor=X Similarity match query Find all triple axles by female skaters with similar launching patterns


Function : Function Location queries Locate video information Find scene with Actor=X Point to the beginning of matched videos Tracking queries Track visual quantities Track the ball through this shot Location of the ball in each of the frames in the shot


Temporal Unit Type : Temporal Unit Type Unit query Complete units of video Find films with Actor=X Subunit Query Subunits of video Find scenes with Actor=X


Requirement Summary for Video Data Model : Requirement Summary for Video Data Model A notion of time A segmented representation for time intervals A relationship between time intervals A set of descriptions associated with each time interval


ViMOD: The Video Data Model : ViMOD: The Video Data Model


Video Data Model : Video Data Model V Video Interval: [tb, te] Temporal Relations: R R=((r1,v1), (r2,v2), …, (rk,vk)) Feature Count: n Feature Type: (w0, w1,…, wn) Feature: (F1, F2, F3,…, Fn)


Segmentation Criteria (I) : Segmentation Criteria (I) The basis on which a particular interval of the video can be chosen Grouping of criteria Syntactic segmentation criteria Domain independent Semantic segmentation criteria Domain specific


Segmentation Criteria (II) : Segmentation Criteria (II)


Video Features and Video Feature Type-- Metadata : Video Features and Video Feature Type -- Metadata


Feature Classification Criteria (I) : Feature Classification Criteria (I) Content Dependence Independent: the feature is not directly available from the video data Meta features e.g. Budget of a video Dependent Data features e.g. Story Temporal Extent: Video or Image


Feature Classification Criteria (II) : Feature Classification Criteria (II) Labeling Domain model based labels Qualitative features (Q-features) Low-level domain independent models Raw features (R-features)


Type of Video Features : Type of Video Features


Meta Features : Meta Features In general, apply to a complete video Examples


Video Q-Features : Video Q-Features Has a value belonging to a finite set of labels Low level property Cinematographic properties Higher level properties Time frame, point of view…


Video Q-Feature Examples : Video Q-Feature Examples


Video R-Features : Video R-Features


Image Q-Features : Image Q-Features


Image R-Features : Image R-Features


ViMOD Architecture : ViMOD Architecture


ViMOD Architecture : ViMOD Architecture Video server Database interface Metadata store Query processor Insertion module User interface


Block Interactions : Block Interactions Data insertion operation Database Interface Metadata store Insertion module User interface Data retrieval operation Query processor User interface Database interface Metadata store


結論 : 結論 要設計一套好的DL，必須 了解數位媒體特性 了解數位媒體的應用 了解系統所要提供的檢索(Query) 設計良好的詮釋資料有助於檢索系統所能提供的功能