I2RP DIR2003

Intelligent Information Retrieval and Presentation with Multimedia Databases: 

Intelligent Information Retrieval and Presentation with Multimedia Databases Floris Wiesman (IKAT/UM) Stefano Bocconi (CWI) Boban Arsenijevic (ULCL/UL) Yulia Bachvarova (CWI) Nico Roos (IKAT/UM) Lambert Schomaker (AI/RUG)

Outline: 

Outline From search to presentation I2RP architecture Query processing & presentation generation Natural languange generation Ontology mappings Conclusions

From search to presentation: 

From search to presentation Standard (multimedia) IR: result of query is ordered list Question-answering system: result of query is answer Our approach: result of query is multimedia presentation containing the answer

I2RP architecture: 

I2RP architecture Presentation generator

Query processing and presentation generation: 

Query processing and presentation generation Currently simple ‘closed’ queries Answers are determined from semantic network Rules determine which information to present Rules determine which modalities to use

Semantic network:Rembrandt’s world: 

Semantic network: Rembrandt’s world Chiaroscuro Caravaggio Caravaggist Italy Pieter Lastman Bible Mythology Rembrandt Saskia Rubens teacher teacher Bol is_founded_by paints paints inspired_by compared_to belongs studies paints paints studies uses Bent Birds Jan Lievens teacher works_with operated_in Portraits History Paintings paints paints paints Night Watch has_genre Hendrickje is_married has_relation Maria Trip has_genre Prophetess Anna has_genre

Metadata: 

Metadata Search in databases by metadata, no content-based retrieval Metadata has to be such that: The system can find what it looks for The system can assemble the retrieved information in a meaningful way The presentation really means what was intended: a new context is created

The information context: 

The information context Information items do not exist standalone, they have a context A presentation needs to combine the retrieved information items in a new context Information sources, from most structured to less structured: Multimedia Databases (e.g., ARIA) Digital Library (e.g, Open Archives Initiative) The Internet

Narrative as a context: 

Narrative as a context Example of narrative structure (Greimas): 6 Actants: Subject, Object, Helper, Opponent, Destinateur, Receiver 4 Narrative Units: Contract, Competence, Performance, Sanction Every character in the story plays a role in the narrative (identified by rules) e.g. Artist biography: roles are main character, family members, teachers, collaborators, students Structuring according to role e.g. family members are grouped in private life section

Multimedia presentation: 

Multimedia presentation

Natural-language generation: 

Natural-language generation Starts from semantic level Semantic representation may contain: Participating concepts Event structure Temporal organization Quantification Relevant discourse functions Transforms selected meaning to natural language sentence

Learning ontology mappings: 

Learning ontology mappings The task: Establish a mapping of concepts in ontology 2 to concepts in ontology 1 Two steps: Establish joint attention: which are common instances of the ontologies? Establish mapping: which operations map the concepts best? (wrt joint attention)

Two example ontologies: 

Two example ontologies Ontology 1 Object Title = Self portrait as St. Paul Artist = Rembrandt Harmensz. van Rijn Materials = oil on canvas Date = 1661-1662 Ontology 2 Artefact Title = Self portrait as the apostle St. Paul Creator family name = Rijn given name = Rembrandt Harmensz. other name = van Material = oil on canvas Period start = 1661 end = 1662 Mapping can be made by copying, splitting, and merging leaf concepts

Establishing joint attention: 

Establishing joint attention do: agent 1: sends instance of concept to agent 2 agent 2: returns instance with highest proportion of words in common until enough common instances found above threshold  result is joint attention set: concepts of agent 1 with instances known to agent 1 & 2

Mapping of concepts: 

Mapping of concepts Mapping of concepts consists of operators: field n: copy leaf concept merge s: merge leaf concepts using separator s split s: split leaf concept at separator s first: copy first part of split last: copy last part of split Separator: none, space, colon, semicolon (,TC) Example: artefact.period.end  field object.date, split(-), last

Establishing a mapping: 

Establishing a mapping Search space consists of all possible mappings Value of a mapping: number of correct mappings on joint attention set Mapping with highest value wins Search is guided by proportion of words that instances have in common Search space is reduced by ignoring mappings between leaf concepts with a low proportion of common words

Conclusions: 

Conclusions We have shown approach for IR from multimedia databases using: Knowledge-based query augmentation Combination of IR results in a single multimedia presentation Natural language generation Automatic ontology mapping Parts are realized as prototypes; to be combined in one system