Embodied Construction Grammar in language (acquisition and) use: Embodied Construction Grammar in language (acquisition and) use Jerome Feldman (jfeldman@icsi.berkeley.edu) Computer Science Division, University of California, Berkeley, and International Computer Science Institute


State of the Art: State of the Art Limited Commercial Speech Applications transcription, simple response systems Statistical NLP for Restricted Tasks tagging, parsing, information retrieval Template-based Understanding programs expensive, brittle, inflexible, unnatural Essentially no NLU in HCI, QA Systems


Slide3: “Of all the above fields the learning of languages would be the most impressive, since it is the most human of these activities. This field seems however to depend rather too much on sense organs and locomotion to be feasible.” Alan M. Turing Intelligent Machinery (1948)


What does language do?: What does language do? “Harry walked to the cafe.” “Harry walked into the cafe.” A sentence can evoke an imagined scene and resulting inferences: Goal of action = at cafe Source = away from cafe cafe = point-like location Goal of action = inside cafe Source = outside cafe cafe = containing location


Language understanding: Language understanding Interpretation (Utterance, Situation) Linguistic knowledge Conceptual knowledge Analysis


Language understanding: analysis & simulation: Language understanding: analysis & simulation “Harry walked to the cafe.” Schema Trajector Goal walk Harry cafe Lexicon Constructicon General Knowledge Belief State Analysis Process Semantic Specification Utterance Simulation


Interpretation: x-schema simulation: Interpretation: x-schema simulation Constructions can specify which schemas and entities are involved in an event, and how they are related profile particular stages of an event set parameters of an event energy walker at goal walker=Harry goal=home Harry is walking home.


Slide8: Phonetics Semantics Pragmatics Morphology Syntax Traditional Levels of Analysis


Slide9: Phonetics Semantics Pragmatics Morphology Syntax “Harry walked into the cafe.” Utterance


Construction Grammar: Construction Grammar to block walk Form Meaning A construction is a form-meaning pair whose properties may not be strictly predictable from other constructions. (Construction Grammar, Goldberg 1995)


Form-meaning mappings for language: Form-meaning mappings for language Form phonological cues word order intonation inflection Meaning event structure sensorimotor control attention/perspective social goals... Linguistic knowledge consists of form-meaning mappings:


Slide12: Constructions as maps between relations Mover + Motion + Direction before(Motion, Direction) before(Mover, Motion) “is” + Action + “ing” before(“is”, Action) suffix(Action, “ing”) Mover + Motion before(Mover, Motion) Form Meaning ProgressiveAction aspect(Action, ongoing) MotionEvent mover(Motion, Mover) DirectedMotionEvent direction(Motion, Direction) mover(Motion, Mover) Complex constructions are mappings between relations in form and relations in meaning.


Embodied Construction Grammar(Bergen and Chang 2002): Embodied Construction Grammar (Bergen and Chang 2002) Embodied representations active perceptual and motor schemas situational and discourse context Construction Grammar Linguistic units relate form and meaning/function. Both constituency and (lexical) dependencies allowed. Constraint-based (Unification) based on feature structures (as in HPSG) Diverse factors can flexibly interact.


Representing image schemas: schema Container roles interior exterior portal boundary Representing image schemas Interior Exterior Boundary Portal Source Path Goal Trajector These are abstractions over sensorimotor experiences. schema Source-Path-Goal roles source path goal trajector schema name role name


Inference and Conceptual Schemas: Inference and Conceptual Schemas Hypothesis: Linguistic input is converted into a mental simulation based on bodily-grounded structures. Components: Semantic schemas image schemas and executing schemas are abstractions over neurally grounded perceptual and motor representations Linguistic units lexical and phrasal construction representations invoke schemas, in part through metaphor Inference links these structures and provides parameters for a simulation engine


Early ExampleUnderstanding News Stories: Early Example Understanding News Stories France fell into recession. Pulled out by Germany In1991, India set out on a path of liberalization. The Government started to loosen its stranglehold on business and removed obstacles to international trade. Now the Government is stumbling in implementing the liberalization plan.


Task: Task Interpret simple discourse fragments/blurbs France fell into recession. Pulled out by Germany Economy moving at the pace of a Clinton jog. US Economy on the verge of falling back into recession after moving forward on an anemic recovery. Indian Government stumbling in implementing Liberalization plan. Moving forward on all fronts, we are going to be ongoing and relentless as we tighten the net of justice. The Government is taking bold new steps. We are loosening the stranglehold on business, slashing tariffs and removing obstacles to international trade.


I/O as Feature Structures: I/O as Feature Structures Indian Government stumbling in implementing liberalization plan


Language understanding: analysis & simulation: Language understanding: analysis & simulation “Harry walked into the cafe.” Analysis Process Semantic Specification Utterance Constructions General Knowledge Belief State Simulation construction WALKED form selff.phon  [wakt] meaning : Walk-Action constraints selfm.time before Context.speech-time selfm..aspect  encapsulated


Slide21: Embodied Construction Grammar provides formal tools for linguistic description and analysis motivated largely by cognitive/functional concerns. Allows precise specifications of structures/processes involved in acquisition of early constructions Embodied constructions (structured maps between form and meaning); lexically specific and more general Usage-based processes of learning new constructions to account for co-occurring utterance-situation pairs Bridge to detailed psycholinguistic and neural imaging experiments


Formal Cognitive Linguistics: Formal Cognitive Linguistics Schemas and frames Image schemas, force dynamics, executing schemas… Constructions Lexical, grammatical, morphological, gestural… Maps Metaphor, metonymy, mental space maps… Mental spaces Discourse, hypothetical, counterfactual…


Embodied constructions: Embodied constructions construction HARRY form : [hEriy] meaning : Harry construction CAFE form : [khaefej] meaning : Cafe Harry cafe Notation Form Meaning Constructions have form and meaning poles that are subject to type constraints.


Schema Formalism: Schema Formalism SCHEMA SUBCASE OF EVOKES AS ROLES : CONSTRAINTS :: :: |


A Simple Example: A Simple Example   SCHEMA hypotenuse SUBCASE OF line-segment EVOKES right-triangle AS rt ROLES Comment inherited from line-segment CONSTRAINTS SELF rt.long-side


Source-Path-Goal: Source-Path-Goal   SCHEMA: spg ROLES: source: Place path: Directed Curve goal: Place trajector: Entity


Translational Motion: Translational Motion   SCHEMA translational motion SUBCASE OF motion EVOKES spg AS s ROLES mover s.trajector source s.source goal s.goal CONSTRAINTS before:: mover.location source after:: mover.location goal


Construction Formalism: Construction Formalism CONSTRUCTION SUBCASE OF CONSTRUCTIONAL EVOKES AS CONSTITUENTS : CONSTRAINTS // as in SCHEMAs FORM ELEMENTS CONSTRAINTS // as in SCHEMAs MEANING // as in SCHEMAs


Representing constructions: TO: The meaning pole may evoke schemas (e.g., image schemas) with a local alias. The meaning pole may include constraints on the schemas (e.g., identification constraints «). Representing constructions: TO local alias identification constraint


The INTO construction : TO vs. INTO: INTO adds a Container schema and appropriate bindings. The INTO construction construction INTO form selff.phon  [Inthuw] meaning evokes Trajector-Landmark as tl Source-Path-Goal as spg Container as cont constraints: tl.trajector « spg.trajector tl.landmark « cont cont.interior « spg.goal cont.exterior « spg.source


Constructions with constituents:The SPATIAL-PREDICATION construction: construction SPATIAL-PREDICATION constructional constituents sp : Trajector-Landmark lm : Thing form spf before lmf meaning spm.landmark « lmm Constructions with constituents: The SPATIAL-PREDICATION construction Constructions may also specify constructional constituents and impose form and meaning constraints on them: order constraints identification constraints order constraint local alias identification constraint


Grammatical Construction Example: Grammatical Construction Example CONSTRUCTION Spatial-PP SUBCASE OF Phrase CONSTRUCTIONAL CONSTITUENTS rel: Spatial-Preposition lm: Referring-Exp CONSTRAINTS rel.case lm.case FORM rel lm


The DIRECTED-MOTION construction: The DIRECTED-MOTION construction construction DIRECTED-MOTION constructional constituents mover : Thing motion : Motion-Process direction : Source-Path-Goal form moverf before motionf motionf before directionf meaning evokes Motion-Event as m m.mover « moverm m.motion « motionm m.path « directionm directionm.trajector « moverm motionm.mover « moverm


Semantic specification: Semantic specification The analysis process produces a semantic specification that includes image-schematic, motor control and conceptual structures provides parameters for a mental simulation


Language Understanding Process: Language Understanding Process


Constructional analysis: Constructional analysis


Semantic Specification: Semantic Specification


Language understanding: analysis & simulation: Language understanding: analysis & simulation “Harry walked into the cafe.” Analysis Process Semantic Specification Utterance Constructions General Knowledge Belief State Simulation construction WALKED form selff.phon  [wakt] meaning : Walk-Action constraints selfm.time before Context.speech-time selfm..aspect  encapsulated


Simulation-based sense disambiguation: Simulation-based sense disambiguation The scientist walked into the laboratory. The scientist walked into the wall. Ease of construing nominal as a CONTAINER determines what sense of into is appropriate: CONTAINER sense CONTACT sense


Simulation-based inference: Simulation-based inference The teacher drifted into the house. The smoke drifted into the house. Detailed inferences can result from simulation. Image-schematic content of prepositions must fit with properties of other elements of sentence. Final location of Trajector = inside cafe Portal = door Final location of Trajector = inside (possibly throughout) cafe Portal = door/window


World knowledge informs simulation: World knowledge informs simulation Physical knowledge of how people and gases interact with houses determines: Relation between Trajector and Interior The smoke drifted into the house and filled it. ?The teacher drifted into the house and filled it. Portal for motion across Boundary The smoke drifted into the house because the window had been left open. ?The teacher drifted into the house because the window had been left open.


ECG applications: ECG applications Grammar (Note: Theme Session on ECG at ICLC 2003, La Rioja) Spatial relations/events (Bergen & Chang 1999; Bretones et al. In press) Verbal morphology (Gurevich 2003, Bergen ms.) Reference: measure phrases (Dodge and Wright 2002), construal resolution (Porzel & Bryant 2003), reflexive pronouns (Sanders 2003) Semantic representations / inference Aspectual inference (Narayanan 1997; Chang, Gildea & Narayanan 1998) Perspective / frames (Chang, Narayanan & Petruck 2002) Metaphorical inference (Narayanan 1997, 1999) Simulation semantics (Narayanan 1997, 1999) Language acquisition Lexical acquisition (Regier 1996, Bailey 1997) Multi-word constructions (Chang & Maia 2001)


Getting From the Utterance to the SemSpecJohno Bryant: Getting From the Utterance to the SemSpec Johno Bryant Need a grammar formalism Embodied Construction Grammar (Bergen & Chang 2002) Need new models for language analysis Traditional methods too limited Traditional methods also don’t get enough leverage out of the semantics.


Embodied Construction Grammar: Embodied Construction Grammar Semantic Freedom Designed to be symbiotic with cognitive approaches to meaning More expressive semantic operators than traditional grammar formalisms Form Freedom Free word order, over-lapping constituency Precise enough to be implemented


Traditional Parsing Methods Fall Short: Traditional Parsing Methods Fall Short PSG parsers too strict Constructions not allowed to leave constituent order unspecified Traditional way of dealing with incomplete analyses is ad-hoc Making sense of incomplete analyses is important when an application must deal with “ill-formed” input (For example, modeling language learning) Traditional unification grammar can’t handle ECG’s deep semantic operators.


Our Analyzer: Our Analyzer Replaces the FSMs used in traditional chunking (Abney 96) with much more powerful machines capable of backtracking called construction recognizers Arranges these recognizers into levels just like in Abney’s work But uses a chart to deal with ambiguity


Our Analyzer (cont’d): Our Analyzer (cont’d) Uses specialized feature structures to deal with ECG’s novel semantic operators Supports a heuristic evaluation metric for finding the “right” analysis Puts partial analyses together when no complete analyses are available The analyzer was designed under the assumption that the grammar won’t cover every meaningful utterance encountered by the system.


System Architecture: System Architecture Learner Grammar/Utterance Chunk Chart Ranked Analyses


The Levels: The Levels The analyzer puts the recognizer on the level assigned by the grammar writer. Assigned level should be greater than or equal to the levels of the construction’s constituents. The analyzer runs all the recognizers on level 1, then level 2, etc. until no more levels. Recognizers on the same level can be mutually recursive.


Recognizers: Recognizers Each Construction is turned into a recognizer Recognizer = active representation seeks form elements/constituents when initiated Unites grammar and process - grammar isn’t just a static piece of knowledge in this model. Checks both form and semantic constraints Contains an internal representation of both the semantics and the form A graph data structure used to represent the form and a feature structure representation for the meaning.


Recognizer Example: Recognizer Example Path Patient Action Agent Mary kicked the ball into the net. This is the initial Constituent Graph for caused-motion.


Recognizer Example: Recognizer Example Construct: Caused-Motion Constituent: Agent Constituent: Action Constituent: Patient Constituent: Path The initial constructional tree for the instance of Caused-Motion that we are trying to create.


Recognizer Example: Recognizer Example


Recognizer Example: Recognizer Example processed Mary kicked the ball into the net. Path Patient Action Agent A node filled with gray is removed.


Recognizer Example: Recognizer Example Construct: Caused-Motion Constituent: Action Constituent: Patient Constituent: Path RefExp: Mary Mary kicked the ball into the net.


Recognizer Example: Recognizer Example


Recognizer Example: Recognizer Example processed Mary kicked the ball into the net. Path Patient Action Agent


Recognizer Example: Recognizer Example Construct: Caused-Motion Verb: kicked Constituent: Patient Constituent: Path RefExp: Mary Mary kicked the ball into the net.


Recognizer Example: Recognizer Example


Recognizer Example: Recognizer Example processed Mary kicked the ball into the net. Path Patient Action Agent According to the Constituent Graph, The next constituent can either be the Patient or the Path.


Recognizer Example: Recognizer Example processed Mary kicked the ball into the net. Path Patient Action Agent


Recognizer Example: Recognizer Example Construct: Caused-Motion Verb: kicked RefExp: Det Noun Constituent: Path RefExp: Mary Mary kicked the ball into the net. Noun Det


Recognizer Example: Recognizer Example


Recognizer Example: Recognizer Example processed Mary kicked the ball into the net. Path Patient Action Agent


Recognizer Example: Recognizer Example Construct: Caused-Motion Verb: kicked RefExp: Det Noun Spatial-Pred: Prep RefExp RefExp: Mary Mary kicked the ball into the net. Noun Det Noun Det Prep RefExp


Recognizer Example: Recognizer Example


Resulting SemSpec: Scene = Caused-Motion Agent = Mary Action = Kick Patient = Path.Trajector = The Ball Path = Into the net Path.Goal = The net After analyzing the sentence, the following identities are asserted in the resulting SemSpec: Resulting SemSpec


Progress: Progress The analyzer (as described so far) is already being put to use in Chang’s thesis work. The levels are well-suited to incremental learning. Syntactic robustness important for generating partial analyses with poor coverage It will also be used this semester for producing SemSpecs for Narayanan’s enactment engine. Put the deep semantics towards parameterizing x-schemas


Chunking: Chunking L0 D N P D N N V-tns Pron Aux V-ing L1 ____NP P_______NP VP NP ______VP L2 ____NP _________PP VP NP ______VP L3 ________________________S_____________S Cite/description


Construction Recognizers: Construction Recognizers You want to put a cloth on your hand ? Like Abney: Unlike Abney: One recognizer per rule Bottom up and level-based Check form and semantics More powerful/slower than FSMs


Chunk Chart: Chunk Chart Interface between chunking and structure merging Each edge is linked to its corresponding semantics. You want to put a cloth on your hand ?


Combining Partial Parses: Combining Partial Parses Prefer an analysis that spans the input utterance with the minimum number of chunks. When no spanning analysis exists, however, we still have a chart full of semantic chunks. The system tries to build a coherent analysis out of these semantics chunks. This is where structure merging comes in.


Structure Merging: Structure Merging Closely related to abductive inferential mechanisms like abduction (Hobbs) Unify compatible structures (find fillers for frame roles) Intuition: Unify structures that would have been co-indexed had the missing construction been defined. There are many possible ways to merge structures. In fact, there are an exponential number of ways to merge structures (NP Hard). But using heuristics cuts down the search space.


Structure Merging Example: Structure Merging Example Utterance:You used to hate to have the bib put on . [Addressee < Animate] Before Merging: After Merging:


Semantic Density: Semantic Density Semantic density is a simple heuristic to choose between competing analyses. Density of an analysis = (filled roles) / (total roles) The system prefers higher density analyses because a higher density suggests that more frame roles are filled in than in competing analyses. Extremely simple / useful? but it certainly can be improved upon.


Summary: ECG: Summary: ECG Linguistic constructions are tied to a model of simulated action and perception Embedded in a theory of language processing Constrains theory to be usable Frees structures to be just structures, used in processing Precise, computationally usable formalism Practical computational applications, like MT and NLU Testing of functionality, e.g. language learning A shared theory and formalism for different cognitive mechanisms Constructions, metaphor, mental spaces, etc.


Issues in Scaling up to Language : Issues in Scaling up to Language Knowledge Lexicon (FrameNet) Constructicon (ECG) Maps (Metaphors, Metonymies) (MetaNet) Conceptual Relations (Image Schemas, X-schemas) Computation Representation (ECG) expressiveness, modularity, compositionality Inference (Simulation Semantics) tractable, distributed, probabilistic concurrent, context-sensitive


The Buy schema: The Buy schema schema Buy subcase of Action evokes Commercial-Transaction as ct roles self « ct.nucleus buyer « actor « ct.customer « ct.agent goods « undergoer « ct.goods


The Sell schema: The Sell schema schema Sell subcase of Action evokes Commercial-Transaction as ct roles self « ct.nucleus seller « actor « ct.vendor « ct.agent goods « undergoer « ct.goods


Extending Inferential Capabilities: Extending Inferential Capabilities Given the formalization of the conceptual schemas How to use them for inferencing? Earlier pilot systems Used metaphor and Bayesian belief networks Successfully construed certain inferences But don’t scale New approach Probabilistic relational models Support an open ontology


Semantic Web: Semantic Web The World Wide Web (WWW) contains a large and expanding information base. HTML is accessible to humans but does not formally describe data in a machine interpretable form. XML remedies this by allowing for the use of tags to describe data (ex. disambiguating crawl) Ontologies are useful to describe objects and their inter-relationships. DAML+OIL (http://www.daml.org) is an markup language based on XML and RDF that is grounded in description logic and is designed to allow for ontology development, transfer, and use on the web.


The ICSI/BerkeleyNeural Theory of Language Project: The ICSI/Berkeley Neural Theory of Language Project ECG


Probabilistic Relation Inference: Probabilistic Relation Inference Scalable Representation of States, domain knowledge, ontologies (Avi Pfeffer 2000, Koller et al. 2001) Merges relational database technolgy with Probabilistic reasoning based on Graphical Models. Domain entities and relational entities Inter-entity relations are probabilistic functions Can capture complex dependencies with both simple and composite slot (chains). Inference exploits structure of the domain


Status of PRMs: Status of PRMs Summer Project Build the basic PRM codebase/infrastructure Fall Project Design Coordinated PRM (CPRM) Build Interface for testing Spring/Summer 03 Implement CPRM to replace Pilot System DBN Test CPRM for QA Related Work Probabilistic OWL (PrOWL) Probabilistic FrameNet


Articulating Projects: Articulating Projects FrameNet – NSF (with Colorado, USD) SmartKom – International Consortium EDU – European Media Lab Acquaint – ARDA (with SIMS, Stanford)


Conclusion: Conclusion NLU is essential to large, open domain QA. Much of the web in unstructured data Substantial Progress in Enabling Technologies Knowledge Representation/Inference Techniques Active Knowledge – X-schemas, Simulation Semantics Dealing With Uncertainty – PRM’s Combining Statistics and Structure. Conceptual Relations – Schemas, Metaphor, ECG Scaling Up CYC, Wordnet, Term-bases FrameNet, Semantic Web, MetaNet Open Source The goal of NLU can be realized, perhaps! Anyway, it’s time to try again.