Motion Capture (Mocap) and Motion Data Related Technologies : Motion Capture (Mocap) and Motion Data Related Technologies Tao Yu Comp 790-058 – Robot Motion Planning Fall 2007 September 24, 2007


OUTLINE: OUTLINE Mocap Overview Representation of Motion Motion Signal Processing Motion Synthesis from Examples


What is motion capture: What is motion capture “Recording of motion for immediate or detailed analysis and playback” David J Sturman, “A Brief History of Motion Capture for Computer Character Animation”, Character Motion Systems, SIGGRAPH 94, Course 9 “The creation of a 3d representation of a live performance” Alberto Menache, Understanding Motion Capture for Computer Animation and Video Games Mocap Overview


Root of motion capture: Root of motion capture Eadweard Muybridge (1830-1904) Etienne-Jules Marey (1830-1904) Harold Edgerton (1903-1990) Mocap Overview


Eadweard Muybridge (1830-1904): Eadweard Muybridge (1830-1904) “Father of the motion picture” several cameras – successive pictures photographs of human and animal motion zoopraxiscope (zoogyroscope, zoetrope) – a device for playing still images in sequence http://www.cotianet.com.br/photo/great/Muybridge.htm http://en.wikipedia.org/wiki/Image:The_Horse_in_Motion.jpg Mocap Overview


Etienne-Jules Marey (1830-1904): Etienne-Jules Marey (1830-1904) birds photographic gun http://www.nrw-forum.de/img_ausst/img_press/Marey.jpg http://www.inrp.fr/Tecne/Acexosp/Actimage/Images/Marey2.jpg http://www.rickwisedp.com/St%20Marys/COMM%20158/images/Marey%20Photo%20Gun.jpg Mocap Overview


Harold Edgerton (1903-1990): Harold Edgerton (1903-1990) high speed and stop motion photography exposures as small as a millionth of a second electronic flash stroboscope http://www.personal.psu.edu/users/a/r/ark176/Assignment%204.htm http://www.personal.psu.edu/users/a/r/ark176/Assignment%204.htm Mocap Overview


Rotoscoping: Rotoscoping Allowed animators to trace cartoon characters over photographed frames of live performances. Invented in 1915 by Max Fleischer Koko the Clown Snow White Mocap Overview


Rotoscoping: Rotoscoping “rotoscoping can be thought of as a primitive form or precursoro to motion capture, where the motion is ‘captured’ painstakingly by hand.” - Sturman Mocap Overview


“modern” era of mocap, 1970’s-present: “modern” era of mocap, 1970’s-present more players commercial players multiple uses 70’s: development of magnetic systems 80’s: development of optical systems 90’s: mocap is hot, 00’s: mocap is used more frequently for feature films Mocap Overview


Overview: Motion capture systems: Overview: Motion capture systems Types of Mocap systems: Outside-In sources (e.g., reflective markers) on body external sensors (e.g., cameras) optical systems Inside-Out sensors on body external sources magnetic systems Inside-In sources and sensor on body mechanical systems Mocap Overview


implementation of a motion capture system: implementation of a motion capture system prosthetic acoustic magnetic optical Liverman, The Animator’s Motion Capture Guide Mocap Overview


mechanical/prosthetic capture: mechanical/prosthetic capture Inside-In external structure attached to performer structure detects changes optic mechanical Mocap Overview


mechanical/prosthetic capture: mechanical/prosthetic capture advantages computes rotations directly portable, unlimited range less expensive can capture multiple performances simultaneously no built in positional reference disadvantages external structure unwieldy cannot change the configuration, i.e., a hand capture can’t be used for an arm Mocap Overview


acoustic capture: acoustic capture Inside-Out still developing Active Markers – transmitters are attached to the performer and sequentially emit audio signal, a “click” receivers measure the time to receive the signal, triangulate and compute the position of the transmitter advantages no occlusion disadvantages cables unwieldy rate of transmission not high enough to support enough transmitters size of the capture area is limited sound reflections can reduce accuracy Mocap Overview


optical capture (passive): optical capture (passive) markers are attached to a performer passive reflective markers more on active markers later a system of cameras record the position of the markers 2-many cameras http://cgm.cs.mcgill.ca/~athens/cs644/Projects/2004/MichelLanglois-PhilippeKunzle/MoCap1.jpg Mocap Overview


optical capture (passive): optical capture (passive) advantages freedom of movement high quality capture high throughput fast sampling (200 fps at a high resolution) can capture fast motions can have a large capture space can capture many markers disadvantages occlusion, markers are can be hidden from the camera additional performers will increase occlusion may be able to add redundant cameras marker crossover, which marker are you looking at? cost $$$ extensive post processing (the marker’s have to be located and identified) Mocap Overview


major optical players: major optical players Motion Analysis http://www.motionanalysis.com/ Films Lord of the Rings King Kong Matrix Final Fantasy Games NBA Live 2004 Grand Theft Auto III Mortal Kombat 4 (Midway) ViconPeak http://www.viconpeak.com/ Films Polar Express Harry Potter and the Prisoner of Azkaban The Hulk Spider Man Games All-Star Baseball 2002 Buffy the Vampire Slayer Everquest II NHL 2K3 (Mocap by Red Eye Studio) Mocap Overview


magnetic capture: magnetic capture Outside/In receivers are attached to the performer’s body compute the position and orientation from receiver to central magnetic transmitter AC and DC systems originally developed for targeting in tactical military aircraft advantages no occlusion disadvantages cabling can interfere with movement (improvements?) capture area can be limited by transmitter metal in vicinity can interfere with system capture volume can be limited Mocap Overview


Mocap pipeline: Mocap pipeline Planning for capture Setup and calibrate system Capture performer, obtain marker positions Retarget to a character Edit/cleanup Mocap Overview


planning: planning motion capture requires serious planning anticipate how the data will be used garbage in garbage out shot list games motions need to be able to blend into one an another capture base motions and transitions which motions transition into which other transitions cycles/loops Mocap Overview


movement flowchart for games: movement flowchart for games Planning and Directing Motion Capture For Games By Melianthe Kines Gamasutra January 19, 2000 URL: http://www.gamasutra.com/features/20000119/kines_01.htm Mocap Overview


processing passive markers: processing passive markers each camera records capture session extraction: markers need to be identified in the image determines 2d position problem: occlusion, marker is not seen use more cameras markers need to be labeled which marker is which? problem: crossover, markers exchange labels may require user intervention compute 3d position: if a marker is seen by at least 2 cameras then its position in 3d space can be determined http://www.xbox.com/NR/rdonlyres/3164D1BE-C1C4-46A1-90F0-26507CF2C9BD/0/ilmnflfever2003lightscam001.jpg Mocap Overview


retargeting: the character: retargeting: the character the character is controlled by skeleton to control the skeleton, need to specify joint rotations muscles? Mocap Overview


retargeting: retargeting capture motion on performer positions of markers are recorded retarget motion on a virtual character motion is usually applied to a skeleton a skeleton is hierarchical linked joints need rotation data! need to convert positions to rotations Mocap Overview


performer→ actor → character: performer→ actor → character the actor is used to convert marker positions to rotational data markers are handles on the actor actor should have similar proportions as the performer joint rotations of the actor are applied to the character there are still issues with proportions Alias Motionbuilder: actor and markers Mocap Overview


retargeting problems: retargeting problems Mocap Overview


retargeting problems: retargeting problems Mocap Overview


Applications: Applications Entertainment Medicine Arts / Education Science / Engineering Mocap Overview


Entertainment: Live Action Films: Entertainment: Live Action Films Computer generated characters in live action films (e.g. Battle Droids and many others in Star Wars Prequels, Gullum in The Lord of the Rings, King Kong in King Kong , Davy Jones in Pirates of Caribbean) Mocap Overview


Entertainment: 3D computer animations: Entertainment: 3D computer animations Characters in computer animated files (e.g. Polar Express, Monster House) Mocap Overview


Entertainment: Video Games: Entertainment: Video Games Video games by Electronic Arts, Gremlin, id, RARE, Square, Konami, Namco, and others, (e.g. Enemy Territory, Devil May Cry) Mocap Overview


Medicine: Medicine Medicine (e.g., gait analysis, rehabilitation) Sports medicine (e.g. injury prevention, performance analyses, performance enhancement) Gait Analysis Service Mocap Overview


Arts / Education: Arts / Education Dance and theatrical performances Archiving (e.g., Marcel Marceau) OSU/ACCAD Mocap Overview


Science / Engineering: Science / Engineering Computer Science (e.g., human motion database, indexing, recognitions) Engineering (e.g., Biped robot developments) Ergonomic product design Military (e.g., field exercises, virtual instructors, and role-playing games) Mocap Overview


Aspects of the Problem: Aspects of the Problem “Gross” Body movement NOT: Appearance Models Facial animation Cloth, clothing, secondary movement Hands How to describe moving target – “Character” How to describe movement mathematically Motion Representation


Human Model: Human Model Humans are complex! Motion Representation Human motion can be understood at a very fine level of detail!


Abstractions: Abstractions Representation of complex human structure with varying degrees of simplification 206 bones, muscles, fat, organs, clothing, … Motion Representation 206 bones, complex joints 53 bones Kinematic joints


Standard simplified modelsof humans: Standard simplified models of humans Small numbers of degrees of freedom for gross motion Articulated figures Rigid pieces Sometimes stretching allowed Kinematic joints Rotations between pieces Motion Representation


An example articulated figure : An example articulated figure The skeleton: 17 joints, each with 3 degrees of freedom (DOFs) Root position (3 DOFs) and orientation (3 DOFs) Totally 57 DOFs Motion Representation


Pose representation: Pose representation Initial configurations of articulated figure (Binding pose) Usually T-Pose Rigid pieces undergo rigid transformation Hierarchical representation of the configuration Motion Representation


Computing positions: Computing positions Each joint i provides a local rotation Ri and a local translation Ti. Concatenating Ri and Ti gives local Mi, the local transform at each joint. The transformation of the point x on joint j is found by concatenating all previous transforms in the hierarchy as follows: Motion Representation


Parameterizing Rotations: Parameterizing Rotations Goal: encode rotations in a vector Rn - > “set of rotations” Give “names” to members of the set of possible rotations Many ways to do this, all flawed No perfect method Use the best one for the job Motion Representation


Goals for Parameterization: Goals for Parameterization Compact (as few variables as possible) Complete Every rotation can be represented 1-to-1 Every rotation has one value Every value has one rotation Singularity free “close” rotations are “close” in value Motion Representation


Parameterizations ofRotations: Parameterizations of Rotations Rotation Matrices Euler Angles Axis Angle formulation Unit Quaternions Motion Representation


Comparison of body representation: Comparison of body representation Motion Representation


Motion definition: Motion definition Motion is a function of time Given time, provide a pose m(t): R -> Rn Often represented as samples Sparse samples + interpolation Dense samples (at frames) Motion Representation


Preliminaries: Preliminaries A motion maps times to configurations m(t): R -> Rn Vector-valued, time-varying signal Motion Signal Processing


Challenge and solution: Challenge and solution Get a specific motion From capture, keyframe, … Specific character, action, mood, … Want something else But need to preserve original But we don’t know what to preserve Can’t characterize motion well enough Apply signal processing techniques to designing, modifying, and adapting animated motion [Bruderlin & Williams 95][Witkin & Popovic 95] Motion Signal Processing


Manipulating motion: Manipulating motion Manipulate time m(t) = m0( f(t) ) F : R -> R “time warp” Manipulate value m(t) = f(m0(t)) F : Rn-> Rn F : (R, Rn)-> Rn Motion Signal Processing


Motion blending: Motion blending “Add” two motions together Really interpolate m(t) = a m0(t) + (1-a) m1(t) Note: this is a per-frame operation Interpolation between poses Interpolating root position – Linear interpolation Interpolating joint rotation – SLERP (Spherical Linear intERPolation) Motion Signal Processing


How to use blending: How to use blending Interpolate similar motions Be sure to make time correspondence Transition between motions Time varying blend (a=0 -> 1) Over a short period of time A bad pose isn’t such a big deal Avoids discontinuities m(t) = a(t) m0(t) + (1-a(t)) m1(t) Motion Signal Processing


Blending for transition motion: Blending for transition motion Okan et al. Siggraph ’02 Motion Signal Processing


Time warping: Time warping Dynamic Time Warping (DTW): Non-linear signal matching procedure originating in field of speech recognition Finding the optimal sample correspondences between the two signals by computing the global minimum value of warping cost function Motion Signal Processing


Time warping + interpolation: Time warping + interpolation Motion Signal Processing


Displacement map: Displacement map A.K.A Motion Warping Motion Signal Processing


Displacement map: Displacement map A.K.A Motion Warping Changing the shape of a signal locally through a displacement map while maintaining continuity and preserving the global shape of the signal. Motion Signal Processing


Displacement map (cntd.): Displacement map (cntd.) Mapping procedure Motion Signal Processing


The Challenge: The Challenge High Quality, Expressive Motion Need motion capture (examples) Flexible, long-running, controllable Need synthesis Synthesis from Examples! Motion Synthesis from Examples


Idea:Put Clips Together: Idea: Put Clips Together New motions from pieces of old ones! Good news: Keeps the qualities of the original (with care) Can create long and novel “streams” (keep putting clips together) Challenges: How to connect clips? How to decide what clips to connect? Motion Synthesis from Examples


Connecting clips: transition: Connecting clips: transition Transitions between motions can be hard Simple method: Blends between aligned motions Cleanup footskate artifacts When motions are similar Believe that blending will “work” Heuristic based on geometry Not perfect method Measure and use threshold Apply threshold conservatively Motion Synthesis from Examples


Similarity Metric: Similarity Metric Factor out invariances and measure Motion Synthesis from Examples


Motion Graphs (Kovar et al. ’02): Motion Graphs (Kovar et al. ’02) Start with a database of motions, each with type and constraint information. Goal: add transitions at opportune points. Other Motion Graph-like projects elsewhere Differ in details. Motion Synthesis from Examples


Motion Graphs: Motion Graphs Idea: automatically add transitions within a motion database Quality: restrict transitions Control: build walks that meet constraints Motion Synthesis from Examples


Using motion graph: Using motion graph Any walk on the graph is a valid motion Generate walks to meet goals Random walks (screen savers) Search to meet constraints An example: Motion Synthesis from Examples 􀂄 Given a path 􀂄 Find a motion that minimizes distance 􀂄 Combinatorial optimization


Why is this good?: Why is this good? Search the graphs for motions Look ahead avoids getting stuck Cleanup motions as generated Plan “around” missing transitions Optimization gets close as possible Motion Synthesis from Examples Not OK for Interactive Apps! Need different tradeoffs