lecture4

Lecture 4: Motion CaptureJinxiang Chai : Lecture 4: Motion Capture Jinxiang Chai


Slide2 : Outline Mocap history Mocap technologies Mocap pipeline Mocap Data Mocap Challenges


Slide3 : Motion Capture “ …recording of motion for immediate or delayed analysis or playback…” David J. Sturman “…is a technique of digitally recording movements for entertainment, sports, and medical applications.” - Wikipedia


Slide4 : History of Motion Capture Eadweard Muybridge (1830-1904) first person to photograph movement sequences


Slide5 : History of Motion Capture Eadweard Muybridge (1830-1904) first person to photograph movement sequences the flying horse Sequence of a horse jumping (courtesy of E. Muybridge)


Slide6 : History of Motion Capture Eadweard Muybridge (1830-1904) first person to photograph movement sequences the flying horse zoopaxiscope Jumping horse (courtesy of E. Muybridge)


Slide7 : History of Motion Capture Eadweard Muybridge (1830-1904) first person to photograph movement sequences the flying horse zoopaxiscope animal locomotion (20k pictures about men, women, children, animals, and birds). Woman walking downstairs (courtesy of E. Muybridge)


Slide8 : Rotoscope Allow animators to trace cartoon character over photographed frames of live performances invented by Max Fleischer in 1915


Slide9 : Rotoscope Allow animators to trace cartoon character over photographed frames of live performances invented by Max Fleischer in 1915 2D manual motion capture A horse animated by rotoscoping from Muybridge’s photos


Slide10 : Allow animators to trace cartoon character over photographed frames of live performances invented by Max Fleischer in 1915 2D manual capture the first cartoon character to be rotoscoped -- “Koko the clown” the human character animation -- snow white and her prince (Walt Disney, 1937) Rotoscope


Slide11 : “3D Rotoscoping”: measuring 3D positions, orientations, velocities or accelerations Current motion capture systems Electromagnetic Electromechanical Fiber optic Optical Current Motion Capture Technologies


Slide12 : Each sensor record 3D position and orientation Each sensor placed on joints of moving object Full-body motion capture needs at least 15 sensors Popular system: http://www.ascension-tech.com/ Electromagnetic Mocap


Slide13 : See video demo! Electromagnetic Mocap


Slide14 : Pros measure 3D position and orientation no occlusion problems can capture multiple subjects simultaneously Cons magnetic perturbations (metal) small capture volume cannot capture deformation (facial expression) hard to capture small bone movement (finger motion) not as accurate as optical mocap system Electromagnetic Mocap


Slide15 : Each sensor measures 3D orientation Electromechanical Mocap


Slide16 : Each sensor measures 3D orientation Each sensor placed on joints of moving object Full-body motion capture needs at least 15 sensors Popular systems: http://www.xsens.com/ Electromechanical Mocap


Slide17 : See video demo! Electromechanical Mocap


Slide18 : Pros measure 3D orientation no occlusion problems can capture multiple subjects simultaneously large capture volume Cons getting 3D position info is not easy cannot capture deformation (facial expression) hard to capture small bone movement (finger motion) not as accurate as optical mocap system Electromechanical Mocap


Slide19 : measures 3D position and orientation of entire tape Binding the tape to the body Popular systems: http://www.measurand.com/ Fiber Optic Mocap


Slide20 : See video demo! Fiber Optic Mocap


Slide21 : Pros measure 3D orientation and position no occlusion problems can capture multiple subjects simultaneously go anywhere mocap system can capture hand/finger motion Cons intrusive capture cannot capture deformation (facial expression) not as accurate as optical mocap system Fiber Optic Mocap


Slide22 : Multiple calibrated cameras (>=8) digitize different views of performance Wears retro-reflective markers Accurately measures 3D positions of markers Optical Mocap


Slide23 : See video demo! Optical Mocap Vicon mocap system: http://www.vicon.com


Slide24 : Pros measure 3D position data also orientation the most accurate capture method very high frame rate can capture very detailed motion (body, finger, facial deformation, etc.) Cons has occlusion problems hard to capture interactions among multiple ppl limited capture volume Optical Mocap


Slide25 : Mocap Pipeline Optical Mocap pipeline Planning Calibration Data processing


Slide26 : Planning Character/prop set up - character skeleton topology (bones/joints number, Dofs for each bone) - location and size of props Marker Setup - the number of markers - where to place markers


Slide27 : Calibration Camera Calibration: determine the location and orientation of each camera determine camera parameters (e.g. focal length) Subject calibration - determine the skeleton size of actors/actresses (.asf file) - relative marker positions in terms of bones - determine the size and location of props


Slide28 : Data Process 3D marker positions (.c3d file) Fill in missing data Mocap data correspondence and labeling Filter mocap data Inverse Kinematics Joint angle data (.amc file) Complete 3D marker trajectories (.c3d file)


Slide29 : Vicon Motion Capture Data Files Each sequence of human motion data contains two files: Skeleton file (.asf): Specify the skeleton model of character Motion data file (.amc): Specify the joint angle values over the frame/time Both files are generated by Vicon softwares


Slide30 : Skeleton File .asf file individual bone information (number of dofs, size, direction, joint limits) bone hierarchy/connections


Slide31 : For each bone begin id bone_id                  //Unique id for each bone name bone_name        //Unique name for each bone direction dX dY dZ    //Vector describing direction of the bone in world coor. system length 7.01722           //Length of the bone axis 0 0 20 XYZ         //Rotation of local coordinate system for                                    //this bone relative to the world coordinate                                    //system. In .AMC file the rotation angles                                     //for this bone for each time frame will be                                    //defined relative to this local coordinate                                     //system dof rx ry rz                //Degrees of freedom for this bone. limits (-160.0 20.0)             (-70.0 70.0)             (-60.0 70.0) end Skeleton File: Bone Info


Slide32 : For each bone begin id 2                 name lfemur        direction 0.34 -0.93 0    length 7.01722           axis 0 0 20 XYZ         dof rx ry rz                limits (-160.0 20.0)             (-70.0 70.0)             (-60.0 70.0) end Skeleton File: Bone Info


Slide33 : For each bone begin id 2                 name lfemur        direction 0.34 -0.93 0    length 7.01722           axis 0 0 20 XYZ         dof rx ry rz                limits (-160.0 20.0)             (-70.0 70.0)             (-60.0 70.0) end Skeleton File: Bone Info begin id 3                 name ltibia        direction 0.34 -0.93 0    length 7.2138           axis 0 0 20 XYZ         dof rx            limits (-10.0 170.0) end


Slide34 : :hierarchy begin root lhipjoint rhipjoint lowerback lhipjoint lfemur lfemur ltibia ltibia lfoot lfoot ltoes rhipjoint rfemur rfemur rtibia rtibia rfoot rfoot rtoes lowerback upperback upperback thorax thorax lowerneck lclavicle rclavicle … end Skeleton File: Hierarchy/Bone Connections


Slide35 : :hierarchy begin root lhipjoint rhipjoint lowerback lhipjoint lfemur lfemur ltibia ltibia lfoot lfoot ltoes rhipjoint rfemur rfemur rtibia rtibia rfoot rfoot rtoes lowerback upperback upperback thorax thorax lowerneck lclavicle rclavicle … end Skeleton File: Hierarchy/Bone Connections root rhipjoint lhipjoint lowerback


Slide36 : :hierarchy begin root lhipjoint rhipjoint lowerback lhipjoint lfemur lfemur ltibia ltibia lfoot lfoot ltoes rhipjoint rfemur rfemur rtibia rtibia rfoot rfoot rtoes lowerback upperback upperback thorax thorax lowerneck lclavicle rclavicle … end Skeleton File: Hierarchy/Bone Connections root rhipjoint lhipjoint lowerback femur


Slide37 : :hierarchy begin root lhipjoint rhipjoint lowerback lhipjoint lfemur lfemur ltibia ltibia lfoot lfoot ltoes rhipjoint rfemur rfemur rtibia rtibia rfoot rfoot rtoes lowerback upperback upperback thorax thorax lowerneck lclavicle rclavicle … end Skeleton File: Hierarchy/Bone Connections root rhipjoint lhipjoint lowerback femur


Slide38 : :hierarchy begin root lhipjoint rhipjoint lowerback lhipjoint lfemur lfemur ltibia ltibia lfoot lfoot ltoes rhipjoint rfemur rfemur rtibia rtibia rfoot rfoot rtoes lowerback upperback upperback thorax thorax lowerneck lclavicle rclavicle … end Skeleton File: Hierarchy/Bone Connections root rhipjoint lhipjoint lowerback femur


Slide39 : :hierarchy begin root lhipjoint rhipjoint lowerback lhipjoint lfemur lfemur ltibia ltibia lfoot lfoot ltoes rhipjoint rfemur rfemur rtibia rtibia rfoot rfoot rtoes lowerback upperback upperback thorax thorax lowerneck lclavicle rclavicle … end Skeleton File: Hierarchy/Bone Connections root rhipjoint lhipjoint lowerback femur


Slide40 : Skeleton File .asf file individual bone information (number of dofs, size, direction, joint limits) bone hierarchy/connections


Slide41 : i // frame number root 2.36756 16.4521 12.3335 -165.118 31.188 -179.889 // root position and orientation lowerback -17.2981 -0.243065 -1.41128 // joint angles for lowerback joint upperback 0.421503 -0.161394 2.20925 // joint angles for thorax joint thorax 10.2185 -0.176777 3.1832 lowerneck -15.0172 -5.84786 -7.55529 upperneck 30.0554 -3.19622 -4.68899 head 12.6247 -2.35554 -0.876544 rclavicle 4.77083e-014 -3.02153e-014 rhumerus -23.3927 30.8588 -91.7324 rradius 108.098 rwrist -35.4375 rhand -5.30059 11.2226 rfingers 7.12502 rthumb 20.5046 -17.7147 lclavicle 4.77083e-014 -3.02153e-014 lhumerus -35.2156 -19.5059 100.612 Motion Data File (.amc) For each frame


Slide42 : i // frame number root 2.36756 16.4521 12.3335 -165.118 31.188 -179.889 // root position and orientation lowerback -17.2981 -0.243065 -1.41128 // joint angles for lowerback joint upperback 0.421503 -0.161394 2.20925 // joint angles for thorax joint thorax 10.2185 -0.176777 3.1832 lowerneck -15.0172 -5.84786 -7.55529 upperneck 30.0554 -3.19622 -4.68899 head 12.6247 -2.35554 -0.876544 rclavicle 4.77083e-014 -3.02153e-014 rhumerus -23.3927 30.8588 -91.7324 rradius 108.098 rwrist -35.4375 rhand -5.30059 11.2226 rfingers 7.12502 rthumb 20.5046 -17.7147 lclavicle 4.77083e-014 -3.02153e-014 lhumerus -35.2156 -19.5059 100.612 Motion Data File (.amc) Motion Data File (.amc) For each frame


Slide43 : Mocap Challenges Capture human and animal motion with high fidelity, resolution, and consistency: - human body, face, hand, skin deformation - animal motion, etc. However, not appropriate for capturing - secondary motion like hair and cloth - lots of animal motions like fish - natural phenomenon (water flowing, fire, etc) - crowd behavior, etc.


Slide44 : Next Four Lectures: Mocap Data Processing 2. Motion warping, Siggraph95 3. Retargetting Motion to New Characters, Siggraph98 4. Interactive Motion Editing, Siggraph99 1. The Process of Motion Capture


Slide45 : Next Four Lectures: Mocap Data Processing 6. Style Translation for Human Motion, siggraph05 7. Action Synposis, Siggraph05 8. Compression of Motion Capture Databases, Siggraph06 5. Expression Cloning, siggraph01