Presentation Transcript
Being There:Capturing and Experiencing aSense of Place : Being There: Capturing and Experiencing a Sense of Place Richard Szeliski
Microsoft Research
Symposium on Computational
Photography and Video
Early art: events : Early art: events Lascaux
Early art: events : Early art: events
Early art: events : Early art: events Bayeux Tapestries
Renaissance art: events : Renaissance art: events Da Vinci
Baroque art: portraits : Baroque art: portraits Rembrandt
Baroque art: landscapes : Baroque art: landscapes Rembrandt
Classical art: landscapes : Classical art: landscapes Corot
Impressionism: landscapes : Impressionism: landscapes Monet
Modern art: multi-perspective : Modern art: multi-perspective Picasso
Impressionism: landscapes : Impressionism: landscapes Monet
Photography: events : Photography: events Henri Cartier-Bresson
Photography: landscapes : Photography: landscapes Ansel Adams
Photography: landscapes : Photography: landscapes Galen Rowell
Photography: landscapes : Photography: landscapes Art Wolfe
Interactive Visual Media : Interactive Visual Media Branching storytelling Games:
problem solving, skill acquisition
sense of place, navigation
Real-world “immersive” media : Real-world “immersive” media Recapturing the sense of place
QTVR: the first real breakthrough in modeling from reality?
Fairly universal for high-end home and car sales, virtual tourism
Limitations: mostly static, discrete jumps
Increasing realism : Increasing realism Better:
field of view and resolution Demo: Space Needle
dynamic range
Additional cues/modalities:
motion / movement Demo: Deception Falls
sound
VR Tools : VR Tools
Fully automated stitch discovery : Fully automated stitch discovery Demo
Image stitching — open issues : Image stitching — open issues Fully automated assembly (no ordering)
Full 2D stitching (multiple rows)
Double image fix-up (de-ghosting)
Merging different exposures
Automated grouping/clustering (detection)
HDR: merging exposures : HDR: merging exposures Inputs Tonemapped output:
no motion compensation or consistency check
HDR: merging exposures : Inputs Tonemapped output:
global+local motion compensation HDR: merging exposures
Video Textures : Video Textures How can we turn a short video clip into continuous video?
Enhance sense of liveness
Use in games and presentations
Video Textures : Video Textures Find cyclic structure in the video
Play frames with random shuffle
Optional region-based analysis
Smooth over discontinuities (morph)
Animating Stills : Animating Stills What if we only have a single beautiful photo (or painting)?
Can we add so liveness to a photo/still?
Animating Stills : Animating Stills
Increasing “immersion” : Increasing “immersion” Add continuous movement
DemoBellevue Botanical Garden : Demo Bellevue Botanical Garden
DemoHDR Home Walkthrough : Demo HDR Home Walkthrough
Video-based walkthroughs : Video-based walkthroughs Add continuous movement
How did we do this?
Is this 3D? 2D? Graphics? Video?
Dimensionality and storage?
Limits of Video-Based Walkthroughs : Limits of Video-Based Walkthroughs Video-based walkthroughs are “rail experiences” of a space-time slice
How do we break the bounds?
Can’t move from side to side
If there is motion in the scene, don’t see it
Capturing all points of view everywhere in space at all times is impractical
A Practical 3D Video Camera : A Practical 3D Video Camera Working volume?
Walls of a room: Virtualized Reality
2D “window”: LightField Array
1D “rail”: Virtual Viewpoint Video
“3D” video : “3D” video
Virtual Viewpoint Video : Virtual Viewpoint Video Capture multiple synchronized video streams
Key to view interpolation: Geometry : Key to view interpolation: Geometry Stereo Camera 1 Camera 2 Image 1 Image 2 Virtual Camera
Don’t match pixels – match segments : Don’t match pixels – match segments Segments contain more information, so they’re easier to match.
Iteratively update depths : Iteratively update depths
Depth through time: : Depth through time:
Matting : Matting Some pixels get influence for multiple surfaces. Background Surface Foreground Surface Image Camera Close up of real image: Multiple colors and depths at boundary pixels…
Find matting information: : Find matting information: Find boundary
strips using depth.
2. Within boundary strips compute the colors and depths of the foreground and background object. Background Foreground Strip Width
Why matting is important : Why matting is important Matting No Matting
Representation : Representation Color Depth MAIN LAYER Color Depth Matting information BOUNDARY LAYER
Rendering pipeline (GPU) : Rendering pipeline (GPU) Render
Main Layer Render
Boundary Layer Composite Render
Main Layer Render
Boundary Layer
Real-time viewpoint control : Real-time viewpoint control “Matrix”-like effects – on demand! (With much, much less hardware.)
Wrapping up… : Wrapping up…
Light Field Rendering : Light Field Rendering Sample and synthesize (capture and render) from a 4D ray space [Lightfield, Levoy & Hanrahan; Lumigraph, Gortler et al., SIGGRAPH’96]
Slow Glass : Slow Glass What if we could capture all the photons and play them all back at a later time?
Light of Other Days Bob Shaw, © 1969 http://www.scifi.com/scifiction/classics/classics_archive/shaw/shaw1.htm
Slow Glass : Slow Glass One could stand the glass beside, say, a woodland lake until the scene emerged, perhaps a year later. If the glass was then removed and installed in a dismal city flat, the flat would—for that year—appear to overlook the woodland lake. During the year it wouldn't be merely a very realistic but still picture—the water would ripple in sunlight, silent animals would come to drink, birds would cross the sky, night would follow day, season would follow season.
Apart from its stupendous novelty value, the commercial success of slow glass was founded on the fact that having a scenedow was the exact emotional equivalent of owning land. The meanest cave dweller could look out on misty parks—and who was to say they weren't his? A man who really owns tailored gardens and estates doesn't spend his time proving his ownership by crawling on his ground, feeling, smelling, tasting it. All he receives from the land are light patterns…
Being There : Being There Artists (and all of us) have always wanted to capture a sense of being there.
Computational photography and video bring us a lot closer:
realism (field of view, resolution, contrast)
movement and sound
immersion and exploration
What does the future hold?
Catch the
buzz on authorSTREAM
Copyright © 2002-2008 authorSTREAM. All rights reserved.