lightfields UVa 18oct05 san

Slide1: 

Light field photography and videography Marc Levoy Computer Science Department Stanford University

List of projects: 

List of projects high performance imaging using large camera arrays light field photography using a handheld plenoptic camera dual photography

High performance imagingusing large camera arrays: 

High performance imaging using large camera arrays Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Eino-Ville Talvala, Emilio Antunez, Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005)

Stanford multi-camera array: 

Stanford multi-camera array 640 × 480 pixels × 30 fps × 128 cameras synchronized timing continuous streaming flexible arrangement

Ways to use large camera arrays: 

Ways to use large camera arrays widely spaced light field capture tightly packed high-performance imaging intermediate spacing synthetic aperture photography

Intermediate camera spacing:synthetic aperture photography: 

Intermediate camera spacing: synthetic aperture photography

Example using 45 cameras[Vaish CVPR 2004]: 

Example using 45 cameras [Vaish CVPR 2004]

Tiled camera array: 

Tiled camera array world’s largest video camera no parallax for distant objects poor lenses limit image quality seamless mosaicing isn’t hard Can we match the image quality of a cinema camera?

Tiled panoramic image(before geometric or color calibration): 

Tiled panoramic image (before geometric or color calibration)

Tiled panoramic image(after calibration and blending): 

Tiled panoramic image (after calibration and blending)

Tiled camera array: 

Tiled camera array world’s largest video camera no parallax for distant objects poor lenses limit image quality seamless mosaicing isn’t hard per-camera exposure metering HDR within and between tiles Can we match the image quality of a cinema camera?

Slide13: 

same exposure in all cameras

High-performance photography as multi-dimensional sampling: 

High-performance photography as multi-dimensional sampling spatial resolution field of view frame rate dynamic range bits of precision depth of field focus setting color sensitivity

Spacetime aperture shaping: 

Spacetime aperture shaping shorten exposure time to freeze motion → dark stretch contrast to restore level → noisy increase (synthetic) aperture to capture more light → decreases depth of field

Slide16: 

center of aperture: few cameras, long exposure → high depth of field, low noise, but action is blurred periphery of aperture: many cameras, short exposure → freezes action, low noise, but low depth of field

Light field photography using a handheld plenoptic camera: 

Light field photography using a handheld plenoptic camera Ren Ng, Marc Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz and Pat Hanrahan (Proc. SIGGRAPH 2005 and TR 2005-02)

Conventional versus light field camera: 

Conventional versus light field camera

Conventional versus light field camera: 

Conventional versus light field camera

Conventional versus light field camera: 

Conventional versus light field camera uv-plane st-plane

Prototype camera: 

Prototype camera 4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens Contax medium format camera Kodak 16-megapixel sensor

Mechanical design: 

Mechanical design microlenses float 500μ above sensor focused using 3 precision screws

Prior work: 

Prior work integral photography microlens array + film application is autostereoscopic effect [Adelson 1992] proposed this camera built an optical bench prototype using relay lenses application was stereo vision, not photography

Digitally stopping-down: 

Digitally stopping-down stopping down = summing only the central portion of each microlens Σ Σ

Digital refocusing: 

Digital refocusing refocusing = summing windows extracted from several microlenses Σ

A digital refocusing theorem: 

A digital refocusing theorem an f / N light field camera, with P × P pixels under each microlens, can produce views as sharp as an f / (N × P) conventional camera – or – it can produce views with a shallow depth of field ( f / N ) focused anywhere within the depth of field of an f / (N × P) camera

Example of digital refocusing: 

Example of digital refocusing

Refocusing portraits: 

Refocusing portraits

Action photography: 

Action photography

Extending the depth of field : 

Extending the depth of field conventional photograph, main lens at f / 22 conventional photograph, main lens at f / 4 light field, main lens at f / 4, after all-focus algorithm [Agarwala 2004]

Macrophotography: 

Macrophotography

Digitally moving the observer: 

Digitally moving the observer moving the observer = moving the window we extract from the microlenses Σ Σ

Example of moving the observer: 

Example of moving the observer

Moving backward and forward: 

Moving backward and forward

Implications: 

Implications cuts the unwanted link between exposure (due to the aperture) and depth of field trades off (excess) spatial resolution for ability to refocus and adjust the perspective sensor pixels should be made even smaller, subject to the diffraction limit 36mm × 24mm ÷ 2.5μ pixels = 266 megapixels 20K × 13K pixels 4000 × 2666 pixels × 20 × 20 rays per pixel

Can we build a light field microscope?: 

Can we build a light field microscope? ability to photograph moving specimens digital refocusing → focal stack → deconvolution microscopy → volume data

Dual Photography: 

Dual Photography Pradeep Sen, Billy Chen, Gaurav Garg, Steve Marschner, Mark Horowitz, Marc Levoy, Hendrik Lensch (Proc. SIGGRAPH 2005)

Slide41: 

Helmholtz reciprocity scene light camera

Slide42: 

Helmholtz reciprocity scene camera light

Slide43: 

photocell scene Measuring transport along a set of paths projector

Slide44: 

scene point light Reversing the paths camera

Slide45: 

Forming a dual photograph scene photocell projector “dual” light “dual” camera

Slide46: 

Forming a dual photograph scene image of scene “dual” light “dual” camera

Physical demonstration: 

Physical demonstration light replaced with projector camera replaced with photocell projector scanned across the scene conventional photograph, with light coming from right dual photograph, as seen from projector’s position and as illuminated from photocell’s position

Related imaging methods: 

Related imaging methods time-of-flight scanner if they return reflectance as well as range but their light source and sensor are typically coaxial scanning electron microscope Velcro® at 35x magnification, Museum of Science, Boston

Slide49: 

The 4D transport matrix scene photocell projector camera

Slide50: 

camera The 4D transport matrix scene projector

Slide51: 

= The 4D transport matrix pq x 1 mn x 1 mn x pq

Slide52: 

= The 4D transport matrix 1 0 0 0 0 mn x pq pq x 1 mn x 1

Slide53: 

= The 4D transport matrix 0 1 0 0 0 mn x pq pq x 1 mn x 1

Slide54: 

= The 4D transport matrix 0 0 1 0 0 mn x pq pq x 1 mn x 1

Slide55: 

The 4D transport matrix

Slide56: 

The 4D transport matrix applying Helmholtz reciprocity... = pq x 1 mn x 1 mn x pq = mn x 1 pq x 1 pq x mn T

Example: 

Example conventional photograph with light coming from right dual photograph as seen from projector’s position

Properties of the transport matrix: 

Properties of the transport matrix little interreflection → sparse matrix many interreflections → dense matrix convex object → diagonal matrix concave object → full matrix Can we create a dual photograph entirely from diffuse reflections?

Dual photographyfrom diffuse reflections: 

Dual photography from diffuse reflections the camera’s view

The relighting problem: 

The relighting problem subject captured under multiple lights one light at a time, so subject must hold still point lights are used, so can’t relight with cast shadows Paul Debevec’s Light Stage 3

The 6D transport matrix: 

The 6D transport matrix

The 6D transport matrix: 

The 6D transport matrix

The advantage of dual photography: 

The advantage of dual photography capture of a scene as illuminated by different lights cannot be parallelized capture of a scene as viewed by different cameras can be parallelized

Slide64: 

scene Measuring the 6D transport matrix projector camera array mirror array camera

Relighting with complex illumination: 

Relighting with complex illumination step 1: measure 6D transport matrix T step 2: capture a 4D light field step 3: relight scene using captured light field scene camera array projector

Running time: 

Running time the different rays within a projector can in fact be parallelized to some extent this parallelism can be discovered using a coarse-to-fine adaptive scan can measure a 6D transport matrix in 5 minutes

Can we measure an 8D transport matrix?: 

Can we measure an 8D transport matrix? scene camera array projector array

Slide68: 

http://graphics.stanford.edu