More Mosaic Madness: More Mosaic Madness 15-463: Computational Photography
Alexei Efros, CMU, Fall 2005 © Jeffrey Martin (jeffrey-martin.com) with a lot of slides stolen from
Steve Seitz and Rick Szeliski
Homography: Homography A: Projective – mapping between any two PPs with the same center of projection
rectangle should map to arbitrary quadrilateral
parallel lines aren’t
but must preserve straight lines
same as: project, rotate, reproject
called Homography
PP2 PP1 To apply a homography H
Compute p’ = Hp (regular matrix multiply)
Convert p’ from homogeneous to image coordinates
Rotational Mosaics: Rotational Mosaics Can we say something more about rotational mosaics?
i.e. can we further constrain our H?
3D → 2D Perspective Projection: 3D → 2D Perspective Projection
K
3D Rotation Model: 3D Rotation Model Projection equations
Project from image to 3D ray
(x0,y0,z0) = (u0-uc,v0-vc,f)
Rotate the ray by camera motion
(x1,y1,z1) = R01 (x0,y0,z0)
Project back into new (source) image
(u1,v1) = (fx1/z1+uc,fy1/z1+vc)
Therefore:
Our homography has only 3,4 or 5 DOF, depending if focal length is known, same, or different.
This makes image registration much better behaved f
Pairwise alignment: Pairwise alignment Procrustes Algorithm [Golub andamp; VanLoan]
Given two sets of matching points, compute R
pi’ = R pi with 3D rays
pi = N(xi,yi,zi) = N(ui-uc,vi-vc,f)
A = Σi pi pi’T = Σi pi piT RT = U S VT = (U S UT) RT
VT = UT RT
R = V UT
Rotation about vertical axis: Rotation about vertical axis What if our camera rotates on a tripod?
What’s the structure of H?
Do we have to project onto a plane? : Do we have to project onto a plane?
Full Panoramas: Full Panoramas What if you want a 360 field of view?
Cylindrical projection: Map 3D point (X,Y,Z) onto cylinder Cylindrical projection unit cylinder
Cylindrical Projection: Cylindrical Projection Y X
Inverse Cylindrical projection: Inverse Cylindrical projection
Cylindrical panoramas: Cylindrical panoramas Steps
Reproject each image onto a cylinder
Blend
Output the resulting mosaic
What are the assumptions here?
Cylindrical image stitching: Cylindrical image stitching What if you don’t know the camera rotation?
Solve for the camera rotations
Note that a rotation of the camera is a translation of the cylinder!
Assembling the panorama: Assembling the panorama Stitch pairs together, blend, then crop
Problem: Drift: Problem: Drift Vertical Error accumulation
small (vertical) errors accumulate over time
apply correction so that sum = 0 (for 360° pan.)
Horizontal Error accumulation
can reuse first/last image to find the right panorama radius
Full-view (360°) panoramas: Full-view (360°) panoramas
Spherical projection: Spherical projection unwrapped sphere Convert to spherical coordinates X Y Z Map 3D point (X,Y,Z) onto sphere f
Spherical Projection: Spherical Projection Y X
Inverse Spherical projection: Inverse Spherical projection X Y Z (x,y,z) (sinθcosφ,cosθcosφ,sinφ) cos φ φ cos θ cos φ sin φ
3D rotation: 3D rotation Rotate image before placing on unrolled sphere (x,y,z) (sinθcosφ,cosθcosφ,sinφ) cos φ φ cos θ cos φ sin φ _ _ _ _ p = R p
Full-view Panorama: Full-view Panorama + + + +
Polar Projection: Polar Projection Extreme 'bending' in ultra-wide fields of view
Other projections are possible: Other projections are possible You can stitch on the plane and then warp the resulting panorama
What’s the limitation here?
Or, you can use these as stitching surfaces
But there is a catch…
Cylindrical reprojection: Cylindrical reprojection
What’s your focal length, buddy?: What’s your focal length, buddy? Focal length is (highly!) camera dependant
Can get a rough estimate by measuring FOV:
Can use the EXIF data tag (might not give the right thing)
Can use several images together and try to find f that would make them match
Can use a known 3D object and its projection to solve for f
Etc.
There are other camera parameters too:
Optical center, non-square pixels, lens distortion, etc.
Distortion: Distortion Radial distortion of the image
Caused by imperfect lenses
Deviations are most noticeable for rays that pass through the edge of the lens No distortion Pin cushion Barrel
Radial distortion: Radial distortion Correct for 'bending' in wide field of view lenses
Use this instead of normal projection
Blending the mosaic: Blending the mosaic An example of image compositing:
the art (and sometime science) of
combining images together…
Feathering: Feathering Encoding transparency
I(x,y) = (aR, aG, aB, a)
Iblend = Ileft + Iright
Setting alpha: simple averaging: Setting alpha: simple averaging Alpha = .5 in overlap region
Image feathering: Image feathering Weight each image proportional to its distance from the edge (distance map [Danielsson, CVGIP 1980]
1. Generate weight map for each image
2. Sum up all of the weights and divide by sum: weights sum up to 1: wi’ = wi / ( ∑i wi)
Setting alpha: center seam: Setting alpha: center seam Alpha = logical(dtrans1andgt;dtrans2) Distance
transform
Setting alpha: blurred seam: Setting alpha: blurred seam Alpha = blurred Distance
transform
Setting alpha: center weighting: Setting alpha: center weighting Alpha = dtrans1 / (dtrans1+dtrans2) Distance
transform
Pyramid Blending: Pyramid Blending For Alpha, use original feather weights to select strongest contributing image
Can be implemented using L-∞ norm: (p = 10)
wi’ = [wip / ( ∑i wip)]1/p
Mid-term Tuesday: Mid-term Tuesday Closed book, closed notes.
You are allowed one sheet of paper, both sides