Beyond Soft Focus dgray 1 15 08

Beyond Soft Focus:Beautifying Portraits without Blur: 

Beyond Soft Focus: Beautifying Portraits without Blur Doug Gray 1/15/08

The digital eye of the beholder: 

The digital eye of the beholder “Beauty is an experience, nothing else. It is not a fixed pattern or an arrangement of features. It is something felt, a glow or a communicated sense of fineness.” - D. H. Lawrence How can we possibly learn something that doesn’t physically exist or have patterns or features ?!?

Background: 

Background http://www.hotornot.com/ has millions of images and over 12 billion votes cast Can we use some of this data to predict beauty directly from image data? (novelty/scientific focus) Photo retouching has been automated, but relies on landmark features and/or selective blurring Leyvand et. al. @ SIGRAPH ‘06 http://www.portraitprofessional.com/

Previous work (Leyvand et. al.): 

Previous work (Leyvand et. al.) Requires landmark features, makes geometric changes

Previous Work (www.portraitprofessional.com): 

Previous Work (www.portraitprofessional.com)

Our Approach Overview: 

Our Approach Overview

How is this different?: 

How is this different? No landmark features We model the concept manifold directly from the image data No manually designed features No explicit smoothness, lightness, or symmetry constraints Potentially applicable to other domains

Ratings Website: 

Ratings Website

Pairwise to Absolute Ratings: 

Pairwise to Absolute Ratings Need absolute score for training Minimize a cost function which penalizes disordering (via gradient descent): Phi is an exponential function +/- indicates a pairwise preference

Active Learning: 

Active Learning Pairs can be chosen to learn scores faster Image 1 chosen w.r.t number of ratings Image 2 chosen w.r.t proximity to image 1 ri is the number or ratings an image has received

Data Collected: 

Data Collected Over 2000 Images, Over 7000 Pairwise Ratings

Convolutional Neural Networks Review: 

Convolutional Neural Networks Review Perceptron Learning Rule ΔWi = η * (Ddesired output – Y) * Xi Backpropagation Algorithm Compute error at each node Gradient is calculated on said error Connections via convolution

Hierarchical Feed-forward Model: 

Hierarchical Feed-forward Model Proposed by Hubel and Wiesel Motivated by dead monkeys and cats Alternating layers of convolution/downsampling Downsampling with max operator Popularized by Fukushima, LeCun, Poggio, et al. K. Fukushima ‘80

Our CNN Architecture: 

Our CNN Architecture Multi-scale approach with 5 layers YCbCr colorspace Random connections between each layer

Prediction Results (plot): 

Prediction Results (plot)

Prediction Results (correlation data): 

Prediction Results (correlation data)

Beauty Derivative?: 

Beauty Derivative?

Semantic Gradient Descent: 

Semantic Gradient Descent Cost function with separate regularization constants for luminance and chrominance Simplified update equation

Beauty Manifold (faces): 

Beauty Manifold (faces) Optimal degree of modification Analogous to Overfitting

Beauty Manifold (eyes & noses): 

Beauty Manifold (eyes & noses) Conclusions: Dark eyes with blue/purple Mascara are attractive Big noses are not…

Beautification / Beastification Results: 

Beautification / Beastification Results

Relationship to Average Faces: 

Relationship to Average Faces Average Faces are thought to be attractive…

Conclusions & Future Work: 

Conclusions & Future Work The concept of beauty can be predicted using artificial neural networks A beautification filter has been built, but is not a replacement for a Photoshop expert Where else can this technique be applied? Can we build a semantic image filter? Live demo, if time is available…