CVPR 2016 paper reading (6)

1. Neuroaesthetics in fashion: modeling the perception of fashionability, Edgar Simo-Serra, Sanja Fidler, Francesc Moreno-Noguer, Raquel Urtasun, in CVPR 2015.

Goal: learn and predict how fashionable a person looks on a photograph, and suggest subtle improvements that user could make to improve her/his appeal.

 

This paper proposes a Conditional Random Field model that jointly reasons about several fashionability factors such as the type of outfit (全套装备） and garments （衣服） the user is wearing， the type of the user, the photograph's setting (e.g., the scenery behind the user), and the fashionability score.

Importantly, the proposed model is able to give rich feed back to the user, conveying which garments or even scenery she/he should change in order to improve fashionability.

 

 

This paper collects a novel dataset that consists of 144,169 user posts from a clothing-oriented social website chictopia.com. In a post, a user publishes one to six photographs of her/himself wearing a new outfit. Generally each photograph shows a different angle of the user or zoons in on different garments. User sometimes also add a description of the outfit, and/or tags of the types and colors of the garments they are wearing.

 

Discovering fashion from weak data:

The energy of the CRF as a sum of energies encoding unaries for each variable as well as non-parametric pairwise pothentials which reflect the correlations between the different random variables:

 

User specific features:

• the logarithm of the number of fans 
• use rekognition to compute attributes of all the images of each post, keep the features for the image with the highest score.

Then compute the unary potentials as the output of a small neural network, produce an 8-D feature map.

 

Outfit features:

bag-of-words approach on the "garments" and "colours" meta-data

 

Setting features:

• the output of a pre-trained scene classifier (multi-layer perceptron, whose input is CNN feature)
• user-provided location: look up the latitude and longitude of the user-provided location, project all the values on the unit sphere, and add some small Guassian noise. Then perform unsupervised clustering using the geodesic distances, and use the geodesic distance from each cluster center as a feature.

 

Fashion:

• delta time: the time between the creation of the post and when the post was crawled as a feature
• bag-of-words on the "tag"
• comments: parse the comments with the sentiment-analysis model, which can predict how positive a review is on a 1- 5 scale, sum the scores for each post.
• style: style classifier pretrained on Flickr80K.

 

Correlations:

use a non-parametric function for each pairwise and let the CRF learn the correlations:

Similarly for the other pairwise potentials.

 

Learn and Inference:

First jointly train the deep networks that are used for feature extraction to predict fashionablity, and estimate the initial latent states using clustering.

Then learn the CRF model using the primal-dual method.