Fine-Grained Image (细粒度图像) – Papers, Codes and Datasets

Table of contents

1. Introduction

2. Survey papers

3. Benchmark datasets

4. Fine-grained image recognition

   1. Fine-grained recognition by localization-classification subnetworks

   2. Fine-grained recognition by end-to-end feature encoding

   3. Fine-grained recognition with external information

      1. Fine-grained recognition with web data / auxiliary data

      2. Fine-grained recognition with multi-modality data

      3. Fine-grained recognition with humans in the loop

5. Fine-grained image retrieval

   1. Unsupervised with pre-trained models

   2. Supervised with metric learning

6. Fine-grained image generation

   1. Generating from fine-grained image distributions

   2. Generating from text descriptions

7. Future directions of FGIA

   1. Automatic fine-grained models

   2. Fine-grained few shot learning

   3. Fine-grained hashing

   4. FGIA within more realistic settings

8. Leaderboard

1. Introduction

---

This homepage lists some representative papers/codes/datasets all about deep learning based fine-grained image, including fine-grained image recognition, fine-grained image retrieval, fine-grained image generation, etc. If you have any questions, please feel free to leave message.

2. Survey papers

---

A Survey on Deep Learning-based Fine-Grained Object Classification and Semantic Segmentation

Bo Zhao, Jiashi Feng, Xiao Wu, and Shuicheng Yan. International Journal of Automation and Computing, 2017.

3. Benchmark datasets

---

Summary of popular fine-grained image datasets. Note that ‘‘BBox’’ indicates whether this dataset provides object bounding box supervisions. ‘‘Part anno.’’ means providing the key part localizations. ‘‘HRCHY’’ corresponds to hierarchical labels. ‘‘ATR’’ represents the attribute labels (e.g., wing color, male, female, etc). ‘‘Texts’’ indicates whether fine-grained text descriptions of images are supplied.

Dataset name	Year	Meta-class	Images	Categories	BBox	Part anno.	HRCHY	ATR	Texts
Oxford flower	2008	Flowers	8,189	102
CUB200	2011	Birds	11,788	200
Stanford Dog	2011	Dogs	20,580	120
Stanford Car	2013	Cars	16,185	196
FGVC Aircraft	2013	Aircrafts	10,000	100
Birdsnap	2014	Birds	49,829	500
NABirds	2015	Birds	48,562	555
DeepFashion	2016	Clothes	800,000	1,050
Fru92	2017	Fruits	69,614	92
Veg200	2017	Vegetable	91,117	200
iNat2017	2017	Plants & Animals	859,000	5,089
RPC	2019	Retail products	83,739	200

4. Fine-grained image recognition

---

Fine-grained recognition by localization-classification subnetworks

• Part-based R-CNNs for Fine-Grained Category Detection
  Ning Zhang, Jeff Donahue, Ross Girshick, and Trevor Darrell. ECCV, 2014. [Code]

• Interaction Part Mining: A Mid-Level Approach for Fine-Grained Action Recognition
  Yang Zhou, Bingbing Ni, Richang Hong, Meng Wang, and Qi Tian. CVPR, 2015.

• Fine-Grained Recognition without Part Annotations
  Jonathan Krause, Hailin Jin, Jianchao Yang, and Li Fei-Fei. CVPR, 2015. [Code]

• The Application of Two-level Attention Models in Deep Convolutional Neural Network for Fine-grained Image Classification
  Tianjun Xiao, Yichong Xu, Kuiyuan Yang, Jiaxing Zhang, Yuxin Peng, and Zheng Zhang. CVPR, 2015.

• Deep LAC: Deep Localization, Alignment and Classification for Fine-grained Recognition
  Di Lin, Xiaoyong Shen, Cewu Lu, and Jiaya Jia. CVPR, 2015.

• Spatial Transformer Networks
  Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. NIPS, 2015. [Code]

• Part-Stacked CNN for Fine-Grained Visual Categorization
  Shaoli Huang, Zhe Xu, Dacheng Tao, and Ya Zhang. CVPR, 2016.

• Mining Discriminative Triplets of Patches for Fine-Grained Classification
  Yaming Wang, Jonghyun Choi, Vlad I. Morariu, and Larry S. Davis. CVPR, 2016.

• SPDA-CNN: Unifying Semantic Part Detection and Abstraction for Fine-grained Recognition
  Han Zhang, Tao Xu, Mohamed Elhoseiny, Xiaolei Huang, Shaoting Zhang, Ahmed Elgammal, and Dimitris Metaxas. CVPR, 2016.

• Picking Deep Filter Responses for Fine-grained Image Recognition
  Xiaopeng Zhang, Hongkai, Xiong, Wengang Zhou, Weiyao Lin, and Qi Tian. CVPR, 2016.

• Look Closer to See Better: Recurrent Attention Convolutional Neural Network for Fine-Grained Image Recognition
  Jianlong Fu, Heliang Zheng, and Tao Mei. CVPR, 2017.

• Fine-Grained Recognition as HSnet Search for Informative Image Parts
  Michael Lam, Behrooz Mahasseni, and Sinisa Todorovic. CVPR, 2017.

• Learning Multi-attention Convolutional Neural Network for Fine-Grained Image Recognition
  Heliang Zheng, Jianlong Fu, Tao Mei, and Jiebo Luo. ICCV, 2017. [Code]

• Weakly Supervised Learning of Part Selection Model with Spatial Constraints for Fine-Grained Image Classification
  Xiangteng He, and Yuxin Peng. AAAI, 2017.

• Localizing by Describing: Attribute-Guided Attention Localization for Fine-Grained Recognition
  Xiao Liu, Jiang Wang, Shilei Wen, Errui Ding, and Yuanqing Lin. AAAI, 2017.

• Learning to Navigate for Fine-grained Classification
  Ze Yang, Tiange Luo, Dong Wang, Zhiqiang Hu, Jun Gao, and Liwei Wang. ECCV, 2018. [Code]

• Multi-Attention Multi-Class Constraint for Fine-grained Image Recognition
  Ming Sun, Yuchen Yuan, Feng Zhou, and Errui Ding. ECCV, 2018. [Code]

• Weakly Supervised Complementary Parts Models for Fine-Grained Image Classification From the Bottom Up
  Weifeng Ge, Xiangru Lin, and Yizhou Yu. CVPR, 2019.

Fine-grained recognition by end-to-end feature encoding

• Hyper-Class Augmented and Regularized Deep Learning for Fine-Grained Image Classification
  Saining Xie, Tianbao Yang, Xiaoyu Wang, and Yuanqing Lin. CVPR, 2015.

• Subset Feature Learning for Fine-Grained Category Classification
  ZongYuan Ge, Christopher McCool, Conrad Sanderson, and Peter Corke. CVPR, 2015.

• Bilinear CNN Models for Fine-grained Visual Recognition
  Tsung-Yu Lin, Aruni RoyChowdhury, and Subhransu Maji. ICCV, 2015. [Code]

• Multiple Granularity Descriptors for Fine-Grained Categorization
  Dequan Wang, Zhiqiang Shen, Jie Shao, Wei Zhang, Xiangyang Xue, and Zheng Zhang. ICCV, 2015.

• Compact Bilinear Pooling
  Yang Gao, Oscar Beijbom, Ning Zhang, and Trevor Darrell. CVPR, 2016. [Code]

• Fine-Grained Image Classification by Exploring Bipartite-Graph Labels
  Feng Zhou, and Yuanqing Lin. CVPR, 2016. [Project Page]

• Kernel Pooling for Convolutional Neural Networks
  Yin Cui, Feng Zhou, Jiang Wang, Xiao Liu, Yuanqing Lin, and Serge Belongie. CVPR, 2017.

• Low-rank Bilinear Pooling for Fine-Grained Classification
  Shu Kong, and Charless Fowlkes. CVPR, 2017. [Code]

• Higher-order Integration of Hierarchical Convolutional Activations for Fine-Grained Visual Categorization
  Sijia Cai, Wangmeng Zuo, and Lei Zhang. ICCV, 2017. [Code]

• Learning a Discriminative Filter Bank within a CNN for Fine-grained Recognition
  Yaming Wang, Vlad I. Morariu, and Larry S. Davis. CVPR, 2018. [Code]

• Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization
  Peihua Li, Jiangtao Xie, Qilong Wang, and Zilin Gao. CVPR, 2018. [Code]

• Maximum-Entropy Fine Grained Classification
  Abhimanyu Dubey, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. NIPS, 2018.

• Pairwise Confusion for Fine-Grained Visual Classification
  Abhimanyu Dubey, Otkrist Gupta, Pei Guo, Ramesh Raskar, Ryan Farrell, and Nikhil Naik. ECCV, 2018. [Code]

• DeepKSPD: Learning Kernel-matrix-based SPD Representation for Fine-Grained Image Recognition
  Melih Engin, Lei Wang, Luping Zhou, and Xinwang Liu. ECCV, 2018.

• Hierarchical Bilinear Pooling for Fine-Grained Visual Recognition
  Chaojian Yu, Xinyi Zhao, Qi Zheng, Peng Zhang, and Xinge You. ECCV, 2018. [Code]

• Grassmann Pooling as Compact Homogeneous Bilinear Pooling for Fine-Grained Visual Classification
  Xing Wei, Yue Zhang, Yihong Gong, Jiawei Zhang, and Nanning Zheng. ECCV, 2018.

• Looking for the Devil in the Details: Learning Trilinear Attention Sampling Network for Fine-Grained Image Recognition
  Heliang Zheng, Jianlong Fu, Zheng-Jun Zha, and Jiebo Luo. CVPR, 2019. [Code]

• Destruction and Construction Learning for Fine-grained Image Recognition
  Yue Chen, Yalong Bai, Wei Zhang, and Tao Mei. CVPR, 2019. [Code]

Fine-grained recognition with external information

   Fine-grained recognition with web data / auxiliary data

• Augmenting Strong Supervision Using Web Data for Fine-Grained Categorization
  Zhe Xu, Shaoli Huang, Ya Zhang, and Dacheng Tao. ICCV, 2015.

• Webly Supervised Learning Meets Zero-shot Learning: A Hybrid Approach for Fine-grained Classification
  Li Niu, Ashok Veeraraghavan, and Vshu Sabbarwal. CVPR, 2018.

• Fine-Grained Visual Categorization using Meta-Learning Optimization with Sample Selection of Auxiliary Data
  Yabin Zhang, Hui Tang, and Kai Jia. ECCV, 2018. [Code]

• Learning from Web Data using Adversarial Discriminative Neural Networks for Fine-Grained Classification
  Xiaoxiao Sun, Liyi Chen, and Jufeng Yang. AAAI, 2019.

   Fine-grained recognition with multi-modality data

• Fine-Grained Image Classification via Combining Vision and Language
  Xiangteng He, and Yuxin Peng. CVPR, 2017.

• Audio Visual Attribute Discovery for Fine-Grained Object Recognition
  Hua Zhang, Xiaochun Cao, and Rui Wang. AAAI, 2018.

• Fine-grained Image Classification by Visual-Semantic Embedding
  Huapeng Xu, Guilin Qi, Jingjing Li, Meng Wang, Kang Xu, and Huan Gao. IJCAI, 2018.

• Knowledge-Embedded Representation Learning for Fine-Grained Image Recognition
  Tianshui Chen, Liang Lin, Riquan Chen, Yang Wu, and Xiannan Luo. IJCAI, 2018.

   Fine-grained recognition with humans in the loop

• Fine-grained Categorization and Dataset Bootstrapping using Deep Metric Learning with Humans in the Loop
  Yin Cui, Feng Zhou, Yuanqing Lin, and Serge Belongie. CVPR, 2016.

5. Fine-grained image retrieval

---

Unsupervised with pre-trained models

• Selective Convolutional Descriptor Aggregation for Fine-Grained Image Retrieval
  Xiu-Shen Wei, Jian-Hao Luo, Jianxin Wu, and Zhi-Hua Zhou. TIP, 2017. [Project Page]

Supervised with metric learning

• Centralized Ranking Loss with Weakly Supervised Localization for Fine-Grained Object Retrieval
  Xiawu Zheng, Rongrong Ji, Xiaoshuai Sun, Yongjian Wu, Feiyue Huang, and Yanhua Yang. IJCAI, 2018.

• Towards Optimal Fine Grained Retrieval via Decorrelated Centralized Loss with Normalize-Scale layer
  Xiawu Zheng, Rongrong Ji, Xiaoshuai Sun, Baochang Zhang, Yongjian Wu, and Feiyue Huang. AAAI, 2019.

6. Fine-grained image generation

---

Generating from fine-grained image distributions

• CVAE-GAN: Fine-Grained Image Generation through Asymmetric Training
  Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, and Gang Hua. ICCV, 2017. [Code]

• FineGAN: Unsupervised Hierarchical Disentanglement for Fine-Grained Object Generation and Discovery
  Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee. CVPR, 2019. [Code]

Generating from text descriptions

• AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
  Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. CVPR, 2018. [Code]

7. Future directions of FGIA

---

Fine-grained few shot learning

• Piecewise classifier mappings: Learning fine-grained learners for novel categories with few examples
  Xiu-Shen Wei, Peng Wang, Lingqiao Liu, Chunhua Shen, and Jianxin Wu. TIP, 2019.

FGIA within more realistic settings

• Fine-grained Recognition in the Wild: A Multi-Task Domain Adaptation Approach
  Timnit Geru, Judy Hoffman, and Li Fei-Fei. ICCV, 2017.

• The iNaturalist Species Classification and Detection Dataset
  Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. CVPR 2018.

• RPC: A Large-Scale Retail Product Checkout Dataset
  Xiu-Shen Wei, Quan Cui, Lei Yang, Peng Wang, and Lingqiao Liu. arXiv: 1901.07249, 2019. [Project Page]

8. Leaderboard

---

The section is being continually updated. Since CUB200-2011 is the most popularly used fine-grained dataset, we list the fine-grained recognition leaderboard by treating it as the test bed.

Method	Publication	BBox	Part	External information	Base model	Image resolution	Accuracy
PB R-CNN	ECCV 2014				Alex-Net	224x224	73.9%
MaxEnt	NIPS 2018				GoogLeNet	TBD	74.4%
PB R-CNN	ECCV 2014				Alex-Net	224x224	76.4%
PS-CNN	CVPR 2016				CaffeNet	454x454	76.6%
MaxEnt	NIPS 2018				VGG-16	TBD	77.0%
Mask-CNN	PR 2018				Alex-Net	448x448	78.6%
PC	ECCV 2018				ResNet-50	TBD	80.2%
DeepLAC	CVPR 2015				Alex-Net	227x227	80.3%
MaxEnt	NIPS 2018				ResNet-50	TBD	80.4%
Triplet-A	CVPR 2016			Manual labour	GoogLeNet	TBD	80.7%
Multi-grained	ICCV 2015			WordNet etc.	VGG-19	224x224	81.7%
Krause et al.	CVPR 2015				CaffeNet	TBD	82.0%
Multi-grained	ICCV 2015			WordNet etc.	VGG-19	224x224	83.0%
TS	CVPR 2016				VGGD+VGGM	448x448	84.0%
Bilinear CNN	ICCV 2015				VGGD+VGGM	448x448	84.1%
STN	NIPS 2015				GoogLeNet+BN	448x448	84.1%
LRBP	CVPR 2017				VGG-16	224x224	84.2%
PDFS	CVPR 2016				VGG-16	TBD	84.5%
Xu et al.	ICCV 2015			Web data	CaffeNet	224x224	84.6%
Cai et al.	ICCV 2017				VGG-16	448x448	85.3%
RA-CNN	CVPR 2017				VGG-19	448x448	85.3%
MaxEnt	NIPS 2018				Bilinear CNN	TBD	85.3%
PC	ECCV 2018				Bilinear CNN	TBD	85.6%
CVL	CVPR 2017			Texts	VGG	TBD	85.6%
Mask-CNN	PR 2018				VGG-16	448x448	85.7%
GP-256	ECCV 2018				VGG-16	448x448	85.8%
KP	CVPR 2017				VGG-16	224x224	86.2%
T-CNN	IJCAI 2018				ResNet	224x224	86.2%
MA-CNN	ICCV 2017				VGG-19	448x448	86.5%
MaxEnt	NIPS 2018				DenseNet-161	TBD	86.5%
DeepKSPD	ECCV 2018				VGG-19	448x448	86.5%
OSME+MAMC	ECCV 2018				ResNet-101	448x448	86.5%
StackDRL	IJCAI 2018				VGG-19	224x224	86.6%
DFL-CNN	CVPR 2018				VGG-16	448x448	86.7%
PC	ECCV 2018				DenseNet-161	TBD	86.9%
KERL	IJCAI 2018			Attributes	VGG-16	224x224	87.0%
HBP	ECCV 2018				VGG-16	448x448	87.1%
Mask-CNN	PR 2018				ResNet-50	448x448	87.3%
DFL-CNN	CVPR 2018				ResNet-50	448x448	87.4%
NTS-Net	ECCV 2018				ResNet-50	448x448	87.5%
HSnet	CVPR 2017				GoogLeNet+BN	TBD	87.5%
MetaFGNet	ECCV 2018			Auxiliary data	ResNet-34	TBD	87.6%
CrossX	ICCV 2019				ResNet-50	448x448	87.7%
DCL	CVPR 2019				ResNet-50	448x448	87.8%
TASN	CVPR 2019				ResNet-50	448x448	87.9%
Ge et al.	CVPR 2019				GoogLeNet+BN	Shorter side is 800 px	90.4%