Families in the Wild (FIW): Large-Scale Kinship Image Database and Benchmarks

ABSTRACT

We present the largest kinship recognition dataset to date, Families in the Wild (FIW). Motivated by the lack of a single, unified dataset for kinship recognition, we aim to provide a dataset that captivates the interest of the research community. With only a small team, we were able to collect, organize, and label over 10,000 family photos of 1,000 families with our annotation tool designed to mark complex hierarchical relationships and local label information in a quick and efficient manner. We include several benchmarks for two image-based tasks, kinship verification and family recognition. For this, we incorporate several visual features and metric learning methods as baselines. Also, we demonstrate that a pre-trained Convolutional Neural Network (CNN) as an off-the-shelf feature extractor outperforms the other feature types. Then, results were further boosted by fine-tuning two deep CNNs on FIW data: (1) for kinship verification, a triplet loss function was learned on top of the network of pre-train weights; (2) for family recognition, a family-specific softmax classifier was added to the network.

References

1. T. Ahonen, A. Hadid, and M. Pietikainen. Face description with local binary patterns: Application to face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 28(12):2037--2041, 2006.
2. M. Almuashi, S. Z. M. Hashim, D. Mohamad, M. H. Alkawaz, and A. Ali. Automated kinship verification and identification through human facial images: a survey. Multimedia Tools and Applications, pages 1--43, 2015.
3. N. Crosswhite, J. Byrne, O. M. Parkhi, C. Stauffer, Q. Cao, and A. Zisserman. Template adaptation for face verification and identification. arXiv preprint arXiv:1603.03958, 2016.
4. J. V. Davis, B. Kulis, P. Jain, S. Sra, and I. S. Dhillon. Information-theoretic metric learning. In Proc. International Conference on Machine Learning (ICML), pages 209--216. ACM, 2007.
5. A. Dehghan, E. Ortiz, R. Villegas, and M. Shah. Who do i look like? determining parent-offspring resemblance via gated autoencoders. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1757--1764, 2014.
6. H. Dibeklioglu, A. Salah, and T. Gevers. Like father, like son: Facial expression dynamics for kinship verification. In Proc. IEEE International Conference on Computer Vision (ICCV), pages 1497--1504, 2013.
7. R. Fang, A. Gallagher, T. Chen, and A. Loui. Kinship classification by modeling facial feature heredity. In 20th IEEE International Conference on Image Processing (ICIP), pages 2983--2987. IEEE, 2013.
8. R. Fang, K. D. Tang, N. Snavely, and T. Chen. Towards computational models of kinship verification. In IEEE International Conference on Image Processing (ICIP), pages 1577--1580. IEEE, 2010.
9. R. B. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. CoRR, abs/1311.2524, 2013.
10. Y. Guo, H. Dibeklioglu, and L. van der Maaten. Graph-based kinship recognition. In International Conference on Pattern Recognition (ICPR), pages 4287--4292, 2014.
11. J. Hu, J. Lu, J. Yuan, and Y.-P. Tan. Large Margin Multi-metric Learning for Face and Kinship Verification in the Wild, pages 252--267. Springer International Publishing, Cham, 2015.
12. G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical Report 07--49, University of Massachusetts, Amherst, October 2007.
13. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014.
14. D. E. King. Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research, 10:1755--1758, 2009.
15. D. G. Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2):91--110, 2004.
16. J. Lu, X. Zhou, Y.-P. Tan, Y. Shang, and J. Zhou. Neighborhood repulsed metric learning for kinship verification. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 36(2):331--345, 2014.
17. O. M. Parkhi, A. Vedaldi, and A. Zisserman. Deep face recognition. In Proc. British Machine Vision Conference (BMVC), 2015.
18. X. Qin, X. Tan, and S. Chen. Tri-subject kinship verification: Understanding the core of A family. CoRR, abs/1501.02555, 2015.
19. F. Schroff, D. Kalenichenko, and J. Philbin. Facenet: A unified embedding for face recognition and clustering. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 815--823, 2015.
20. M. Shao, S. Xia, and Y. Fu. Genealogical face recognition based on ub kinface database. In CVPR 2011 WORKSHOPS, pages 60--65. IEEE, 2011.
21. M. Shao, S. Xia, and Y. Fu. Identity and kinship relations in group pictures. In Human-Centered Social Media Analytics, pages 175--190. Springer, 2014.
22. Y. Sun, X. Wang, and X. Tang. Deep learning face representation from predicting 10,000 classes. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1891--1898, 2014.
23. D. Wang, C. Otto, and A. K. Jain. Face search at scale: 80 million gallery. CoRR, abs/1507.07242, 2015.
24. M. Wang, Z. Li, X. Shu, J. Tang, et al. Deep kinship verification. In IEEE 17th International Workshop on Multimedia Signal Processing (MMSP), pages 1--6. IEEE, 2015.
25. S. Xia, M. Shao, and Y. Fu. Kinship verification through transfer learning. In Proc. International Joint Conferences on Artificial Intelligence (IJCAI), volume 22, pages 2539--2544, 2011.
26. S. Xia, M. Shao, and Y. Fu. Toward kinship verification using visual attributes. In Pattern Recognition (ICPR), 2012 21st International Conference on, pages 549--552. IEEE, 2012.
27. S. Xia, M. Shao, J. Luo, and Y. Fu. Understanding kin relationships in a photo. IEEE Transactions on Multimedia, 14(4):1046--1056, 2012.
28. F. Xiong, M. Gou, O. Camps, and M. Sznaier. Person re-identification using kernel-based metric learning methods. In European Conference on Computer Vision (ECCV), pages 1--16. Springer, 2014.
29. H. Yan, J. Lu, W. Deng, and X. Zhou. Discriminative multimetric learning for kinship verification. IEEE Transactions on Information Forensics and Security, 9(7):1169--1178, 2014.
30. H. Yan, J. Lu, and X. Zhou. Prototype-based discriminative feature learning for kinship verification. IEEE Transactions on Cybernetics, 45(11):2535--2545, 2015.
31. X. Zhou, J. Hu, J. Lu, Y. Shang, and Y. Guan. Kinship verification from facial images under uncontrolled conditions. In Proc. ACM International Conference on Multimedia, pages 953--956. ACM, 2011.
32. X. Zhou, J. Lu, J. Hu, and Y. Shang. Gabor-based gradient orientation pyramid for kinship verification under uncontrolled environments. In Proc. ACM International Conference on Multimedia, pages 725--728. ACM, 2012.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

  • Machine learning

    • Learning paradigms

      • Supervised learning

        • Supervised learning by classification

    • Machine learning approaches

      • Neural networks

• General and reference

  • Cross-computing tools and techniques

    • Evaluation

    • Experimentation

    • Metrics

• Human-centered computing

  • Collaborative and social computing

    • Collaborative and social computing design and evaluation methods

      • Social network analysis

• Information systems

  • Data management systems

    • Database design and models

• Security and privacy

  • Security services

    • Authentication

      • Biometrics

• Theory of computation

  • Theory and algorithms for application domains

    • Database theory

      • Data modeling

    • Machine learning theory

      • Kernel methods

        • Support vector machines