Beauty Is in the Eye of the Beholder: Demographically Oriented Analysis of Aesthetics in Photographs

Abstract

Aesthetics is a subjective concept that is likely to be perceived differently among people of different ages, genders, and cultural backgrounds. While techniques that directly compute this concept in images has seen increasing attention by the multimedia and machine-learning community, there are very few attempts at encoding the influences from the photographer’s viewpoint. This work demonstrates how the aesthetic quality of photos can be better learned by accounting for the demographic background of a photographer. A new AVA-PD (Photographer Demographic) dataset is created to supplement the AVA dataset by providing photographers’ age, gender and location attributes. Two deep convolutional neural network (CNN) architectures are proposed to utilize demographic information for aesthetic prediction of photos; both are shown to yield better prediction capabilities compared to most existing approaches. By leveraging on AVA-PD meta-data, we also present some additional machine-learnable tasks such as identifying the photographer and predicting photography styles from a person’s gallery of photos.

References

1. Yaniv Bar, Noga Levy, and Lior Wolf. 2014. Classification of artistic styles using binarized features derived from a deep neural network. In Proceedings of the Workshop at the European Conference on Computer Vision. 71--84.
2. Peter Niclas Broer, Sabrina Juran, Yuen-Jong Liu, Katie Weichman, Neil Tanna, Marc E. Walker, Reuben Ng, and John A. Persing. 2014. The impact of geographic, ethnic, and demographic dynamics on the perception of beauty. J. Cranio. Surg. 25, 2 (2014), e157--e161.
3. Camilo J. Cela-Conde, Francisco J. Ayala, Enric Munar, Fernando Maestú, Marcos Nadal, Miguel A. Capó, David del Río, Juan J. López-Ibor, Tomás Ortiz, Claudio Mirasso et al. 2009. Sex-related similarities and differences in the neural correlates of beauty. Proc. Nat. Acad. Sci. 106, 10 (2009), 3847--3852.
4. Wei-Ta Chu and Yi-Ling Wu. 2016. Deep correlation features for image style classification. In Proceedings of the ACM on Multimedia Conference. ACM, 402--406.
5. Ritendra Datta, Dhiraj Joshi, Jia Li, and James Z. Wang. 2006. Studying aesthetics in photographic images using a computational approach. In Proceedings of the European Conference on Computer Vision. Springer, 288--301.
6. Ritendra Datta, Jia Li, and James Z. Wang. 2007. Learning the consensus on visual quality for next-generation image management. In Proceedings of the 15th ACM International Conference on Multimedia. ACM, 533--536.
7. Sagnik Dhar, Vicente Ordonez, and Tamara L. Berg. 2011. High level describable attributes for predicting aesthetics and interestingness. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’11). IEEE, 1657--1664.
8. Zhe Dong, Xu Shen, Houqiang Li, and Xinmei Tian. 2015. Photo quality assessment with DCNN that understands image well. In Proceedings of the Conference on Multimedia Modeling (MMM’15). 524--535.
9. Yong-Lian Hii, John See, Magzhan Kairanbay, and Lai-Kuan Wong. 2017. Multigap: Multi-pooled inception network with text augmentation for aesthetic prediction of photographs. In Proceedings of the IEEE International Conference on Image Processing (ICIP’17). IEEE, 1722--1726.
10. Thomas Jacobsen. 2010. Beauty and the brain: Culture, history and individual differences in aesthetic appreciation. J. Anatomy 216, 2 (2010), 184--191.
11. Xin Jin, Jingying Chi, Siwei Peng, Yulu Tian, Chaochen Ye, and Xiaodong Li. 2016. Deep image aesthetics classification using inception modules and fine-tuning connected layer. In Proceedings of the 8th International Conference on Wireless Communications 8 Signal Processing (WCSP’16). IEEE, 1--6.
12. Magzhan Kairanbay, John See, and Lai-Kuan Wong. 2016. Aesthetic evaluation of facial portraits using compositional augmentation for deep CNNs. In Proceedings of the Asian Conference on Computer Vision. Springer, 462--474.
13. Magzhan Kairanbay, John See, and Lai-Kuan Wong. 2018. Towards demographic-based photographic aesthetics prediction for portraitures. In Proceedings of the International Conference on Multimedia Modeling. Springer, 531--543.
14. Magzhan Kairanbay, John See, Lai-Kuan Wong, and Yong-Lian Hii. 2017. Filling the gaps: Reducing the complexity of networks for multi-attribute image aesthetic prediction. In Proceedings of the IEEE International Conference on Image Processing (ICIP’17). IEEE, 3051--3055.
15. Yueying Kao, Ran He, and Kaiqi Huang. 2016a. Visual aesthetic quality assessment with multi-task deep learning. Retrieved from arXiv preprint arXiv:1604.049705 (2016).
16. Yueying Kao, Ran He, and Kaiqi Huang. 2017. Deep aesthetic quality assessment with semantic information. IEEE Trans. Image Proc. 26, 3 (2017), 1482--1495.
17. Yueying Kao, Kaiqi Huang, and Steve Maybank. 2016b. Hierarchical aesthetic quality assessment using deep convolutional neural networks. Sig. Proc.: Image Commun. 47 (2016), 500--510.
18. Sergey Karayev, Matthew Trentacoste, Helen Han, Aseem Agarwala, Trevor Darrell, Aaron Hertzmann, and Holger Winnemoeller. 2013. Recognizing image style. Retrieved from: arXiv preprint arXiv:1311.3715.
19. Yan Ke, Xiaoou Tang, and Feng Jing. 2006. The design of high-level features for photo quality assessment. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), Vol. 1. IEEE, 419--426.
20. Shu Kong, Xiaohui Shen, Zhe Lin, Radomir Mech, and Charless Fowlkes. 2016. Photo aesthetics ranking network with attributes and content adaptation. In Proceedings of the European Conference on Computer Vision. Springer, 662--679.
21. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Proceedings of the Conference on Advances in Neural Information Processing Systems. 1097--1105.
22. Challenging Technologies LLC. 2018. DpChallenge dataset. Retrieved from: http://www.dpchallenge.com/.
23. Xin Lu, Zhe Lin, Hailin Jin, Jianchao Yang, and James Z. Wang. 2014. Rapid: Rating pictorial aesthetics using deep learning. In Proceedings of the 22nd ACM International Conference on Multimedia. ACM, 457--466.
24. Xin Lu, Zhe Lin, Hailin Jin, Jianchao Yang, and James Z. Wang. 2015a. Rating image aesthetics using deep learning. IEEE Trans. Multimed. 17, 11 (2015), 2021--2034.
25. Xin Lu, Zhe Lin, Xiaohui Shen, Radomir Mech, and James Z. Wang. 2015b. Deep multi-patch aggregation network for image style, aesthetics, and quality estimation. In Proceedings of the IEEE International Conference on Computer Vision. 990--998.
26. Long Mai, Hailin Jin, and Feng Liu. 2016. Composition-preserving deep photo aesthetics assessment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 497--506.
27. Luca Marchesotti, Florent Perronnin, Diane Larlus, and Gabriela Csurka. 2011. Assessing the aesthetic quality of photographs using generic image descriptors. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’11). IEEE, 1784--1791.
28. Takahiko Masuda, Richard Gonzalez, Letty Kwan, and Richard E. Nisbett. 2008. Culture and aesthetic preference: Comparing the attention to context of East Asians and Americans. Person. Soc. Psych. Bull. 34, 9 (2008), 1260--1275.
29. Naila Murray, Luca Marchesotti, and Florent Perronnin. 2012. AVA: A large-scale database for aesthetic visual analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’12). 2408--2415.
30. Pere Obrador, Xavier Anguera, Rodrigo de Oliveira, and Nuria Oliver. 2009. The role of tags and image aesthetics in social image search. In Proceedings of the 1st SIGMM Workshop on Social Media. ACM, 65--72.
31. Pere Obrador, Ludwig Schmidt-Hackenberg, and Nuria Oliver. 2010. The role of image composition in image aesthetics. In Proceedings of the 17th IEEE International Conference on Image Processing (ICIP’10). IEEE, 3185--3188.
32. Miriam Redi, Nikhil Rasiwasia, Gaurav Aggarwal, and Alejandro Jaimes. 2015. The beauty of capturing faces: Rating the quality of digital portraits. In Proceedings of the 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG’15), Vol. 1. IEEE, 1--8.
33. Jian Ren, Xiaohui Shen, Zhe Lin, Radomír Mech, and David J. Foran. 2017. Personalized image aesthetics. In Proceedings of the IEEE International Conference on Computer Vision. 638--647.
34. Andreas E. Savakis, Stephen P. Etz, and Alexander C. P. Loui. 2000. Evaluation of image appeal in consumer photography. In Proceedings of the Conference on Human Vision and Electronic Imaging V, Vol. 3959. International Society for Optics and Photonics, 111--121.
35. Katharina Schwarz, Patrick Wieschollek, and Hendrik P. A. Lensch. 2018. Will people like your image? Learning the aesthetic space. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV’18). 2048--2057.
36. Tiancheng Sun, Yulong Wang, Jian Yang, and Xiaolin Hu. 2017. Convolution neural networks with two pathways for image style recognition. IEEE Trans. Image Proc. 26, 9 (2017), 4102--4113.
37. Xiaoshuai Sun, Hongxun Yao, Rongrong Ji, and Shaohui Liu. 2009. Photo assessment based on computational visual attention model. In Proceedings of the 17th ACM International Conference on Multimedia. ACM, 541--544.
38. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2818--2826.
39. Christopher Thomas and Adriana Kovashka. 2016. Seeing behind the camera: Identifying the authorship of a photograph. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3494--3502.
40. Stanford University. 2018. ImageNet dataset. Retrieved from: http://www.image-net.org/.
41. Guolong Wang, Junchi Yan, and Zheng Qin. 2018. Collaborative and attentive learning for personalized image aesthetic assessment. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI’18). 957--963.
42. Weining Wang, Mingquan Zhao, Li Wang, Jiexiong Huang, Chengjia Cai, and Xiangmin Xu. 2016. A multi-scene deep learning model for image aesthetic evaluation. Sig. Proc.: Image Commun. 47 (2016), 511--518.
43. Zhangyang Wang, Ding Liu, Shiyu Chang, Florin Dolcos, Diane Beck, and Thomas Huang. 2017. Image aesthetics assessment using Deep Chatterjee’s machine. In Proceedings of the International Joint Conference on Neural Networks (IJCNN’17). IEEE, 941--948.
44. Lai-Kuan Wong and Kok-Lim Low. 2009. Saliency-enhanced image aesthetics class prediction. In Proceedings of the 16th IEEE International Conference on Image Processing (ICIP’09). IEEE, 997--1000.
45. Luming Zhang, Yue Gao, Roger Zimmermann, Qi Tian, and Xuelong Li. 2014. Fusion of multichannel local and global structural cues for photo aesthetics evaluation. IEEE Trans. Image Proc. 23, 3 (2014), 1419--1429.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Image and video acquisition

        • Computational photography

  • Machine learning

    • Learning paradigms

      • Supervised learning

    • Machine learning approaches

      • Neural networks