Guided Selfies using Models of Portrait Aesthetics

ABSTRACT

We introduce techniques enabling interactive guidance for better self-portrait photos ("selfies") using a smartphone cam- era. Aesthetic quality is estimated using empirical models for three parameterized composition principles: face size, face position, and lighting direction. The models are built using 2,700 crowdworker assessments of highly-controlled synthetic selfies. These are generated by manipulating a virtual camera and lighting when rendering a realistic 3D model of a human to methodically explore the parameter space. A camera application uses the models to estimate the aesthetic quality of a live selfie preview based on parameters measured by computer vision. The photographer is guided towards a better selfie by directional hints overlaid on the live preview. A study shows the technique provides a 26% increase in aesthetic quality compared to a standard camera application.

References

1. ArtInCam LTD. 2014. Camera51. (2014). https://www.camera51.com
2. Subhabrata Bhattacharya, Rahul Sukthankar, and Mubarak Shah. 2010. A Framework for Photo-quality Assessment and Enhancement Based on Visual Aesthetics. In Proceedings of the International Conference on Multimedia (MM '10). ACM, New York, NY, USA, 271--280.
3. Stephen A. Brewster and Jody Johnston. 2008. Multimodal Interfaces for Camera Phones. In Proceedings of the 10th International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI '08). ACM, New York, NY, USA, 387--390.
4. Scott Carter, John Adcock, John Doherty, and Stacy Branham. 2010. NudgeCam: Toward Targeted, Higher Quality Media Capture. In Proceedings of the 18th ACM International Conference on Multimedia (MM '10). ACM, New York, NY, USA, 615--618.
5. Géry Casiez, Nicolas Roussel, and Daniel Vogel. 2012. 1 filter: a simple speed-based low-pass filter for noisy input in interactive systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2527--2530.
6. Xiaowu Chen, Xin Jin, Hongyu Wu, and Qinping Zhao. 2015. Learning templates for artistic portrait lighting analysis. IEEE Transactions on Image Processing 24, 2 (2015), 608--618.
7. John Child. 2008. Studio Photography: Essential Skills. Focal Press.
8. Casio Computer Co. 2016. Life Style Digital Cameras. (2016). http://www.casio-intl.com/asia-mea/en/dc/lineup/ Hardware: Digital Cameras.
9. Ritendra Datta, Dhiraj Joshi, Jia Li, and JamesZ. Wang. 2006. Studying Aesthetics in Photographic Images Using a Computational Approach. In Computer Vision -- ECCV 2006, Aleš Leonardis, Horst Bischof, and Axel Pinz (Eds.). Lecture Notes in Computer Science, Vol. 3953. Springer Berlin Heidelberg, 288--301.
10. S. Dhar, V. Ordonez, and T.L. Berg. 2011. High level describable attributes for predicting aesthetics and interestingness. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on. 1657--1664.
11. Christina Dickson. 2009. 6 Tips for Perfect Composition in Portrait Photography. (Sept. 2009). http://digital-photography-school.com/ 6-tips-for-perfect-composition-in-portrait-photography/
12. Felix Esser. 2014. Camera51 Android app makes you a master photographer by guiding your framing. (Oct. 2014). http://www.digitaltrends.com/photography/camera51-android-app-makes-master-photographer-guiding-framing/
13. Ohad Fried, Eli Shechtman, Dan B. Goldman, and Adam Finkelstein. 2016. Perspective-aware Manipulation of Portrait Photos. ACM Trans. Graph. 35, 4, Article 128 (July 2016), 10 pages.
14. Dmitry O Gorodnichy and Gerhard Roth. 2004. Nouse 'use your nose as a mouse' perceptual vision technology for hands-free games and interfaces. Image and Vision Computing 22, 12 (2004), 931--942.
15. Melanie Hall. 2013. Family albums fade as the young put only themselves in picture. (June 2013). http://www.telegraph.co.uk/technology/
16. Darlene Hildebrandt. 2013. Lighting Ratios to Make or Break your Portrait. (2013). http://digital-photography-school.com/lighting-ratios-to-make-or-break-your-portrait/
17. Bill Hurter. 2007. Portrait Photographer's Handbook (3 ed.). Amherst Media.
18. Wei Jiang, A.C. Loui, and C.D. Cerosaletti. 2010. Automatic aesthetic value assessment in photographic images. In Multimedia and Expo (ICME), 2010 IEEE International Conference on. 920--925.
19. Mahdi M Kalayeh, Misrak Seifu, Wesna LaLanne, and Mubarak Shah. 2015. How to take a good selfie?. In Proceedings of the 23rd ACM international conference on Multimedia. ACM, 923--926.
20. Shehroz S. Khan and Daniel Vogel. 2012. Evaluating Visual Aesthetics in Photographic Portraiture. In Proceedings of the Eighth Annual Symposium on Computational Aesthetics in Graphics, Visualization, and Imaging (CAe '12). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 55--62. http://dl.acm.org/citation.cfm?id=2328888.2328898
21. Masahito Kumano, Kuniaki Uehara, and Yasuo Ariki. 2006. Online training-oriented video shooting navigation system based on real-time camerawork evaluation. In Multimedia and Expo, 2006 IEEE International Conference on. IEEE, 1281--1284.
22. Congcong Li and Tsuhan Chen. 2009. Aesthetic Visual Quality Assessment of Paintings. Selected Topics in Signal Processing, IEEE Journal of 3, 2 (April 2009), 236--252.
23. LuMee LLC. 2016a. LuMee Phone Case. (2016). https://lumee.com/collections/all {Hardware: Phone Accessory}.
24. Ty-Lite LLC. 2016b. Ty-Lite Phone Case. (2016). https://ty-lite.com {Hardware: Phone Accessory}.
25. Wei Luo, Xiaogang Wang, and Xiaoou Tang. 2011. Content-based photo quality assessment. In Computer Vision (ICCV), 2011 IEEE International Conference on. 2206--2213.
26. Shuang Ma, Yangyu Fan, and Chang Wen Chen. 2014. Finding your spot: A photography suggestion system for placing human in the scene. In 2014 IEEE International Conference on Image Processing (ICIP). 556--560.
27. M. Males, A. Hedi, and M. Grgic. 2013. Aesthetic quality assessment of headshots. In ELMAR, 2013 55th International Symposium. 89--92.
28. Lev Manovich. 2014. SelfieCity. (2014). http://selfiecity.net
29. Filippo Mazza, Matthieu Perreira Da Silva, Patrick Le Callet, and IEJ Heynderickx. 2015. What do you think of my picture? Investigating factors of influence in profile images context perception. In IS&T/SPIE Electronic Imaging. International Society for Optics and Photonics, 93940D.
30. Christopher McAdam, Craig Pinkerton, and Stephen A. Brewster. 2010. Novel Interfaces for Digital Cameras and Camera Phones. In Proceedings of the 12th International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI '10). ACM, New York, NY, USA, 143--152.
31. P. Obrador, L. Schmidt-Hackenberg, and N. Oliver. 2010. The role of image composition in image aesthetics. In Image Processing (ICIP), 2010 17th IEEE International Conference on. 3185--3188.
32. Bahareh Rahmanian and Joseph G Davis. 2014. User interface design for crowdsourcing systems. In Proceedings of the 2014 International Working Conference on Advanced Visual Interfaces. ACM, 405--408.
33. Miriam Redi, Nikhil Rasiwasia, Gaurav Aggarwal, and Alejandro Jaimes. 2015. The Beauty of Capturing Faces: Rating the Quality of Digital Portraits. CoRR abs/1501.07304 (2015). http://arxiv.org/abs/1501.07304
34. Patrick Rice. 2005. Professional Techniques for Black & White Digital Photography. Amherst Media, Buffalo, NY.
35. Jose San pedro and Karen Church. 2013. The Sound of Light: Induced Synesthesia for Augmenting the Photography Experience. In CHI '13 Extended Abstracts on Human Factors in Computing Systems (CHI EA '13). ACM, New York, NY, USA, 745--750.
36. Lai-Kuan Wong and Kok-Lim Low. 2009. Saliency-enhanced image aesthetics class prediction. In Image Processing (ICIP), 2009 16th IEEE International Conference on. 997--1000.
37. Yan Xu, Joshua Ratcliff, James Scovell, Gheric Speiginer, and Ronald Azuma. 2015. Real-time Guidance Camera Interface to Enhance Photo Aesthetic Quality. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 1183--1186.
38. Lei Yao, Poonam Suryanarayan, Mu Qiao, James Z Wang, and Jia Li. 2012. Oscar: On-site composition and aesthetics feedback through exemplars for photographers. International Journal of Computer Vision 96, 3 (2012), 353--383.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)