Gazi Karam Illahi
Enrico Masala
Antti Oulasvirta
Skip Abstract Section
Abstract
Cloud gaming enables playing high-end games, originally designed for PC or game console setups, on low-end devices such as netbooks and smartphones, by offloading graphics rendering to GPU-powered cloud servers. However, transmitting the high-resolution video requires a large amount of network bandwidth, even though it is a compressed video stream. Foveated video encoding (FVE) reduces the bandwidth requirement by taking advantage of the non-uniform acuity of human visual system and by knowing where the user is looking. Based on a consumer-grade real-time eye tracker and an open source cloud gaming platform, we provide a cloud gaming FVE prototype that is game-agnostic and requires no modifications to the underlying game engine. In this article, we describe the prototype and its evaluation through measurements with representative games from different genres to understand the effect of parametrization of the FVE scheme on bandwidth requirements and to understand its feasibility from the latency perspective. We also present results from a user study on first-person shooter games. The results suggest that it is possible to find a “sweet spot” for the encoding parameters so the users hardly notice the presence of foveated encoding but at the same time the scheme yields most of the achievable bandwidth savings.
- Hamed Ahmadi, Saman Zad Tootaghaj, Mahmoud Reza Hashemi, and Shervin Shirmohammadi. 2014. A game attention model for efficient bit rate allocation in cloud gaming. Multim. Syst. 20, 5 (Oct. 2014), 485--501.Google Scholar
Digital Library
- Zahid Akhtar and Tiago H. Falk. 2017. Audio-visual multimedia quality assessment: A comprehensive survey. IEEE Access 5 (2017), 21090--21117. DOI:https://doi.org/10.1109/ACCESS.2017.2750918Google ScholarCross Ref
- Rachel Albert, Anjul Patney, David Luebke, and Joohwan Kim. 2017. Latency requirements for foveated rendering in virtual reality. ACM Trans. Appl. Percept. 14, 4, Article 25 (Sept. 2017), 13 pages. DOI:https://doi.org/10.1145/3127589Google ScholarDigital Library
- Ayub Bokani. 2014. Empirical evaluation of real-time video foveation. In Proceedings of the Workshop on Design, Quality and Deployment of Adaptive Video Streaming. ACM, 45--46.Google ScholarDigital Library
- Speedtest by Ookla. 2019. Monthly Comparisons of Internet Speeds from Around the World. Retrieved from https://www.speedtest.net/global-index.Google Scholar
- Wei Cai, Min Chen, and Victor C. M. Leung. 2014. Toward gaming as a service. IEEE Internet Comput. 18, 3 (May 2014), 12--18. DOI:https://doi.org/10.1109/MIC.2014.22Google ScholarCross Ref
- Wei Cai, Conghui Zhou, Victor C. M. Leung, and Min Chen. 2013. A cognitive platform for mobile cloud gaming. In Proceedings of the IEEE 5th International Conference on Cloud Computing Technology and Science, Vol. 1. 72--79. DOI:https://doi.org/10.1109/CloudCom.2013.17Google ScholarDigital Library
- Kuan-Ta Chen, Yu-Chun Chang, Hwai-Jung Hsu, De-Yu Chen, Chun-Ying Huang, and Cheng-Hsin Hsu. 2014. On the quality of service of cloud gaming systems. IEEE Trans. Multim. 16, 2 (2014), 480--495.Google ScholarDigital Library
- Kuan-Ta Chen, Yu-Chun Chang, Po-Han Tseng, Chun-Ying Huang, and Chin-Laung Lei. 2011. Measuring the latency of cloud gaming systems. In Proceedings of the 19th ACM International Conference on Multimedia (MM’11). ACM, New York, NY, 1269--1272. DOI:https://doi.org/10.1145/2072298.2071991Google ScholarDigital Library
- Zhenzhong Chen and Christine Guillemot. 2010. Perceptually-friendly H.264/AVC video coding based on foveated just-noticeable-distortion model. IEEE Trans. Circ. Syst. Vid. Technol. 20, 6 (June 2010), 806--819. DOI:https://doi.org/10.1109/TCSVT.2010.2045912Google Scholar
- Sharon Choy, Bernard Wong, Gwendal Simon, and Catherine Rosenberg. 2012. The brewing storm in cloud gaming: A measurement study on cloud to end-user latency. In Proceedings of the 11th Workshop on Network and Systems Support for Games (NetGames’12). IEEE Press, Piscataway, NJ, Article 2, 6 pages. Retrieved from http://dl.acm.org/citation.cfm?id=2501560.2501563.Google ScholarCross Ref
- Mark Claypool and Kajal Claypool. 2006. Latency and player actions in online games. Commun. ACM 49, 11 (Nov. 2006), 40--45. DOI:https://doi.org/10.1145/1167838.1167860Google ScholarDigital Library
- Jonathan Deber, Ricardo Jota, Clifton Forlines, and Daniel Wigdor. 2015. How much faster is fast enough? User perception of latency 8 latency improvements in direct and indirect touch. In Proceedings of the 33rd ACM Conference on Human Factors in Computing Systems (CHI’15), 1, 1 (2015), 1827--1836. DOI:https://doi.org/10.1145/2702123.2702300Google Scholar
- Agostino Gibaldi, Mauricio Vanegas, Peter J. Bex, and Guido Maiello. 2017. Evaluation of the Tobii EyeX eye tracking controller and Matlab toolkit for research. Behav. Res. Meth. 49, 3 (01 June 2017), 923--946. DOI:https://doi.org/10.3758/s13428-016-0762-9Google Scholar
- Chun-Ying Huang, Cheng-Hsin Hsu, Yu-Chun Chang, and Kuan-Ta Chen. 2013. GamingAnywhere: An open cloud gaming system. In Proceedings of the 4th ACM Multimedia Systems Conference (MMSys’13). ACM, New York, NY, 36--47. DOI:https://doi.org/10.1145/2483977.2483981Google ScholarDigital Library
- Chun-Ying Huang, Cheng-Hsin Hsu, Yu-Chun Chang, and Kuan-Ta Chen. 2013. GamingAnywhere: An open cloud gaming system. In Proceedings of the 4th ACM Multimedia Systems Conference (MMSys’13). ACM, New York, NY, 36--47. DOI:https://doi.org/10.1145/2483977.2483981Google ScholarDigital Library
- Wijnand Ijsselsteijn, Yvonne de Kort, Karolien Poels, Audrius Jurgelionis, and Francesco Bellotti. 2007. Characterising and measuring user experiences in digital games. In Proceedings of the International Conference on Advances in Computer Entertainment.Google Scholar
- Gazi Illahi, Matti Siekkinen, and Enrico Masala. 2017. Foveated video streaming for cloud gaming. In Proceedings of the IEEE 19th International Workshop on Multimedia Signal Processing (MMSP’17). 1--6. DOI:https://doi.org/10.1109/MMSP.2017.8122235Google ScholarCross Ref
- Laurent Itti. 2004. Automatic foveation for video compression using a neurobiological model of visual attention. IEEE Trans. Image Proc. 13, 10 (Oct. 2004), 1304--1318. DOI:https://doi.org/10.1109/TIP.2004.834657Google ScholarDigital Library
- JVT. 2003. Joint video team (JVT) of ISO/IEC MPEG and ITU-T VCEG. Draft ITU-T recommendation and final draft international standard of joint video specification (ITU-T Rec. H264 ISO/IEC 14496--10 AVC), document JVT-G050d35, 7th Meeting.Google Scholar
- M. Jarschel, D. Schlosser, S. Scheuring, and T. Hoßfeld. 2011. An evaluation of QoE in cloud gaming based on subjective tests. In Proceedings of the 5th International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing. 330--335.DOI:https://doi.org/10.1109/IMIS.2011.92Google Scholar
- Teemu Kämäräinen, Matti Siekkinen, Antti Ylä-Jääski, Wenxiao Zhang, and Pan Hui. 2017. A measurement study on achieving imperceptible latency in mobile cloud gaming. In Proceedings of the 8th ACM on Multimedia Systems Conference (MMSys’17). ACM, New York, NY, 88--99. DOI:https://doi.org/10.1145/3083187.3083191Google ScholarDigital Library
- Zhicheng Li, Shiyin Qin, and Laurent Itti. 2011. Visual attention guided bit allocation in video compression. Image Vis. Comput. 29, 1 (2011), 1--14. DOI:https://doi.org/10.1016/j.imavis.2010.07.001Google ScholarDigital Library
- Pietro Lungaro, Rickard Sjöberg, Alfredo J. F. Valero, Ashutosh Mittal, and Konrad Tollmar. 2018. Gaze-aware streaming solutions for the next generation of mobile VR experiences. IEEE Trans. Vis. Comput. Graph. 24, 4 (Apr. 2018), 1535--1544. DOI:https://doi.org/10.1109/TVCG.2018.2794119Google ScholarDigital Library
- Pietro Lungaro and Konrad Tollmar. 2017. QoE design tradeoffs for foveated content provision. In Proceedings of the 9th International Conference on Quality of Multimedia Experience (QoMEX’17). DOI:https://doi.org/10.1109/QoMEX.2017.7965669Google ScholarCross Ref
- Twan Maintz. 2005. Digital and medical image processing. Universiteit Utrecht (2005).Google Scholar
- Loren Merritt and Rahul Vanam. 2006. x264: A high performance H. 264/AVC encoder. Retrieved from http://neuron2.net/library/avc/overview_x264_v8_5.pdf.Google Scholar
- Iman S. Mohammadi, Mahmoud-Reza Hashemi, and Mohammad Ghanbari. 2015. An object-based framework for cloud gaming using player’s visual attention. In Proceedings of the IEEE International Conference on Multimedia Expo Workshops (ICMEW’15). 1--6. DOI:https://doi.org/10.1109/ICMEW.2015.7169781Google ScholarCross Ref
- Sebastian Möller, Dennis Pommer, Justus Beyer, and Jannis Rake-Revelant. 2013. Factors influencing gaming QoE: Lessons learned from the evaluation of cloud gaming services. In Proceedings of the 4th International Workshop on Perceptual Quality of Systems (PQS’13). 1--5.Google ScholarCross Ref
- Jim Mullin, Lucy Smallwood, Anna Watson, and Gillian Wilson. 2001. New techniques for assessing audio and video quality in real-time interactive communications. IHM-HCI Tutorial 2001, Lille, France. 1--63.Google Scholar
- Shruti Patil, Yu Chen, and Tajana Simunic Rosing. 2015. GazeTube: Gaze-Based adaptive video playback for bandwidth and power optimizations. In Proceedings of the IEEE Global Communications Conference (GLOBECOM’15). IEEE, 1--6.Google ScholarCross Ref
- Romass Pauliks, Konstantins Tretjaks, Kirils Belahs, and Romass Pauliks. 2013. A survey on some measurement methods for subjective video quality assessment. In Proceedings of the World Congress on Computer and Information Technology (WCCIT’13). DOI:https://doi.org/10.1109/WCCIT.2013.6618758Google ScholarCross Ref
- Feng Qian, Lusheng Ji, Bo Han, and Vijay Gopalakrishnan. 2016. Optimizing 360 video delivery over cellular networks. In Proceedings of the 5th Workshop on All Things Cellular: Operations, Applications and Challenges. ACM, 1--6.Google ScholarDigital Library
- Yashas Rai, Marcus Barkowsky, and Patrick Le Callet. 2016. Role of spatio-temporal distortions in the visual periphery in disrupting natural attention deployment. In Proceedings of the Conference on Human Vision and Electronic Imaging (HVEI’16). 1--6. DOI:https://doi.org/10.2352/ISSN.2470-1173.2016.16HVEI-117Google ScholarCross Ref
- Yashas Rai and Patrick Le Callet. 2017. Do gaze disruptions indicate the perceived quality of non-uniformly coded natural scenes? Electron. Imag. 14 (2017), 104--109. DOI:https://doi.org/10.2352/ISSN.2470-1173.2017.14.HVEI-124Google ScholarCross Ref
- Michele Rucci, Paul V. McGraw, and Richard J. Krauzlis. 2016. Fixational eye movements and perception. Vis. Res. 118 (2016), 1--4.Google ScholarCross Ref
- Jihoon Ryoo, Kiwon Yun, Dimitris Samaras, Samir R. Das, and Gregory Zelinsky. 2016. Design and evaluation of a foveated video streaming service for commodity client devices. In Proceedings of the 7th International Conference on Multimedia Systems (MMSys’16). ACM, New York, NY, Article 6, 11 pages.Google ScholarDigital Library
- Natela Shanidze, Saeideh Ghahghaei, and Preeti Verghese. 2016. Accuracy of eye position for saccades and smooth pursuit. J. Vis. 16, 15 (2016), 23--23.Google ScholarCross Ref
- Ryan Shea, Jiangchuan Liu, Edith C.-H. Ngai, and Yong Cui. 2013. Cloud gaming: Architecture and performance. IEEE Netw. 27, 4 (2013), 16--21.Google ScholarCross Ref
- Ivan Slivar, Mirko Suznjevic, Lea Skorin-Kapov, and Maja Matijasevic. 2014. Empirical QoE study of in-home streaming of online games. In Proceedings of the 13th Annual Workshop on Network and Systems Support for Games. 1--6.Google ScholarDigital Library
- Rand S. Swenson. 2006. Review of Clinical and Functional Neuroscience. Retrieved from https://www.dartmouth.edu/rswenson/NeuroSci/index.html.Google Scholar
- Tobii. 2015. Developer’s Guide Tobii EyeX SDK for C/C++. Retrieved from http://developer-files.tobii.com/wp-content/uploads/2016/03/Developers-Guide-C-Cpp.pdf.Google Scholar
- Niraj Tolia, Mahadev Satyanarayanan, and David G. Andersen. 2006. Quantifying interactive user experience on thin clients. Computer 39, 3 (2006), 46--52. DOI:https://doi.org/10.1109/MC.2006.101Google ScholarDigital Library
- Brian A. Wandell. 1995. Foundations of Vision. Vol. 8. Sinauer Associates, Sunderland, MA.Google Scholar
- Zhou Wang and Alan C. Bovik. 2006. Foveated image and video coding. In Digital Video, Image Quality, and Perceptual Coding, Hong Ren Wu and Kamisetty Ramamohan Rao (Eds.). CRC Press, Boca Raton, FL, 431--457.Google Scholar
- Zhou Wang, Ligang Lu, and Alan. C. Bovik. 2003. Foveation scalable video coding with automatic fixation selection. IEEE Trans. Image Proc. 12, 2 (Feb. 2003), 243--254. DOI:https://doi.org/10.1109/TIP.2003.809015Google Scholar
- Alireza Zare, Alireza Aminlou, Miska M. Hannuksela, and Moncef Gabbouj. 2016. HEVC-compliant tile-based streaming of panoramic video for virtual reality applications. In Proceedings of the ACM on Multimedia Conference (MM’16). ACM, New York, NY, 601--605.Google ScholarDigital Library
Index Terms
- Cloud Gaming with Foveated Video Encoding
Recommendations
Content-aware video encoding for cloud gaming
MMSys '19: Proceedings of the 10th ACM Multimedia Systems ConferenceCloud gaming allows users with thin-clients to play complex games on their end devices as the bulk of processing is offloaded to remote servers. A thin-client is only required to have basic decoding capabilities which exist on most modern devices. The ...
Game Categorization for Deriving QoE-Driven Video Encoding Configuration Strategies for Cloud Gaming
Special Section on Delay-Sensitive Video Computing in the Cloud and Special Section on Extended MMSys-NOSSDAV Best PapersCloud gaming has been recognized as a promising shift in the online game industry, with the aim of implementing the “on demand” service concept that has achieved market success in other areas of digital entertainment such as movies and TV shows. The ...
A game attention model for efficient bit rate allocation in cloud gaming
The widespread availability of broadband internet access and the growth in server-based processing have provided an opportunity to run games away from the player into the cloud and offer a new promising service known as cloud gaming. The concept of ...
Comments