Mohamed Hegazy
Mohamed Hefeeda
ABSTRACT
Cloud gaming allows users with thin-clients to play complex games on their end devices as the bulk of processing is offloaded to remote servers. A thin-client is only required to have basic decoding capabilities which exist on most modern devices. The result of the remote processing is an encoded video that gets streamed to the client. As modern games are complex in terms of graphics and motion, the encoded video requires high bandwidth to provide acceptable Quality of Experience (QoE) to end users. The cost incurred by the cloud gaming service provider to stream the encoded video at such high bandwidth grows rapidly with the increase in the number of users. In this paper, we present a content-aware video encoding method for cloud gaming (referred to as CAVE) to improve the perceptual quality of the streamed video frames with comparable bandwidth requirements. This is a challenging task because of the stringent requirements on latency in cloud gaming, which impose additional restrictions on frame sizes as well as processing time to limit the total latency perceived by clients. Unlike many of the previous works, the proposed method is suitable for the state-of-the-art High Efficiency Video Coding (HEVC) encoder, which by itself offers substantial bitrate savings compared to prior encoders. The proposed method leverages information from the game such as the Regions-of-Interest (ROIs), and optimizes the quality by allocating different amounts of bits to various areas in the video frames. Through actual implementation in an open-source cloud gaming platform, we show that the proposed method achieves quality gains in ROIs that can be translated to bitrate savings between 21% and 46% against the baseline HEVC encoder and between 12% and 89% against the closest work in the literature.
- Advanced Micro Devices, Inc. 2018. How to Capture Your Gameplay Using Radeon ReLive. Retrieved November 05, 2018 from https://support.amd.com/en-us/kb-articles/Pages/DH-023.aspx#StreamingGoogle Scholar
- Hamed Ahmadi, Saman Zad Tootaghaj, Mahmoud Reza Hashemi, and Shervin Shirmohammadi. 2014. A game attention model for efficient bit rate allocation in cloud gaming. Multimedia Systems 20, 5 (2014), 485--501.Google ScholarDigital Library
- Gisle Bjontegaard. 2001. Calculation of average PSNR differences between RD-curves. Video Coding Experts Group-M33 (2001).Google Scholar
- Business Wire, Inc. 2018. Cloud Gaming Market Analysis By Platform, Service Type & Geography, With Forecasts to 2023. Retrieved October 08, 2018 from https://www.businesswire.com/news/home/20180424005697/en/Cloud-Gaming-Market-Analysis-Platform-Service-TypeGoogle Scholar
- Wei Cai, Ryan Shea, Chun-Ying Huang, Kuan-Ta Chen, Jiangchuan Liu, Victor C.M. Leung, and Cheng-Hsin Hsu. 2016. A Survey on Cloud Gaming: Future of Computer Games. IEEE Access 4 (2016), 7605--7620.Google ScholarCross Ref
- Hyomin Choi, Junghak Nam, Jonghun Yoo, D. Sim, and Ivan Bajic. 2012. Rate control based on unified RQ model for HEVC. ITU-T SG16 Contribution, JCTVC-H0213 (2012), 1--13.Google Scholar
- Kareem Choudhry. 2018. Project xCloud: Gaming with you at the center. Retrieved October 08, 2018 from https://blogs.microsoft.com/blog/2018/10/08/project-xcloud-gaming-with-you-at-the-center/Google Scholar
- Sharon Choy, Bernard Wong, Gwendal Simon, and Catherine Rosenberg. 2012. The brewing storm in cloud gaming: A measurement study on cloud to end-user latency. In Proc. of the ACM International Workshop on Network and Systems Support for Games. 2:1--2:6.Google ScholarCross Ref
- Seong-Ping Chuah and Ngai-Man Cheung. 2014. Layered Coding for Mobile Cloud Gaming. In Proc. of the ACM International Workshop on Massively Multiuser Virtual Environments. 1--6. Google ScholarDigital Library
- Mark Claypool and Kajal Claypool. 2010. Latency Can Kill: Precision and Deadline in Online Games. In Proc. of the ACM International Conference on Multimedia Systems. 215--222. Google ScholarDigital Library
- NVIDIA Corporation. 2018. NVIDIA GeForce. Retrieved October 08, 2018 from https://www.nvidia.com/en-us/geforce/products/geforce-now/Google Scholar
- Valve Corporation. 2018. Extend your Steam gaming experience to your phone, tablet, or TV over your local network. Retrieved November 05, 2018 from https://store.steampowered.com/steamlink/aboutGoogle Scholar
- John M. Henderson, James R. Brockmole, Monica S. Castelhano, and Michael Mack. 2007. Chapter 25 - Visual saliency does not account for eye movements during visual search in real-world scenes. In Eye Movements. Elsevier, 537 - III.Google Scholar
- Hua-Jun Hong, De-Yu Chen, Chun-Ying Huang, Kuan-Ta Chen, and Cheng-Hsin Hsu. 2015. Placing virtual machines to optimize cloud gaming experience. IEEE Transactions on Cloud Computing 3, 1 (2015), 42--53.Google ScholarCross Ref
- Hua-Jun Hong, Chih-Fan Hsu, Tsung-Han Tsai, Chun-Ying Huang, Kuan-Ta Chen, and Cheng-Hsin Hsu. 2015. Enabling Adaptive Cloud Gaming in an Open-Source Cloud Gaming Platform. IEEE Transactions on Circuits and Systems for Video Technology 25, 12 (2015), 2078--2091.Google ScholarCross Ref
- Catherine Hsiao. 2018. Pushing the limits of streaming technology. Retrieved October 08, 2018 from https://blog.google/technology/developers/pushing-limits-streaming-technology/Google Scholar
- Chun-Ying Huang, Cheng-Hsin Hsu, Yu-Chun Chang, and Kuan-Ta Chen. 2013. GamingAnywhere: An Open Cloud Gaming System. In Proc. of the ACM International Conference on Multimedia Systems. 36--47. Google ScholarDigital Library
- Chun-Ying Huang, Yu-Ling Huang, Yu-Hsuan Chi, Kuan-Ta Chen, and Cheng-Hsin Hsu. 2015. To Cloud or Not to Cloud: Measuring the Performance of Mobile Gaming. In Proc. of the ACM International Workshop on Mobile Gaming. 19--24. Google ScholarDigital Library
- Gazi Illahi, Matti Siekkinen, and Enrico Masala. 2017. Foveated video streaming for cloud gaming. In Proc. of the IEEE International Workshop on Multimedia Signal Processing. 1--6.Google ScholarCross Ref
- MulticoreWare Inc. 2018. About x265. Retrieved October 19, 2018 from http://x265.org/about/Google Scholar
- Instructables. 2017. How to Make a Simple Game in Unity 3D. Retrieved Januaury 19, 2019 from https://www.instructables.com/id/How-to-make-a-simple-game-in-Unity-3D/Google Scholar
- Yihao Ke, Guoqiao Ye, Di Wu, Yipeng Zhou, Edith Ngai, and Han Hu. 2017. GECKO: Gamer Experience-Centric Bitrate Control Algorithm for Cloud Gaming. In Proc. of the ACM International Conference on Image and Graphics. 325--335.Google ScholarCross Ref
- Jong-Seok Lee and Touradj Ebrahimi. 2012. Perceptual video compression: A survey. IEEE Journal of Selected Topics in Signal Processing 6, 6 (2012), 684--697.Google ScholarCross Ref
- Bin Li, Houqiang Li, Li Li, and Jinlei Zhang. 2012. Rate control by R-lambda model for HEVC. ITU-T SG16 Contribution, JCTVC-K0103 (2012), 1--5.Google Scholar
- Bin Li, Houqiang Li, Li Li, and Jinlei Zhang. 2014. λ-Domain Rate Control Algorithm for High Efficiency Video Coding. IEEE Transactions on Image Processing 23, 9 (2014), 3841--3854.Google ScholarCross Ref
- Shengxi Li, Mai Xu, Xin Deng, and Zulin Wang. 2015. Weight-based R-λ rate control for perceptual HEVC coding on conversational videos. Signal Processing: Image Communication 38 (2015), 127--140. Google ScholarDigital Library
- Zhi Li, Anne Aaron, Ioannis Katsavounidis, Anush Moorthy, and Megha Manohara. 2018. Toward A Practical Perceptual Video Quality Metric. Retrieved October 18, 2018 from https://medium.com/netflix-techblog/toward-a-practical-perceptual-video-quality-metric-653f208b9652Google Scholar
- Xiaofei Liao, Li Lin, Guang Tan, Hai Jin, Xiaobin Yang, Wei Zhang, Bo Li, Xiaofei Liao, Li Lin, Guang Tan, Hai Jin, Xiaobin Yang, Wei Zhang, and Bo Li. 2016. LiveRender: A Cloud Gaming System Based on Compressed Graphics Streaming. IEEE/ACM Transactions on Networking 24, 4 (2016), 2128--2139.Google ScholarDigital Library
- LiquidSky Software, Inc. 2018. LiquidSky. Retrieved October 08, 2018 from https://liquidsky.com/Google Scholar
- Yao Liu, Sujit Dey, and Yao Lu. 2015. Enhancing video encoding for cloud gaming using rendering information. IEEE Transactions on Circuits and Systems for Video Technology 25, 12 (2015), 1960--1974.Google ScholarCross Ref
- Marwa Meddeb, Marco Cagnazzo, and Béatrice Pesquet-Popescu. 2014. Region-of-interest-based rate control scheme for high-efficiency video coding. APSIPA Transactions on Signal and Information Processing 3 (2014), e16.Google ScholarCross Ref
- Robert J. Peters and Laurent Itti. 2008. Applying Computational Tools to Predict Gaze Direction in Interactive Visual Environments. ACM Transactions on Applied Perception 5, 2, Article 9 (2008), 19 pages. Google ScholarDigital Library
- Ashwin Rao, Arnaud Legout, Yeon-sup Lim, Don Towsley, Chadi Barakat, and Walid Dabbous. 2011. Network Characteristics of Video Streaming Traffic. In Proc. of the ACM International Conference on Emerging Networking Experiments and Technologies. 25:1--25:12. Google ScholarDigital Library
- Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You Only Look Once: Unified, Real-Time Object Detection. In Proc. of the IEEE International Conference on Computer Vision and Pattern Recognition. 779--788.Google ScholarCross Ref
- Ivan Slivar, Lea Skorin-Kapov, and Mirko Suznjevic. 2016. Cloud Gaming QoE Models for Deriving Video Encoding Adaptation Strategies. In Proc. of the ACM International Conference on Multimedia Systems. 18:1--18:12.Google ScholarDigital Library
- Sony Interactive Entertainment LLC. 2016. Playstation Now. Retrieved October 08, 2018 from https://www.playstation.com/en-ca/explore/playstationnow/Google Scholar
- Gary J. Sullivan, Jens-Rainer Ohm, Woo-Jin Han, Thomas Wiegand, et al. 2012. Overview of the High Efficiency Video Coding (HEVC) Standard. IEEE Transactions on Circuits and Systems for Video Technology 22, 12 (2012), 1649--1668. Google ScholarDigital Library
- Kairan Sun and Dapeng Wu. 2015. Video rate control strategies for cloud gaming. Journal of Visual Communication and Image Representation 30 (2015), 234--241. Google ScholarDigital Library
- Shanshe Wang, Siwei Ma, Shiqi Wang, Debin Zhao, and Wen Gao. 2013. Quadratic ρ-domain based rate control algorithm for HEVC. In Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing. 1695--1699.Google Scholar
- Zhou Wang and Alan C. Bovik. 2001. Embedded foveation image coding. IEEE Transactions on Image Processing 10, 10 (2001), 1397--1410. Google ScholarDigital Library
- Zhou Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. 2004. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600--612. Google ScholarDigital Library
- Zhongzhu Yang, Li Song, Zhengyi Luo, and Xiangwen Wang. 2014. Low delay rate control for HEVC. In Proc. of the IEEE International Symposium on Broadband Multimedia Systems and Broadcasting. 1--5.Google ScholarCross Ref
Index Terms
- Content-aware video encoding for cloud gaming
Recommendations
Cloud Gaming with Foveated Video Encoding
Cloud gaming enables playing high-end games, originally designed for PC or game console setups, on low-end devices such as netbooks and smartphones, by offloading graphics rendering to GPU-powered cloud servers. However, transmitting the high-resolution ...
Using graphics rendering contexts to enhance the real-time video coding for mobile cloud gaming
MM '11: Proceedings of the 19th ACM international conference on MultimediaThe emerging cloud gaming service has been growing rapidly, but not yet able to reach mobile customers due to many limitations, such as bandwidth and latency. We introduce a 3D image warping assisted real-time video coding method that can potentially ...
Content-aware error-resilient transcoding using prioritized intra-refresh for video streaming
Transmitting video data over wireless networks can be very unreliable due to packet-loss, leading to serious video quality degradation which is annoying to human perception. The lost packets not only affect the quality of current frame, but also lead to ...
Comments