Identifying Modes of User Engagement with Online News and Their Relationship to Information Gain in Text

ABSTRACT

Prior work established the benefits of server-recorded user engagement measures (e.g. clickthrough rates) for improving the results of search engines and recommendation systems. Client-side measures of post-click behavior received relatively little attention despite the fact that publishers have now the ability to measure how millions of people interact with their content at a fine resolution using client-side logging. In this study, we examine patterns of user engagement in a large, client-side log dataset of over 7.7 million page views (including both mobile and non-mobile devices) of 66,821 news articles from seven popular news publishers. For each page view we use three summary statistics: dwell time, the furthest position the user reached on the page, and the amount of interaction with the page through any form of input (touch, mouse move, etc.). We show that simple transformations on these summary statistics reveal six prototypical modes of reading that range from scanning to extensive reading and persist across sites. Furthermore, we develop a novel measure of information gain in text to capture the development of ideas within the body of articles and investigate how information gain relates to the engagement with articles. Finally, we show that our new measure of information gain is particularly useful for predicting reading of news articles before publication, and that the measure captures unique information not available otherwise.

References

1. Eugene Agichtein, Eric Brill, and Susan Dumais. 2006. Improving web search ranking by incorporating user behavior information Proc. SIGIR 2006.
2. Ioannis Arapakis, Mounia Lalmas, B. Barla Cambazoglu, Mari-Carmen Marcos, and Joemon M. Jose. 2014. User engagement in online news: Under the scope of sentiment, interest, affect, and gaze. JASIST, Vol. 65, 10 (2014), 1988--2005.
3. Ralf Biedert, Jörn Hees, Andreas Dengel, and Georg Buscher. 2012. A robust realtime reading-skimming classifier. In Proc. ETRA 2012.
4. Sven Birkerts. 2006. The Gutenberg elegies: The fate of reading in an electronic age. Macmillan.
5. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. JMLR, Vol. 3, Jan (2003), 993--1022.
6. Georg Buscher, Edward Cutrell, and Meredith R. Morris. 2009. What do you see when you're surfing?: using eye tracking to predict salient regions of web pages. In Proc. CHI 2009.
7. Christopher S. Campbell and Paul P. Maglio. 2001. A robust algorithm for reading detection. In Proc. PUI 2001.
8. Mark Claypool, Phong Le, Makoto Wased, and David Brown. 2001. Implicit Interest Indicators. In Proc. of IUI 2001. 33--40.
9. Microsoft Canada Consumer insights. 2015. Attention spans. (2015). https://advertising.microsoft.com/en/WWDocs/User/display/cl/researchreport/31966/en/microsoft-attention-spans-research-report.pdf
10. Geoffrey B. Duggan and Stephen J. Payne. 2009. Text skimming: the process and effectiveness of foraging through text under time pressure. Journal of Experimental Psychology: Applied, Vol. 15, 3 (2009), 228.
11. Geoffrey B. Duggan and Stephen J. Payne. 2011. Skim reading by satisficing: evidence from eye tracking Proc. of CHI'2011. ACM.
12. Steve Fox, Kuldeep Karnawat, Mark Mydland, Susan Dumais, and Thomas White. 2005. Evaluating implicit measures to improve web search. Transactions on Information Systems (TOIS) Vol. 23, 2 (2005), 147--168.
13. Ruth Garner. 1987. Metacognition and reading comprehension.Ablex Publishing.
14. Daniel G. Goldstein, Siddharth Suri, R. Preston McAfee, Matthew Ekstrand-Abueg, and Fernando Diaz. 2014. The economic and cognitive costs of annoying display advertisements. Journal of Marketing Research Vol. 51, 6 (2014), 742--752.
15. Qi Guo and Eugene Agichtein. 2012. Beyond dwell time: estimating document relevance from cursor movements and other post-click searcher behavior. In Proc. WWW 2012.
16. Rafiq H. Hijazi and Robert W. Jernigan. 2009. Modelling compositional data using Dirichlet regression models. Applied Probability & Statistics Vol. 4, 1 (2009), 77--91.
17. William Horton, Lee Taylor, Arthur Ignacio, and Nancy L. Hoft. 1996. The Web page design cookbook: all the ingredients you need to create 5-star Web pages. New York: Wiley, 1996 (1996).
18. Jeff Huang, Ryen W. White, Georg Buscher, and Kuansan Wang. 2012. Improving searcher models using mouse cursor activity Proc. SIGIR 2012.
19. Clayton J. Hutto and Eric Gilbert. 2014. Vader: A parsimonious rule-based model for sentiment analysis of social media text Proc. ICWSM 2014.
20. Peter H. Johnston. 1983. Reading comprehension assessment: A cognitive basis.ERIC.
21. Youngho Kim, Ahmed Hassan, Ryen W. White, and Imed Zitouni. 2014. Modeling dwell time to predict click-level satisfaction Proc. WSDM 2014.
22. J. Peter Kincaid, Robert P. Fishburne Jr., Richard L. Rogers, and Brad S. Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical Report. DTIC Document.
23. Dmitry Lagun and Eugene Agichtein. 2015. Inferring searcher attention by jointly modeling user interactions and content salience Proc. SIGIR 2015.
24. Dmitry Lagun and Mounia Lalmas. 2016. Understanding User Attention and Engagement in Online News Reading Proc. of WSDM 2016.
25. Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents Proc. ICML 2014).
26. Janette Lehmann, Mounia Lalmas, Elad Yom-Tov, and Georges Dupret. 2012. Models of user engagement. In Proc. UMAP 2012.
27. Guoqiang Liao. 2011. On the Development of Reading Ability. Theory and Practice in Language Studies Vol. 1, 3 (2011), 302--305.
28. Chao Liu, Ryen W. White, and Susan Dumais. 2010. Understanding web browsing behaviors through weibull analysis of dwell time Proc. SIGIR 2010.
29. Ziming Liu. 2005. Reading behavior in the digital environment: Changes in reading behavior over the past ten years. Journal of documentation Vol. 61, 6 (2005), 700--712.
30. Michael E. Masson. 1982. Cognitive processes in skimming stories. Journal of Experimental Psychology: Learning, Memory, and Cognition, Vol. 8, 5 (1982).
31. Amy Mitchell, Jeffrey Gottfried, Michael Barthel, and Elisa Shearer. 2016. The Modern News Consumer, News Attitudes and Practices in the Digital Era. (2016).
32. Masahiro Morita and Yoichi Shinoda. 1994. Information filtering based on user behavior analysis and best match text retrieval Proc. SIGIR 1994.
33. Vidhya Navalpakkam, LaDawn Jentzsch, Rory Sayres, Sujith Ravi, Amr Ahmed, and Alex Smola. 2013. Measurement and modeling of eye-mouse behavior in the presence of nonlinear page layouts Proc. WWW 2013.
34. Jakob Nielsen. 2008. How Little Do Users Read. (2008). http://www.nngroup.com/articles/how-little-do-users-read/
35. Christine Nuttall. 1996. Teaching reading skills in a foreign language. ERIC.
36. Peter Pirolli and Stuart Card. 1999. Information foraging. Psychological review Vol. 106, 4 (1999), 643.
37. Kenneth R. Pugh, Bennett A. Shaywitz, Sally E. Shaywitz, R. Todd Constable, Pawel Skudlarski, Robert K. Fulbright, Richard A. Bronen, Donald P. Shankweiler, Leonard Katz, Jack M. Fletcher, and John C. Gore. 1996. Cerebral organization of component processes in reading. Brain, Vol. 119, 4 (1996), 1221--1238.
38. Keith Rayner. 1998. Eye movements in reading and information processing: 20 years of research. Psychological bulletin Vol. 124, 3 (1998), 372.
39. Carole Rich. 2015. Writing and reporting news: A coaching method. Cengage Learning. 259 pages.
40. Alexandre N. Tuch, Javier A. Bargas-Avila, Klaus Opwis, and Frank H. Wilhelm. 2009. Visual complexity of websites: Effects on users' experience, physiology, performance, and memory. International Journal of Human-Computer Studies, Vol. 67, 9 (2009), 703 -- 715.
41. Ou Wu, Yunfei Chen, Bing Li, and Weiming Hu. 2011. Evaluating the visual quality of web pages using a computational aesthetic approach Proc. WSDM 2011.
42. Xing Yi, Liangjie Hong, Erheng Zhong, Nanthan N. Liu, and Suju Rajan. 2014. Beyond clicks: dwell time for personalization. In Proc. RecSys 2014.

Index Terms

• Information systems

  • Information retrieval

    • Document representation

      • Content analysis and feature selection

      • Data encoding and canonicalization

      • Document structure

    • Retrieval tasks and goals

      • Information extraction

    • Specialized information retrieval

      • Multimedia and multimodal retrieval