Rating Effects on Social News Posts and Comments

Abstract

At a time when information seekers first turn to digital sources for news and opinion, it is critical that we understand the role that social media plays in human behavior. This is especially true when information consumers also act as information producers and editors through their online activity. In order to better understand the effects that editorial ratings have on online human behavior, we report the results of a two large-scale in vivo experiments in social media. We find that small, random rating manipulations on social media posts and comments created significant changes in downstream ratings, resulting in significantly different final outcomes. We found positive herding effects for positive treatments on posts, increasing the final rating by 11.02% on average, but not for positive treatments on comments. Contrary to the results of related work, we found negative herding effects for negative treatments on posts and comments, decreasing the final ratings, on average, of posts by 5.15% and of comments by 37.4%. Compared to the control group, the probability of reaching a high rating ( ⩾ 2,000) for posts is increased by 24.6% when posts receive the positive treatment and for comments it is decreased by 46.6% when comments receive the negative treatment.

References

1. Ashton Anderson, Daniel Huttenlocher, Jon Kleinberg, and Jure Leskovec. 2012. Discovering value from community activity on focused question answering sites: A case study of stack overflow. In SIGKDD. ACM, New York, 850--858. https://doi.org/10.1145/2339530.2339665
2. Lisa R. Anderson and Charles A. Holt. 1997. Information cascades in the laboratory. American Economic Review 87, 5 (1997), 847--862. www.jstor.org/stable/2951328
3. Sinan Aral, Lev Muchnik, and Arun Sundararajan. 2009. Distinguishing influence-based contagion from homophily-driven diffusion in dynamic networks. Proceedings of the National Academy of Sciences 106, 51 (2009), 21544--21549. https://doi.org/10.1073/pnas.0908800106
4. Solomon E. Asch. 1955. Opinions and social pressure. Scientific American 193, 5 (1955), 31--35. https://doi.org/10.1038/scientificamerican1155-31
5. Ricardo Baeza-Yates, Arjen P. Vries, Hugo Zaragoza, B. Barla Cambazoglu, Vanessa Murdock, Ronny Lempel, and Fabrizio Silvestri (Eds.). 2012. Advances in Information Retrieval. Lecture Notes in Computer Science, Vol. 7224. Springer, Berlin. https://doi.org/10.1007/978-3-642-28997-2
6. Eytan Bakshy, Brian Karrer, and Lada A. Adamic. 2009. Social influence and the diffusion of user-created content. In EC. ACM, New York, 325--334. https://doi.org/10.1145/1566374.1566421
7. Miguel Ángel Ballester and Pedro Rey-Biel. 2009. Does uncertainty lead to sincerity? Simple and complex voting mechanisms. Social Choice and Welfare 33, 3 (Feb. 2009), 477--494. http://link.springer.com/10.1007/s00355-009-0374-8
8. Abhijit V. Banerjee. 1992. A simple model of herd behavior. Quarterly Journal of Economics 107, 3 (1992), 797--817. https://doi.org/10.2307/2118364
9. John J. Bartholdi and James B. Orlin. 1991. Single transferable vote resists strategic voting. Social Choice and Welfare 8, 4 (1991), 341--354. https://doi.org/10.1007/BF00183045
10. John J. Bartholdi, Craig A. Tovey, and Michael A. Trick. 1989. The computational difficulty of manipulating an election. Social Choice and Welfare 6, 3 (1989), 227--241. https://doi.org/10.1007/BF00295861
11. Sushil Bikhchandani, David Hirshleifer, and Ivo Welch. 1992. A theory of fads, fashion, custom, and cultural change as informational cascades. Journal of Political Economy 100, 5 (Oct. 1992), 992--1026. https://doi.org/10.2307/2138632
12. Daren C. Brabham. 2008. Crowdsourcing as a model for problem solving: An introduction and cases. Convergence: The International Journal of Research into New Media Technologies 14, 1 (2008), 75--90. https://doi.org/10.1177/1354856507084420
13. Georg Buscher, Edward Cutrell, and Meredith Ringel Morris. 2009. What do you see when you’re surfing?: Using eye tracking to predict salient regions of web pages. In SIGCHI. ACM, New York, 21--30. https://doi.org/10.1145/1518701.1518705
14. Boaçhan Çelen and Shachar Kariv. 2004. Distinguishing informational cascades from herd behavior in the laboratory. American Economic Review 94, 3 (June 2004), 484--498. http://www.aeaweb.org/articles.php?doi=10.1257/0002828041464461
15. Pei-Yu Chen, Samita Dhanasobhon, and Michael Smith. 2008. All reviews are not created equal: The disaggregate impact of reviews and reviewers at amazon.com. Available at SSRN (2008). https://doi.org/10.2139/ssrn.918083
16. Judith A. Chevalier and Dina Mayzlin. 2006. The effect of word of mouth on sales: Online book reviews. Journal of Marketing Research 43, 3 (Aug. 2006), 345--354. https://doi.org/10.1509/jmkr.43.3.345
17. Vincent Conitzer, Toby Walsh, and Lirong Xia. 2011. Dominating manipulations in voting with partial information. In AAAI. AAAI Press, Palo Alto, CA, 638--643.
18. Chrysanthos Dellarocas, Xiaoquan (Michael) Zhang, and Neveen F. Awad. 2007. Exploring the value of online product reviews in forecasting sales: The case of motion pictures. Journal of Interactive Marketing 21, 4 (Jan. 2007), 23--45. https://doi.org/10.1002/dir.20087
19. Wenjing Duan, Bin Gu, and Andrew B. Whinston. 2008a. Do online reviews matter? -- An empirical investigation of panel data. Decision Support Systems 45, 4 (Nov. 2008), 1007--1016. https://doi.org/10.1016/j.dss.2008.04.001
20. Wenjing Duan, Bin Gu, and Andrew B. Whinston. 2008b. The dynamics of online word-of-mouth and product sales: An empirical investigation of the movie industry. Journal of Retailing 84, 2 (May 2008), 233--242. http://papers.ssrn.com/abstract=1101054
21. Amir Fayazi, Kyumin Lee, James Caverlee, and Anna Squicciarini. 2015. Uncovering crowdsourced manipulation of online reviews. In SIGIR. ACM, New York, NY, 233--242. https://doi.org/10.1145/2766462.2767742
22. Francis Galton. 1907. The ballot box. Nature 75 (1907), 509--510.
23. Saptarshi Ghosh, Bimal Viswanath, Farshad Kooti, Naveen Kumar Sharma, Gautam Korlam, Fabricio Benevenuto, Niloy Ganguly, and Krishna Phani Gummadi. 2012. Understanding and combating link farming in the twitter social network. In WWW. ACM Press, New York, NY, 61. https://doi.org/10.1145/2187836.2187846
24. Allan Gibbard. 1973. Manipulation of voting schemes: A general result. Econometrica 41, 4 (1973), 587--601. https://doi.org/10.2307/1914083
25. Eric Gilbert. 2013. Widespread underprovision on reddit. In CSCW. ACM Press, New York, NY, 803. https://doi.org/10.1145/2441776.2441866
26. Maria Glenski, Corey Pennycuff, and Tim Weninger. 2017. Consumers and curators: Browsing and voting patterns on reddit. arXiv preprint arXiv:1703.05267 (2017).
27. William D. Hamilton. 1971. Geometry for the selfish herd. Journal of Theoretical Biology 31, 2 (1971), 295--311. https://doi.org/10.1016/0022-5193(71)90189-5
28. Ward A. Hanson and Daniel S. Putler. 1996. Hits and misses: Herd behavior and online product popularity. Marketing Letters 7, 4 (Oct. 1996), 297--305. https://doi.org/10.1007/BF00435537
29. Eszter Hargittai and Gina Walejko. 2008. The participation divide: Content creation and sharing in the digital age. Information, Community and Society 11, 2 (2008), 239--256. https://doi.org/10.1080/13691180801946150
30. David A. Hirshleifer. 1995. The blind leading the blind: Social influence, fads and informational cascades. In The New Economics of Human Behaviour, Kathryn Ierulli and Mariano Tommasi (Eds.). Cambridge University Press, Cambridge, England, Chapter 12, 188--215. http://papers.ssrn.com/abstract=1278625
31. Nan Hu, Paul A. Pavlou, and Jennifer Zhang. 2006. Can online reviews reveal a product’s true quality? In EC. ACM Press, New York, NY, 324--330. https://doi.org/10.1145/1134707.1134743
32. Olga Kharif. 2012. “Likejacking”: Spammers hit social media. Business Week (May 2012). http://www.businessweek.com/articles/2012-05-24/likejacking-spammers-hit-social-media
33. Himabindu Lakkaraju, Julian J. McAuley, and Jure Leskovec. 2013. What’s in a name? Understanding the interplay between titles, content, and communities in social media.. In ICWSM. AAAI Press, Palo Alto, CA, 311--320.
34. Kyumin Lee, James Caverlee, Zhiyuan Cheng, and Daniel Z. Sui. 2014. Campaign extraction from social media. ACM Transactions on Intelligent Systems and Technology 5, 1, Article 9 (Jan. 2014), 28 pages. https://doi.org/10.1145/2542182.2542191
35. Kristina Lerman and Tad Hogg. 2014. Leveraging position bias to improve peer recommendation. PLoS One 9, 6 (2014), e98914. https://doi.org/10.1371/journal.pone.0098914
36. Jure Leskovec, Lars Backstrom, and Jon Kleinberg. 2009. Meme-tracking and the dynamics of the news cycle. In SIGKDD. ACM Press, New York, NY, 497--506. https://doi.org/10.1145/1557019.1557077
37. Jure Leskovec, Mary McGlohon, Christos Faloutsos, Natalie S. Glance, and Matthew Hurst. 2007. Patterns of cascading behavior in large blog graphs. In SDM. SIAM, Philadelphia, PA, 551--556. https://doi.org/10.1137/1.9781611972771.60
38. Jure Leskovec, Ajit Singh, and Jon Kleinberg. 2006. Patterns of influence in a recommendation network. In PAKDD. Springer, Berlin, 380--389. https://doi.org/10.1007/11731139_44
39. X. Li and L. M. Hitt. 2008. Self-selection and information role of online product reviews. Information Systems Research 19, 4 (Dec. 2008), 456--474. https://doi.org/10.1287/isre.1070.0154
40. Jan Lorenz, Heiko Rauhut, Frank Schweitzer, and Dirk Helbing. 2011. How social influence can undermine the wisdom of crowd effect. Proceedings of the National Academy of Sciences 108, 22 (2011), 9020--9025. https://doi.org/10.1073/pnas.1008636108
41. Yue Lu, Malu Castellanos, Umeshwar Dayal, and ChengXiang Zhai. 2011. Automatic construction of a context-aware sentiment lexicon. In WWW. ACM Press, New York, NY, 347. https://doi.org/10.1145/1963405.1963456
42. Yue Lu, Panayiotis Tsaparas, Alexandros Ntoulas, and Livia Polanyi. 2010. Exploiting social context for review quality prediction. In WWW. ACM Press, New York, NY, 691. https://doi.org/10.1145/1772690.1772761
43. Yue Lu and Chengxiang Zhai. 2008. Opinion integration through semi-supervised topic modeling. In WWW. ACM Press, New York, NY, 121. https://doi.org/10.1145/1367497.1367514
44. Yue Lu, ChengXiang Zhai, and Neel Sundaresan. 2009. Rated aspect summarization of short comments. In WWW. ACM Press, New York, NY, 131. https://doi.org/10.1145/1526709.1526728
45. Michael Luca. 2011. Reviews, reputation, and revenue: The case of yelp.com. Harvard Business Review 12, 016 (Oct. 2011). http://hbswk.hbs.edu/item/6833.html
46. Dipjyoti Majumdar and Arunava Sen. 2004. Ordinally Bayesian incentive compatible voting rules. Econometrica 72, 2 (March 2004), 523--540. https://doi.org/10.2307/3598911
47. Charles F. Manski. 1993. Identification of endogenous social effects: The reflection problem. Review of Economic Studies 60, 3 (1993), 531--542. http://www.jstor.org/stable/2298123
48. Benjamin Markines, Ciro Cattuto, and Filippo Menczer. 2009. Social spam detection. In AIRWeb. ACM Press, New York, NY, 41. https://doi.org/10.1145/1531914.1531924
49. Lev Muchnik, Sinan Aral, and Sean J. Taylor. 2013. Social influence bias: A randomized experiment. Science 341, 6146 (Aug. 2013), 647--651. https://doi.org/10.1126/science.1240466
50. Seth A. Myers and Jure Leskovec. 2012. Clash of the contagions: Cooperation and competition in information diffusion. In ICDM. IEEE, 539--548. https://doi.org/10.1109/ICDM.2012.159
51. Harold Nguyen. 2013. State of Social Media Spam. Technical Report. Nexgate Research.
52. Stanley Le Baron Payne. 2014. The Art of Asking Questions: Studies in Public Opinion, 3. Vol. 3. Princeton University Press, Princeton, NJ.
53. Owen Priestley. 2015. User experience methodology: From the physical to the emotional. Learned Publishing 28, 4 (2015), 317--320.
54. Matthew J. Salganik, Peter Sheridan Dodds, and Duncan J. Watts. 2006. Experimental study of inequality and unpredictability in an artificial cultural market. Science 311, 5762 (Feb. 2006), 854--856. https://doi.org/10.1126/science.1121066
55. Matthew J. Salganik and Duncan J. Watts. 2008. Leading the herd astray: An experimental study of self-fulfilling prophecies in an artificial cultural market. Social Psychology Quarterly 71, 4 (Jan. 2008), 338. https://doi.org/10.1177/019027250807100404
56. Mark Allen Satterthwaite. 1975. Strategy-proofness and arrow’s conditions: Existence and correspondence theorems for voting procedures and social welfare functions. Journal of Economic Theory 10, 2 (1975), 187--217. https://doi.org/10.1016/0022-0531(75)90050-2
57. Alan T. Sorensen. 2007. Bestseller lists and product variety. Journal of Industrial Economics 55, 4 (2007), 715--738. https://doi.org/10.1111/j.1467-6451.2007.00327.x
58. Greg Stoddard. 2015. Popularity and quality in social news aggregators: A study of reddit and hacker news. In WWW. IW3C2, Geneva, 815--818. https://doi.org/10.1145/2740908.2742470
59. James Surowiecki. 2005. The Wisdom of Crowds. Anchor, New York, NY. 336 pages.
60. Dan Tynan. 2012. Social spam is taking over the internet. IT World (April 2012). http://www.itworld.com/article/2832566/it-management/social-spam-is-taking-over-the-internet.html.
61. Trevor van Mierlo. 2014. The 1% rule in four digital health social networks: An observational study. Journal of Medical Internet Research 16, 2 (Feb. 2014), e33. https://doi.org/10.2196/jmir.2966
62. Tim Weninger, Thomas James Johnston, and Maria Glenski. 2015. Random voting effects in social-digital spaces: A case study of reddit post submissions. In Hypertext and Social Media. ACM, New York, NY, 293--297. https://doi.org/10.1145/2700171.2791054
63. Tim Weninger, Xihao Avi Zhu, and Jiawei Han. 2013. An exploration of discussion threads in social news sites: A case study of the reddit community. In ASONAM. ACM Press. https://doi.org/10.1145/2492517.2492646
64. Anita Williams Woolley, Christopher F. Chabris, Alex Pentland, Nada Hashmi, and Thomas W. Malone. 2010. Evidence for a collective intelligence factor in the performance of human groups. Science 330, 6004 (Oct. 2010), 686--688. https://doi.org/10.1126/science.1193147
65. Fang Wu and Bernardo A. Huberman. 2008. How public opinion forms. In Internet and Network Economics, Christos Papadimitriou and Shuzhong Zhang (Eds.). Lecture Notes in Computer Science, Vol. 5385. Springer, Berlin. https://doi.org/10.1007/978-3-540-92185-1_39
66. Xianchao Zhang, Shaoping Zhu, and Wenxin Liang. 2012. Detecting spam and promoting campaigns in the twitter social network. In ICDM. IEEE, 1194--1199. https://doi.org/10.1109/ICDM.2012.28
67. Feng Zhu and Xiaoquan (Michael) Zhang. 2010. Impact of online consumer reviews on sales: The moderating role of product and consumer characteristics. Journal of Marketing 74, 2 (March 2010), 133--148. http://journals.ama.org/doi/abs/10.1509/jmkg.74.2.133

Index Terms

• Human-centered computing

  • Collaborative and social computing

    • Collaborative and social computing systems and tools

      • Social networking sites

  • Human computer interaction (HCI)

    • Interaction paradigms

      • Collaborative interaction