Michael D. Ekstrand
F. Maxwell Harper
Martijn C. Willemsen
Joseph A. Konstan
ABSTRACT
Recent developments in user evaluation of recommender systems have brought forth powerful new tools for understanding what makes recommendations effective and useful. We apply these methods to understand how users evaluate recommendation lists for the purpose of selecting an algorithm for finding movies. This paper reports on an experiment in which we asked users to compare lists produced by three common collaborative filtering algorithms on the dimensions of novelty, diversity, accuracy, satisfaction, and degree of personalization, and to select a recommender that they would like to use in the future. We find that satisfaction is negatively dependent on novelty and positively dependent on diversity in this setting, and that satisfaction predicts the user's final selection. We also compare users' subjective perceptions of recommendation properties with objective measures of those same characteristics. To our knowledge, this is the first study that applies modern survey design and analysis techniques to a within-subjects, direct comparison study of recommender algorithms.
Supplemental Material
- D. Bollen, B. Knijnenburg, M. Willemsen, and M. Graus. Understanding Choice Overload in Recommender Systems. In Proc. ACM RecSys 2010. ACM, 2010, pp. 63--70. DOI: 10.1145/1864708.1864724. Google Scholar
Digital Library
- Ò. Celma and P. Herrera. A New Approach to Evaluating Novel Recommendations. In Proc. ACM RecSys 2008. ACM, 2008, pp. 179--186. DOI: 10.1145/1454008.1454038. Google ScholarDigital Library
- M. D. Ekstrand. Towards Recommender Engineering: Tools and Experiments in Recommender Differences. PhD thesis. Minneapolis, MN: University of Minnesota, 2014. 263 pp.Google Scholar
- M. Ekstrand, P. Kannan, J. Stemper, J. Butler, J. A. Konstan, and J. Riedl. Automatically Building Research Reading Lists. In Proc. ACM RecSys 2010. ACM, 2010, pp. 159--166. DOI: 10.1145/1864708.1864740. Google ScholarDigital Library
- M. Ekstrand, M. Ludwig, J. A. Konstan, and J. Riedl. Rethinking the Recommender Research Ecosystem: Reproducibility, Openness, and LensKit. In Proc. ACM RecSys 2011. ACM, 2011, pp. 133--140. DOI: 10.1145/2043932.2043958. Google ScholarDigital Library
- S. Funk. Netflix Update: Try This at Home. The Evolution of Cybernetics. 2006. URL: http://sifter.org/~simon/journal/20061211.html (visited on 04/08/2010).Google Scholar
- J. Herlocker, J. A. Konstan, and J. Riedl. An Empirical Analysis of Design Choices in Neighborhood-Based Collaborative Filtering Algorithms. Inf. Retr. 5(4):287--310, 2002. DOI: 10.1023/A:1020443909834. Google ScholarDigital Library
- J. Herlocker, J. A. Konstan, L. Terveen, and J. Riedl. Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1):5--53, 2004. DOI: 10.1145/963770.963772. Google ScholarDigital Library
- C. K. Hsee and J. Zhang. General Evaluability Theory. Persp. Psych. Sci. 5(4):343--355, 2010. DOI: 10.1177/1745691610374586.Google ScholarCross Ref
- R. B. Kline. Principles and Practice of Structural Equation Modeling. Guilford Press, New York, 1998. 354 pp.Google Scholar
- B. Knijnenburg, M. Willemsen, Z. Gantner, H. Soncu, and C. Newell. Explaining the User Experience of Recommender Systems. UMUAI, 22(4):441--504, 2012. DOI: 10.1007/s11257-011-9118-4. Google ScholarDigital Library
- S. McNee, N. Kapoor, and J. A. Konstan. Don't Look Stupid: Avoiding Pitfalls When Recommending Research Papers. In Proc. ACM CSCW 2006. CSCW '06. ACM, Banff, Alberta, Canada, 2006, p. 171. DOI: 10.1145/1180875.1180903. Google ScholarDigital Library
- S. McNee, J. Riedl, and J. A. Konstan. Being accurate is not enough: how accuracy metrics have hurt recommender systems. In Ext. Abs. ACM CHI 2010. ACM, Montréal, Québec, Canada, 2006, pp. 1097--1101. DOI: 10.1145/1125451.1125659. Google ScholarDigital Library
- S. McNee, J. Riedl, and J. A. Konstan. Making recommendations better: an analytic model for human-recommender interaction. In Ext. Abs. ACM CHI 2006. ACM, 2006, pp. 1103--1108. DOI: 10.1145/1125451.1125660. Google ScholarDigital Library
- A. Paterek. Improving regularized singular value decomposition for collaborative filtering. In KDD Cup and Workshop 2007, 2007.Google Scholar
- P. Pu, L. Chen, and R. Hu. A user-centric evaluation framework for recommender systems. In Proc. ACM RecSys 2011. In RecSys '11. ACM, 2011, pp. 157--164. DOI: 10.1145/2043932.2043962. Google ScholarDigital Library
- R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2014.Google Scholar
- Y. Rosseel. lavaan: An R Package for Structural Equation Modeling. J. Stat. Soft. 48(2):1--36, 2012.Google ScholarCross Ref
- B. Sarwar, G. Karypis, J. Konstan, and J. Reidl. Item-based collaborative filtering recommendation algorithms. In Proc. ACM WWW '01. ACM, 2001, pp. 285--295. DOI: 10.1145/371920.372071. Google ScholarDigital Library
- G. Shani and A. Gunawardana. Evaluating Recommendation Systems. In F. Ricci, L. Rokach, B. Shapira, and P. B. Kantor, editors, Recommender Systems Handbook, pp. 257--297. Springer, 2010.Google Scholar
- R. Torres, S. McNee, M. Abel, J. A. Konstan, and J. Riedl. Enhancing Digital Libraries with TechLens+. In Proc. ACM IEEE JCDL 2004. ACM, 2004, pp. 228--236. DOI: 10.1145/996350.996402. Google ScholarDigital Library
- S. Vargas and P. Castells. Rank and Relevance in Novelty and Diversity Metrics for Recommender Systems. In Proc. ACM RecSys 2011. ACM, 2011, pp. 109--116. DOI: 10.1145/2043932.2043955. Google ScholarDigital Library
- J. Vig, S. Sen, and J. Riedl. The Tag Genome: Encoding Community Knowledge to Support Novel Interaction. ACM Trans. Interact. Intell. Syst. 2(3):13:1--13:44, 2012. DOI: 10.1145/2362394.2362395. Google ScholarDigital Library
- M. C. Willemsen, M. P. Graus, and B. P. Knijnenburg. Understanding the Role of Latent Feature Diversification on Choice Difficulty and Satisfaction. Under review, 2014.Google Scholar
- M. Zhang and N. Hurley. Avoiding Monotony: Improving the Diversity of Recommendation Lists. In Proc. ACM RecSys 2008. ACM, 2008, pp. 123--130. DOI: 10.1145/1454008.1454030. Google ScholarDigital Library
- T. Zhou, Z. Kuscsik, J.-G. Liu, M. Medo, J. R. Wakeling, and Y.-C. Zhang. Solving the apparent diversity-accuracy dilemma of recommender systems. PNAS, 107(10):4511--4515, 2010. DOI: 10.1073/pnas.1000488107.Google ScholarCross Ref
- C.-N. Ziegler, S. McNee, J. A. Konstan, and G. Lausen. Improving recommendation lists through topic diversification. In Proc. ACM WWW 2005. ACM, 2005, pp. 22--32. DOI: 10.1145/1060745.1060754. Google ScholarDigital Library
Index Terms
- User perception of differences in recommender algorithms
Recommendations
Acquiring User Information Needs for Recommender Systems
WI-IAT '13: Proceedings of the 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT) - Volume 03Most recommender systems attempt to use collaborative filtering, content-based filtering or hybrid approach to recommend items to new users. Collaborative filtering recommends items to new users based on their similar neighbours, and content-based ...
Investigating serendipity in recommender systems based on real user feedback
SAC '18: Proceedings of the 33rd Annual ACM Symposium on Applied ComputingOver the past several years, research in recommender systems has emphasized the importance of serendipity, but there is still no consensus on the definition of this concept and whether serendipitous items should be recommended is still not a well-...
User Personality and User Satisfaction with Recommender Systems
In this study, we show that individual users' preferences for the level of diversity, popularity, and serendipity in recommendation lists cannot be inferred from their ratings alone. We demonstrate that we can extract strong signals about individual ...
Comments