Metacognitive Judgments in Searching as Learning (SAL) Tasks: Insights on (Mis-) Calibration, Multimedia Usage, and Confidence

ABSTRACT

Metacognitive self-assessments of one's learning performance (calibration) are important elements of Searching as Learning (SAL) tasks. In this SAL study, N = 115 participants were asked to learn for up to 30 minutes about the formation of thunderstorms and lightning by using any suitable internet resources (including multimedia resources). Participants rated their performance in comparison to other participants (placement), estimated the percentage of correct answers (estimation), and indicated their confidence in the correctness of their answers (confidence) in a multiple-choice knowledge test that was filled in one week before (T1) and directly after (T2) the learning phase. Participants furthermore rated the 'familiarity' of terms that do or do not exist in the context of meteorology (overclaiming). Learners tended to underestimate their performance at T1 and there were indicators of a potential Dunning-Kruger effect. Overall, placement and estimation ratings tended to be more accurate at T2. Surprisingly, confidence ratings increased approximately equally for correct as well as incorrect answers. A propensity for overclaiming was positively correlated with most confidence measures and the amount of time learners spent on YouTube was correlated to lower confidence scores. Implications for the design of SAL tasks and SAL studies are discussed.

References

1. Pertti Vakkari. 2016. Searching as learning: A systematization based on literature. Journal of Information Science, 42(1), 7--18. http://doi.org/10.1177/0165551515615833
2. Ran Yu, Ujwal Gadiraju, Peter Holtz, Markus Rokicki, Philipp Kemkes, and Stefan Dietze. 2018. (June) Predicting user knowledge gain in informational search sessions. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (pp. 75--84). ACM. http://doi.org/10.1145/3209978.3210064
3. Ujwal Gadiraju, Ran Yu, Stefan Dietze, and Peter Holtz. 2018. (March). Analyzing knowledge gain of users in informational search sessions on the web. In Proceedings of the 2018 Conference on Human Information Interaction & Retrieval (pp. 2--11). ACM. http://doi.org/10.1145/3176349.3176381
4. Nilavra Bhattacharya, and Jacek Gwizdka, 2019 (March). Measuring Learning During Search: Differences in Interactions, Eye-Gaze, and Semantic Similarity to Expert Knowledge. In Proceedings of the 2019 Conference on Human Information Interaction and Retrieval (pp. 63--71). http://doi.org/10.1145/3295750.3298926
5. Anett Hoppe, Peter Holtz, Yvonne Kammerer, Ran Yu, Stefan Dietze, and Ralph Ewerth. 2018. Current Challenges for Studying Search as Learning Processes. In Proceedings of Learning and Education with Web Data, Amsterdam, Netherlands, May 2018 (LILE2018), 4 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn
6. Saskia Brand-Gruwel, Iwan Wopereis, and Amber Walraven. 2009. A descriptive model of information problem solving while using internet. Computers & Education, 53(4), 1207--1217. https://doi.org/10.1016/j.compedu.2009.06.004
7. John H. Flavell. 1979. Metacognition and cognitive monitoring: A new area of cognitive--developmental inquiry. American psychologist, 34(10), 906--911. http://dx.doi.org/10.1037/0003-066X.34.10.906
8. Thomas O. Nelson, and John Dunlosky. 1992. How shall we explain the delayed-judgment-of-learning effect? Psychological Science, 3(5), 317--319. https://doi.org/10.1111/j.1467--9280.1992.tb00681.x
9. Roger Azevedo. 2005. Computer environments as metacognitive tools for enhancing learning. Educational Psychologist, 193--197. https://doi.org/10.1207/s15326985ep4004_1
10. Richard E. Mayer. 2014. Cognitive theory of multimedia learning. In The Cambridge Handbook of Multimedia Learning, Second Edition. https://doi.org/10.1017/CBO9781139547369.005
11. Don A. Moore, and Paul J. Healy. 2008. The trouble with overconfidence. Psychological review, 115(2), 502--517. https://doi.org/10.1037/0033--295X.115.2.502
12. John Dunlosky, and Katherine A. Rawson. 2012. Overconfidence produces underachievement: Inaccurate self evaluations undermine students' learning and retention. Learning and Instruction, 22(4), 271--280. https://doi.org/10.1016/j.learninstruc.2011.08.003
13. Jeri L. Little, and Mark A. McDaniel. 2015. Individual differences in category learning: Memorization versus rule abstraction. Memory & cognition, 43(2), 283--297. https://doi.org/10.3758/s13421-014-0475--1
14. Rakefet Ackerman, and Morris Goldsmith. 2011. Metacognitive regulation of text learning. Journal of Experimental Psychology: Applied, 17, 18--32. http://dx.doi.org/10.1037/a0022086
15. Rakefet Ackerman, and Tirza Lauterman. 2012. Taking reading comprehension exams on screen or on paper? A metacognitive analysis of learning texts under time pressure. Computers in Human Behavior, 28, 1816--1828. https://doi.org/10.1016/j.chb.2012.04.023
16. Adam L. Alter, and Daniel M. Oppenheimer. 2009. Uniting the tribes of fluency to form a metacognitive nation. Personality and social psychology review, 13(3), 219--235. https://doi.org/10.1177/1088868309341564
17. Allison J. Jaeger, and Jennifer Wiley. 2014. Do illustrations help or harm metacomprehension accuracy? Learning and Instruction, 34, 58--73. http://dx.doi.org/10.1016/j.learninstruc.2014.08.002
18. Alwine Lenzner, Wolfgang Schnotz, and Andreas Müller. 2013. The role of decorative pictures in learning. Instructional Science, 41(5), 811--831. http://doi.org/10.1007/s11251-012--9256-z
19. Michael Kardas, and Ed O'Brien. 2018. Easier seen than done: Merely watching others perform can foster an illusion of skill acquisition. Psychological science, Advance online publication. https://doi.org/10.1177/0956797617740646
20. David Dunning. 2011. The Dunning--Kruger effect: On being ignorant of one's own ignorance. In Advances in experimental social psychology (Vol. 44, pp. 247--296). Academic Press. http://dx.doi.org/10.1016/B978-0--12--385522-0.00005--6
21. Claudio Aqueveque. 2018. Ignorant experts and erudite novices: Exploring the Dunning-Kruger effect in wine consumers. Food Qual. Prefer. 65, (2018), 181--184. https://doi.org/10.1016/j.foodqual.2017.12.007
22. Ian G. Anson. 2018. Partisanship, Political Knowledge, and the Dunning-Kruger Effect. Polit. Psychol. 39, 5 (2018), 1173--1192. https://doi.org/10.1111/pops.12490
23. Shirley Gibbs, Kevin Moore, Gary Steel, and Alan McKinnon. 2017. The Dunning-Kruger Effect in a workplace computing setting. Comput. Human Behav. 72, (2017), 589--595. https://doi.org/10.1016/j.chb.2016.12.084
24. Gordon Pennycook, Robert M. Ross, Derek J. Koehler, and Jonathan A. Fugelsang. 2017. Dunning--Kruger effects in reasoning: Theoretical implications of the failure to recognize incompetence. Psychon. Bull. Rev. 24, 6 (December 2017), 1774--1784. https://doi.org/10.3758/s13423-017--1242--7
25. Barbara Coombes. 2009. Generation Y: Are they really digital natives or more like digital refugees? Synergy 7, 1 (2009), 31--40.
26. Shinichi Monoi, Nancy O'Hanlon, and Karen R. Diaz. 2005. Online searching skills: Development of an inventory to assess self-efficacy. J. Acad. Librariansh. 31, 2 (2005), 98--105. https://doi.org/10.1016/j.acalib.2004.12.005
27. Delroy Paulhus, and Patrick Dubois. (2014). Application of the Overclaiming Technique to Scholastic Assessment. Educational and Psychological Measurement, 74(6), 975--990. https://doi.org/10.1177/0013164414536184
28. Stav Atir, Emily Rosenzweig, and David Dunning. 2015. When Knowledge Knows No Bounds: Self-Perceived Expertise Predicts Claims of Impossible Knowledge. Psychol. Sci. 26, 8 (2015), 1295--1303. https://doi.org/10.1177/0956797615588195
29. Stefan Engeser. 2005. Messung des expliziten Leistungsmotivs: Kurzform der Achievement Motives Scale. Retrieved 10.08.2010 from http://www.uni-trier.de/ fileadmin/fb1/prof/PSY/PGA/bilder/Engeser__2005__Kurzform_der_AMS.pdf.
30. Shane Frederick. 2005. Cognitive Reflection and Decision Making. J. Econ. Perspect. 19, 4 (November 2005), 25--42. https://doi.org/10.1257/089533005775196732
31. Gerald Matthews, Sian Campbell, Shona Falconer, Lucy A. Joyner, Jane Huggins, Kirby Gilliland, . . . Joel S. Warm. 2002. Fundamental dimensions of subjective state in performance settings: Task engagement, distress, and worry. Emotion, 2(4), 315--340. http://dx.doi.org/10.1037/1528--3542.2.4.315
32. Richard E. Mayer and Roxana Moreno. 1998. A split-attention effect in multimedia learning: Evidence for dual processing systems in working memory. Journal of Educational Psychology 90, 2: 312--320. https://doi.org/10.1037/0022-0663.90.2.312
33. Florian Schmidt-Weigand and Katharina Scheiter. 2011. The role of spatial descriptions in learning from multimedia. Computers in Human Behavior 27, 1: 22--28. https://doi.org/10.1016/j.chb.2010.05.007
34. Lin-Miao Lin, Dewayne Moore, and Karen M. Zabrucky. 2001. An assessment of students' calibration of comprehension and calibration of performance using multiple measures. Reading Psychology, 22(2), 111--128. http://dx.doi.org/10.1080/027027101300213083
35. Priscilla Bell and David Volckmann. 2011. Knowledge surveys in general chemistry: Confidence, overconfidence, and performance. J. Chem. Educ. (2011). https://doi.org/10.1021/ed100328c
36. Marian Krajc and Andreas Ortmann. 2008. Are the unskilled really that unaware? An alternative explanation. Journal of Economic Psychology, 29, 724--738. https://doi.org/10.1016/j.joep.2007.12.006
37. Thomas Schlösser, David Dunning, Kerri L. Johnson, and Justin Kruger. 2013. How unaware are the unskilled? Empirical tests of the "signal extraction" counterexplanation for the Dunning-Kruger effect in self-evaluation of performance. J. Econ. Psychol. 39, (2013), 85--100. https://doi.org/10.1016/j.joep.2013.07.004
38. Darwin P. Hunt. 2003. The concept of knowledge and how to measure it. Journal of Intellectual Capital, Vol. 4 No. 1, pp. 100--113. https://doi.org/10.1108/14691930310455414
39. Asher Koriat. 1997. Monitoring one's own knowledge during study: A cue-utilization approach to judgments of learning. Journal of Experimental Psychology: General, 126(4), 349--370. http://dx.doi.org/10.1037/0096--3445.126.4.349
40. Christopher K. Hertzog, John Dunlosky, Ann E. Robinson, and Daniel Peter Kidder. 2003. Encoding fluency is a cue used for judgments about learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 29(1), 22--34. http://dx.doi.org/10.1037/0278--7393.29.1.22
41. Peter Gerjets, Yvonne Kammerer, and Benita Werner. 2011. Measuring spontaneous and instructed evaluation processes during Web search: Integrating concurrent thinking-aloud protocols and eye-tracking data. Learning and Instruction, 21(2), 220--231. http://dx.doi.org/10.1016/j.learninstruc.2010.02.005
42. Robert A. Bjork, John Dunlosky, and Nate Kornell. 2013. Self-Regulated Learning: Beliefs, Techniques, and Illusions. Annual Review of Psychology, 64,417--444. https://doi.org/10.1146/annurev-psych-113011--143823
43. Joachim Kimmerle, Johannes Moskaliuk, Aileen Oeberst, and Ulrike Cress. 2015. Learning and collective knowledge construction with social media: A process-oriented perspective. Educational Psychologist, 50(2), 120--137. https://doi.org/10.1080/00461520.2015.1036273
44. Mathieu d'Aquin, Alessandro Adamou, Stefan Dietze, Besnik Fetahu, Ujwal Gadiraju, Ilire Hasani-Mavriqi, Peter Holtz, Joachim Kimmerle, Dominik Kowald, Elisabeth Lex, and Susana López-Sola. 2017 (September). AFEL: Towards Measuring Online Activities Contributions to Self-directed Learning. In ARTEL@ EC-TEL. http://ceur-ws.org/Vol-1997/paper5.pdf
45. Mathieu d'Aquin, Dominik Kowald, Angela Fessl, Elisabeth Lex, and Stefan Thalmann. 2018, (April). Afel-analytics for everyday learning. In Companion of the The Web Conference 2018 on The Web Conference 2018 (pp. 439--440). International World Wide Web Conferences Steering Committee. https://doi.org/10.1145/3184558.3186206

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods

      • User models