Augmenting human memory using personal lifelogs

ABSTRACT

Memory is a key human facility to support life activities, including social interactions, life management and problem solving. Unfortunately, our memory is not perfect. Normal individuals will have occasional memory problems which can be frustrating, while those with memory impairments can often experience a greatly reduced quality of life. Augmenting memory has the potential to make normal individuals more effective, and those with significant memory problems to have a higher general quality of life. Current technologies are now making it possible to automatically capture and store daily life experiences over an extended period, potentially even over a lifetime. This type of data collection, often referred to as a personal life log (PLL), can include data such as continuously captured pictures or videos from a first person perspective, scanned copies of archival material such as books, electronic documents read or created, and emails and SMS messages sent and received, along with context data of time of capture and access and location via GPS sensors. PLLs offer the potential for memory augmentation. Existing work on PLLs has focused on the technologies of data capture and retrieval, but little work has been done to explore how these captured data and retrieval techniques can be applied to actual use by normal people in supporting their memory. In this paper, we explore the needs for augmenting human memory from normal people based on the psychology literature on mechanisms about memory problems, and discuss the possible functions that PLLs can provide to support these memory augmentation needs. Based on this, we also suggest guidelines for data for capture, retrieval needs and computer-based interface design. Finally we introduce our work-in-process prototype PLL search system in the iCLIPS project to give an example of augmenting human memory with PLLs and computer based interfaces.

References

1. Byrne, D. and Jones, G., "Creating Stories for Reflection from Multimodal Lifelog Content: An Initial Investigation," in Designing for Reflection on Experience, Workshop at CHI 2009, Boston, MA, U.S.A., 2009.
2. Berry, E., et al., "The use of a wearable camera, SenseCam, as a pictorial diary to improve autobiographical memory in a patient with limbic encephalitis: A preliminary report," Neuropsychological Rehabilitation: An International Journal, vol. 17, pp. 582--601, 2007.
3. Bell, G. and Gemmell, J., Total Recall: Dutton 2009.
4. Kelly, L. and Jones, G., "Examining the Utility of Affective Response in Search of Personal Lifelogs," in 5th Workshop on Emotion in HCI, British HCI Conference, Cambridge, U.K, 2009.
5. Devaul, R. W., "The memory glasses: wearable computing for just-in-time memory support," Massachusetts Institute of Technology, 2004.
6. Lee, H., et al., "Constructing a SenseCam visual diary as a media process," Multimedia Systems, vol. 14, pp. 341--349, 2008.
7. Kawamura, T., et al., "Ubiquitous Memories: a memory externalization system using physical objects," Personal Ubiquitous Comput., vol. 11, pp. 287--298, 2007.
8. Farringdon, J. and Oni, V., "Visual Augmented Memory (VAM)," presented at the Proceedings of the 4th IEEE International Symposium on Wearable Computers, 2000.
9. Vemuri, S., et al., "iRemember: a personal, long-term memory prosthesis," presented at the Proceedings of the 3rd ACM workshop on Continuous archival and retrival of personal experences, Santa Barbara, California, USA, 2006.
10. Lamming, M. and Flynn, M., "Forget-me-not: intimate computing in support of human memory," in Proceedings FRIEND21 Symposium on Next Generation Human Interfaces, Tokyo Japan, 1994.
11. Byrne, D., et al., "Multiple Multimodal Mobile Devices: Lessons Learned from Engineering Lifelog Solutions," in Handbook of Research on Mobile Software Engineering: Design, Implementation and Emergent Applications, ed: IGI Publishing, 2010.
12. Schacter, D. L., The seven sins of memory. Boston: Houghton Mifflin, 2001.
13. Baddeley, A., "The episodic buffer: a new component of working memory?," Trends in Cognitive Sciences, vol. 4, pp. 417--423, 2000.
14. Baddeley, A. D., et al., "Working Memory," in Psychology of Learning and Motivation. vol. Volume 8, ed: Academic Press, 1974, pp. 47--89.
15. Anderson, J. and Bower, G., Human associative memory: A brief edition: Lawrence Erlbaum, 1980.
16. Loftus, E., "Memory Distortion and False Memory Creation," vol. 24, ed, 1996, pp. 281--295.
17. Elsweiler, D., et al., "Towards memory supporting personal information management tools," J. Am. Soc. Inf. Sci. Technol., vol. 58, pp. 924--946, 2007.
18. Lamming, M., et al., "The Design of a Human Memory Prosthesis," The Computer Journal, vol. 37, pp. 153--163, March 1, 1994 1994.
19. Hayes, G. R., et al., "The Personal Audio Loop: Designing a Ubiquitous Audio-Based Memory Aid," ed, 2004, pp. 168--179.
20. Rankin, J. L. and Collins, M., "Adult Age Differences in Memory Elaboration," J Gerontol, vol. 40, pp. 451--458, July 1, 1985 1985.
21. Eldridge, M. A., et al., "Autobiographical memory and daily schemas at work," Memory, vol. 2, pp. 51--74, 1994.
22. Hirose, Y., "iFlashBack: A Wearable Electronic Mnemonics to Retain Episodic Memory Visually Real by Video Aided Rehearsal," presented at the Proceedings of the 2005 IEEE Conference 2005 on Virtual Reality, 2005.
23. Sellen, A. J., et al., "Do life-logging technologies support memory for the past?: an experimental study using sensecam," presented at the Proceedings of the SIGCHI conference on Human factors in computing systems, San Jose, California, USA, 2007.
24. Mann, S., "Continuous lifelong capture of personal experience with EyeTap," presented at the Proceedings of the the 1st ACM workshop on Continuous archival and retrieval of personal experiences, New York, New York, USA, 2004.
25. Helmes, J., et al., "The other brother: re-experiencing spontaneous moments from domestic life," presented at the Proceedings of the 3rd International Conference on Tangible and Embedded Interaction, Cambridge, United Kingdom, 2009.
26. Tulving, E., Elements of episodic memory: Oxford University Press New York, 1983.
27. Kelly, L. and Jones, G. J. F., "Venturing into the labyrinth: the information retrieval challenge of human digital memories," presented at the Workshop on Supporting Human Memory with Interactive Systems, Lancaster, UK, 2007.
28. Kelly, L., et al., "A study of remembered context for information access from personal digital archives," presented at the Proceedings of the second international symposium on Information interaction in context, London, United Kingdom, 2008.
29. Rhodes, B. J., "The wearable remembrance agent: a system for augmented memory," presented at the Proceedings of the 1st IEEE International Symposium on Wearable Computers, 1997.
30. Robertson, S., et al., "Simple BM25 extension to multiple weighted fields," presented at the Proceedings of the thirteenth ACM international conference on Information and knowledge management, Washington, D.C., USA, 2004.
31. Kim, J., et al., "A Probabilistic Retrieval Model for Semistructured Data," presented at the Proceedings of the 31th European Conference on IR Research on Advances in Information Retrieval, Toulouse, France, 2009.
32. Gemmell, J., et al., "Passive capture and ensuing issues for a personal lifetime store," presented at the Proceedings of the the 1st ACM workshop on Continuous archival and retrieval of personal experiences, New York, New York, USA, 2004.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

• Information systems

  • Information retrieval