Personal Knowledge Base Construction from Text-based Lifelogs

ABSTRACT

Previous work on lifelogging focuses on life event extraction from image, audio, and video data via wearable sensors. In contrast to wearing an extra camera to record daily life, people are used to log their life on social media platforms. In this paper, we aim to extract life events from textual data shared on Twitter and construct personal knowledge bases of individuals. The issues to be tackled include (1) not all text descriptions are related to life events, (2) life events in a text description can be expressed explicitly or implicitly, (3) the predicates in the implicit events are often absent, and (4) the mapping from natural language predicates to knowledge base relations may be ambiguous. A joint learning approach is proposed to detect life events in tweets and extract event components including subjects, predicates, objects, and time expressions. Finally, the extracted information is transformed to knowledge base facts. The evaluation is performed on a collection of lifelogs from 18 Twitter users. Experimental results show our proposed system is effective in life event extraction, and the constructed personal knowledge bases are expected to be useful to memory recall applications.

References

1. Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary G. Ives. 2007. Dbpedia: A Nucleus for a Web of Open Data. In Proceedings of the 6th International the Semantic Web and 2nd Asian Conference on Asian Semantic Web Conference, 722--735.
2. Kurt D. Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A Collaboratively Created Graph Database for Structuring Human Knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, 1247--1250.
3. Richard Caruana. 1993. Multitask learning: a knowledge based source of inductive bias. In Proceedings of the Tenth International Conference on Machine Learning.
4. Smitashree Choudhury and Harith Alani. 2015. Detecting Presence of Personal Events in Twitter Streams. In Proceedings of the International Conference on Social Informatics, 157--166.
5. Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jian Zhao. 2015. Event extraction via dynamic multi-pooling convolutional neural networks. In Proceedings of ACL, 167--176.
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
7. Duc-Tien Dang-Nguyen, Luca Piras, Michael Riegler, Liting Zhou, Mathias Lux, and Cathal Gurrin. 2018. Overview of ImageCLEFlifelog 2018: Daily Living Understanding and Lifelog Moment Retrieval. In: CLEF 2018 Working Notes. CEUR Workshop Proceedings, CEURWS.org http://ceur-ws.org, Avignon, France.
8. Thomas Dickinson, Miriam Fernández, Lisa A. Thomas, Paul Mulholland, Pamela Briggs, and Harith Alani. 2016. Identifying Important Life Events from Twitter Using Semantic and Syntactic Patterns. In Proceedings of the 15th International Conference WWW, 143--150.
9. Rowanne Fleck and Geraldine Fitzpatrick. 2009. Teachers' and tutors' social reflection around SenseCam images. In International Journal of Human-Computer Studies, 67(12): 1024--1036.
10. Jenny Rose Finkel, Christopher D. Manning, and Andrew Y. Ng. 2006. Solving the problem of cascading errors: approximate Bayesian inference for linguistic annotation pipelines. In Proceedings of EMNLP, 618--626.
11. Cathal Gurrin, Hideo Joho, Frank Hopfgartner, Liting Zhou, and Rami Albatal. 2016. Overview of ntcir lifelog task. In Proceedings of the 11th NTCIR Conference on Evaluation of Information Access Technologies, NTCIR-12. National Center of Sciences.
12. Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In Proceedings of ICASSP 2013, 6645--6649.
13. Alex Graves and Jürgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks, 18(5--6), 602--610.
14. Eben Harrell. 2010. Remains of the day: can a new device help amnesia patients outsource memory? Time Magazine, 46--51.
15. Richard Harper, Dave W. Randall, Nicola Smyth, Cara Evans, L. Heledd, Ronald E. Moore. 2008. The Past is a Different Place: They Do Things Differently There. In Proceedings of the 7th ACM conference on Designing interactive systems, 271--280.
16. Steve Hodges, Lyndsay Williams, Emma Berry, Shahram Izadi, James Srinivasan, Alex Butler, Gavin Smyth, Narinder Kapur, and Ken Woodberry. 2006. SenseCam: A retrospective memory aid. In Proceedings of International Conference on Ubiquitous Computing, 177--193.
17. Sen Hu, Lei Zou, and Xinbo Zhang. 2018. A State-transition Framework to Answer Complex Questions over Knowledge Base. In Proceedings of EMNLP, 2098--2108.
18. Alan Jackoway, Hanan Samet, and Jagan Sankaranarayanan. 2012. Identification of live news events using twitter. In Proceedings of the 3rd ACM SIGSPATIAL International Workshop on Location-Based Social Networks, 25--32.
19. Martin Källström. 2013. Lifelogging camera: the narrative clip. Retrieved from http://getnarrative.com/
20. Jacqueline Kerr, Simon J. Marshall, Suneeta Godbole, Jacqueline Chen, Amanda Legge, Aiden R. Doherty, Paul Richard Kelly, Melody Oliver, Hannah Badland, Charlie Foster. 2013. Using the SenseCam to Improve Classifications of Sedentary Behaviour in Free-Living Settings. In American journal of preventive medicine, 44(3): 290--296.
21. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of NAACL-HLT, 260--270.
22. Jiwei Li and Claire Cardie. 2014, Timeline Generation: Tracking individuals on Twitter. In Proceedings of the 23rd international conference on World wide web, 643--652.
23. Chin-Ho Lin, Hen-Hsen Huang, and Hsin-Hsi Chen. 2018. Learning to Map Natural Language Statements into Knowledge Base Representations for Knowledge Base Construction. In Proceedings of LREC.
24. John D. Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data.
25. Jiwei Li, Alan Ritter, Claire Cardie, and Eduard Hovy. 2014. Major Life Event Extraction from Twitter based on Congratulations/Condolences Speech Acts. In Proceedings of the EMNLP, 1997--2007.
26. Davis Liang and Yan Shu. 2017. Deep Automated Multi-task Learning. In Proceedings of the The 8th International Joint Conference on Natural Language Processing, 55--60.
27. Xueliang Liu, Raphaël Troncy, and Benoit Huet. 2011. Using social media to identify events. In Proceedings of the 3rd ACM SIGMM international workshop on Social media, 3--8.
28. Takuya Maekawa. 2013. A sensor device for automatic food lifelogging that is embedded in home ceiling light: a preliminary investigation. In Proceedings of the 7th International Conference on Pervasive Computing Technologies for Healthcare, 405--407.
29. Rajesh Elara Mohan, Lee Hyowon, K. S. Jaichandar, and Carlos Acosta Antonio Calderon. 2012. LifeVision: Integrating heart rate sensing in lifelogging camera for accurate risk diagnosis for the elderly. In Proceedings of the 6th International Conference on Rehabilitation Engineering & Assistive Technology, 35.
30. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. 2014, The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, 55--60.
31. Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grishman. 2016. Joint event extraction via recurrent neural networks. In Proceedings of NAACL, 300--309.
32. Thien Huu Nguyen and Ralph Grishman. 2015. Event detection and domain adaptation with convolutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, 365--371.
33. Mitsuo Nohara, Nobuhide Kotsuka, Masayuki Hashimoto, and Hiroki Horiuchi. 2010. A study on food-log application to a medical-care consult via telecommunications. In Proceedings of the 16th International Conference on Virtual Systems and Multimedia, 88--91.
34. Gillian O'Loughlin, Sarah Jane Cullen, Adrian McGoldrick, Siobhán O'Connor, Richard J. Blain, Shane O'Malley, and Giles D. Warrington. 2013, Using a Wearable Camera to Increase the Accuracy of Dietary Analysis. In American journal of preventive medicine, 44(3): 290--296.
35. Juan José Soler, Fernando Cuartero, and Manuel Roblizo. 2012, Twitter as a tool for predicting elections results. In Proceedings of the International Conference on Advances in Social Networks Analysis and Mining, 1194--1200.
36. Anders Søgaard and Yoav Goldberg. 2016. Deep multi-task learning with low level tasks supervised at lower layers. In Proceedings of ACL, 231--235.
37. Takeshi Sakaki, Makoto Okazaki, and Yutaka Matsuo. 2010. Earthquake shakes twitter users: real-time event detection by social sensors. In Proceedings of the 19th international conference on World wide web, 851--860.
38. Krishna Chaitanya Sanagavarapu, Alakananda Vempala, and Eduardo Blanco. 2017. Determining Whether and When People Participate in the Events They Tweet About. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. ACL, 641--646.
39. Tsung-Han Yang, Hen-Hsen Huang, An-Zi Yen, and Hsin-Hsi Chen. 2018, Transfer of Frames from English FrameNet to Construct Chinese FrameNet: A Bilingual Corpus-Based Approach. In Proceedings of LREC.
40. Xuchen Yao, and Benjamin Van Durme. 2014. Information extraction over structured data: Question answering with freebase. In Proceedings of ACL, 956--966.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Natural language processing

      • Information extraction

• Information systems

  • World Wide Web

    • Web searching and information discovery

      • Personalization