Market2Dish: A Health-aware Food Recommendation System

ABSTRACT

In order to help people develop healthy eating habits, we present a personalized health-aware food recommendation system, calledMarket2Dish. Market2Dish could recognize the ingredients in the micro-videos taken from the market, characterize the health conditions of users from their social media accounts, and ultimately recommend users with the personalized healthy foods. Specifically, we employ a word-class interaction based text classification model to learn the fine-grained similarity between sparse health features on the social media platforms and pre-defined health concepts, and then a category-aware hierarchical memory network based recommender is introduced to learn the user-recipe interactions for better food recommendations. Moreover, we demonstrate this system as an online app for real-time interactions with users.

References

1. Yongsheng An, Yu Cao, Jingjing Chen, Chong-Wah Ngo, Jia Jia, Huanbo Luan, and Tat-Seng Chua. 2017. PIC2DISH: A customized cooking assistant system. In Proceedings of the 25th International Conference on Multimedia. ACM, 1269--1273.
2. Jing-Jing Chen, Chong-Wah Ngo, Fu-Li Feng, and Tat-Seng Chua. 2018. Deep understanding of cooking procedure for cross-modal recipe retrieval. In Proceedings of the 26th International Conference on Multimedia. ACM, 1020--1028.
3. Chen Cui, Wenjie Wang, Xuemeng Song, Minlie Huang, Xin-Shun Xu, and Liqiang Nie. 2019. User Attention-guided Multimodal Dialog Systems. In Proceedings of the 42Nd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 445--454.
4. David Elsweiler, Christoph Trattner, and Morgan Harvey. 2017. Exploiting food choice biases for healthier recipe recommendation. In Proceedings of the 40th International SIGIR Conference on Research and Development in Information Retrieval. ACM, 575--584.
5. Aleksandr Farseev and Tat-Seng Chua. 2017. Tweet can be fit: Integrating data from wearable sensors and multiple social networks for wellness profile learning. ACM Transactions on Information Systems , Vol. 35, 4 (2017), 42.
6. Mouzhi Ge, Francesco Ricci, and David Massimo. 2015. Health-aware food recommender system. In Proceedings of the 9th ACM Conference on Recommender Systems. ACM, 333--334.
7. Hokuto Kagaya, Kiyoharu Aizawa, and Makoto Ogawa. 2014. Food detection and recognition using convolutional neural network. In Proceedings of the 22nd International Conference on Multimedia. ACM, 1085--1088.
8. Yoshiyuki Kawano and Keiji Yanai. 2014. Foodcam-256: a large-scale real-time mobile food recognition system employing high-dimensional features and compression of classifier weights. In Proceedings of the 22nd International Conference on Multimedia. ACM, 761--762.
9. Keigo Kitamura, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2008. Food log by analyzing food images. In Proceedings of the 16th International Conference on Multimedia. ACM, 999--1000.
10. Tomasz Kusmierczyk, Christoph Trattner, and Kjetil Nørvåg. 2015. Temporality in online food recipe consumption and production. In Proceedings of the 24th International Conference on World Wide Web. ACM, 55--56.
11. Weiqing Min, Shuqiang Jiang, Shuhui Wang, Jitao Sang, and Shuhuan Mei. 2017. A delicious recipe analysis framework for exploring multi-modal recipes with various attributes. In Proceedings of the 25th International Conference on Multimedia. ACM, 402--410.
12. Ferda Ofli, Yusuf Aytar, Ingmar Weber, Raggi Al Hammouri, and Antonio Torralba. 2017. The food perception gap on instagram and its relation to health. In Proceedings of the 26th International Conference on World Wide Web. ACM, 509--518.
13. Satoshi Sanjo and Marie Katsurai. 2017. Recipe popularity prediction with deep visual-semantic fusion. In Proceedings of the Conference on Information and Knowledge Management. ACM, 2279--2282.
14. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, 2818--2826.
15. Christoph Trattner and David Elsweiler. 2017. Investigating the healthiness of internet-sourced recipes: implications for meal planning and recommender systems. In Proceedings of the 26th International Conference on World Wide Web. ACM, 489--498.
16. Liping Wang, Qing Li, Na Li, Guozhu Dong, and Yu Yang. 2008. Substructure similarity measurement in Chinese recipes. In Proceedings of the 17th International Conference on World Wide Web. ACM, 979--988.
17. Shuohang Wang and Jing Jiang. 2016. Learning natural language inference with LSTM. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics . ACL, 1442--1451.
18. Wenjie Wang, Minlie Huang, Xin-Shun Xu, Fumin Shen, and Liqiang Nie. 2018. Chat more: Deepening and widening the chatting topic via a deep model. In Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 255--264.
19. Hui Wu, Michele Merler, Rosario Uceda-Sosa, and John R Smith. 2016. Learning to make better mistakes: Semantics-aware visual food recognition. In Proceedings of the 24th International Conference on Multimedia. ACM, 172--176.
20. Shulin Yang, Mei Chen, Dean Pomerleau, and Rahul Sukthankar. 2010. Food recognition using statistics of pairwise local features. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, 2249--2256.

Index Terms

• Applied computing

  • Life and medical sciences

    • Health care information systems

• Information systems

  • Information retrieval

    • Retrieval tasks and goals

      • Recommender systems