research-article

A multisensory approach for the design of food and drink enhancing sonic systems

Authors:
Carlos Velasco

BI Norwegian Business School & University of Oxford, Nydalsveien, Oslo, Norway

BI Norwegian Business School & University of Oxford, Nydalsveien, Oslo, Norway
View Profile

,
Felipe Reinoso Carvalho

Vrije Universiteit Brussel & KU Leuven VUB/ETRO - Pleinlaan, Brussels

Vrije Universiteit Brussel & KU Leuven VUB/ETRO - Pleinlaan, Brussels
View Profile

,
Olivia Petit

INSEEC Business School, Quai de Bacalan, Bordeaux, France

INSEEC Business School, Quai de Bacalan, Bordeaux, France
View Profile

,
Anton Nijholt

University of Twente, Human Media Interaction, Enschede, The Netherlands

University of Twente, Human Media Interaction, Enschede, The Netherlands
View Profile

MHFI '16: Proceedings of the 1st Workshop on Multi-sensorial Approaches to Human-Food InteractionNovember 2016Article No.: 7Pages 1–7https://doi.org/10.1145/3007577.3007578

Published:16 November 2016Publication History

MHFI '16: Proceedings of the 1st Workshop on Multi-sensorial Approaches to Human-Food Interaction

Pages 1–7

ABSTRACT

Everyday eating and drinking experiences involve multiple, interrelated, sensory inputs. However, when it comes to human-food interaction design (HFI) research, certain senses have received more attention than others have. Here, we focus on audition, a sense that has received limited attention in such context. In particular, we highlight the role of food/drink-related eating sounds, as a potential input for human-food interaction design. We review some of the few systems that have built on such sounds within food and drink contexts. We also present a multisensory design framework and discuss how the systematic connections that exist between the senses may provide some guidelines for the integration of eating sounds in HFI design. Finally, we present some key prospects that we foresee for research in technology design in HFI.

References

Auvray, M. and Spence, C. 2008. The multisensory perception of flavor. Conscious. Cogn. 17(3), 1016--1031. http://dx.doi.org/10.1016/j.concog.2007.06.005Google ScholarCross Ref
Spence, C. 2015. Multisensory flavor perception. Cell 161(1), 24--35. http://dx.doi.org/10.1016/j.cell.2015.03.007Google ScholarCross Ref
Prescott, J. 1999. Flavour as a psychological construct: implications for perceiving and measuring the sensory qualities of foods. Food Qual. Prefer. 10(4), 349--356. http://dx.doi.org/10.1016/S0950-3293(98)00048-2Google ScholarCross Ref
Prescott, J. 2015. Multisensory processes in flavour perception and their influence on food choice. Curr. Opin. Food Sci. 3, 47--52. http://dx.doi.org/10.1016/j.cofs.2015.02.007Google ScholarCross Ref
Haverkamp, M. 2009. Synesthetic design-building multi-sensory arrangements. In Kai Bronner & Rainer Hirt (eds.), Audio Branding: Brands, Sound and Communication (pp. 164--181). Nomos, Berlin, Germany.Google Scholar
Haverkamp, M. 2010. Synesthetic approach for evaluation of the cross-sensory quality of multi-media applications. In Quality of Multimedia Experience (QoMEX) (Trondheim, Norway, June 21--23, 2010). QoMEX. IEEE, 136--141. DOI= http://dx.doi.org/10.1109/QOMEX.2010.5516107Google Scholar
Haverkamp, M. 2013. Synesthetic design: Handbook for a multi-sensory approach. Birkhäuser Verlag, Basel.Google Scholar
Spence, C. 2015. Eating with our ears: assessing the importance of the sounds of consumption on our perception and enjoyment of multisensory flavour experiences. Flavour 4:3. http://dx.doi.org/10.1186/2044-7248-4-3Google ScholarCross Ref
Zampini, M. and Spence, C. 2010. Assessing the role of sound in the perception of food and drink. Chemosen. Perception, 3(1), 57--67. http://dx.doi.org/10.1007/s12078-010-9064-2Google ScholarCross Ref
Reinoso Carvalho, F., Velasco, C., van Ee, R., Leboeuf, Y. and Spence, C. 2016. Music influences hedonic and taste ratings in beer. Front. Psychol. 7: 636. http://dx.doi.org/10.3389/fpsyg.2016.00636Google ScholarCross Ref
Vickers, Z. and Bourne, M. C. 1976. Crispness in foods: A review. J. Food Sci. 41(5), 1153--1157. http://dx.doi.org/10.1111/j.1365-2621.1976.tb14406.xGoogle ScholarCross Ref
Zampini, M. and Spence, C. 2004. The role of auditory cues in modulating the perceived crispness and staleness of potato chips. J. Sense. Stud. 19(5), 347--363. h http://dx.doi.org/10.1111/j.1745-459x.2004.080403.xGoogle ScholarCross Ref
Velasco, C., Jones, R., King, S. and Spence, C. 2013. The sound of temperature: What information do pouring sounds convey concerning the temperature of a beverage. J. Sense. Stud. 28(5), 335--345. http://dx.doi.org/10.1111/joss.12052Google ScholarCross Ref
Yan, K. S. and Dando, R. 2015. A crossmodal role for audition in taste perception. J. Exp. Psychol. Hum. Percept. 41(3), 590--596. http://dx.doi.org/10.1037/xhp0000044Google ScholarCross Ref
Spence, C. and Piqueras-Fiszman, B. 2013. Technology at the dining table. Flavour 2:16. http://dx.doi.org/10.1186/2044-7248-2-16Google ScholarCross Ref
Spence, C. and Piqueras-Fiszman, B. 2014. The perfect meal: the multisensory science of food and dining. Wiley-Blackwell, Oxford, UK.Google Scholar
Spence, C. 2012. Auditory contributions to flavour perception and feeding behaviour. Physiol. Behav. 107(4), 505--515. http://dx.doi.org/10.1016/j.physbeh.2012.04.022Google ScholarCross Ref
Obrist, M., Velasco, C., Vi, C. T., Ranasinghe, N., Israr, A., Cheok, A. D., Spence, C. and Gopalakrishnakone, P. 2016. Touch, taste, & smell user interfaces: The future of multisensory HCI. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '16). ACM, New York, NY, USA, 3285--3292. DOI= http://dx.doi.org/10.1145/2851581.2856462 Google ScholarDigital Library
Hashimoto, Y., Nagaya, N., Kojima, M., Miyajima, S., Ohtaki, J., Yamamoto, A., Mitani, T. and Inami, M. 2006. Straw-like user interface: virtual experience of the sensation of drinking using a straw. In Proceedings of the 2006 ACM SIGCHI international conference on Advances in computer entertainment technology (ACE '06). ACM, New York, NY, USA, Article 42. DOI= http://dx.doi.org/10.1145/1178823.1178873 Google ScholarDigital Library
Hashimoto, Y., Inami, M. and Kajimoto. H. 2008. Straw-Like User Interface (II): A New Method of Presenting Auditory Sensations for a More Natural Experience. In Proceedings of the 6th international conference on Haptics: Perception, Devices and Scenarios (EuroHaptics '08), Manuel Ferre (Ed.). Springer-Verlag, Berlin, Heidelberg, 484--493. DOI= http://dx.doi.org/10.1007/978-3-540-69057-3_62 Google ScholarDigital Library
Petit, O., Cheok, A. D., Spence, C., Velasco, C. and Karunanayaka, K. T. 2015. Sensory marketing in light of new technologies. In Proceedings of the 12th International Conference on Advances in Computer Entertainment Technology (ACE '15). ACM, New York, NY, USA, Article 53, 4 pages. DOI= http://dx.doi.org/10.1145/2832932.2837006 Google ScholarDigital Library
Koizumi, N., Tanaka, H., Uema, Y. and Inami, M. 2011. Chewing jockey: augmented food texture by using sound based on the cross-modal effect. In Proceedings of the 8th International Conference on Advances in Computer Entertainment Technology (ACE '11). ACM, New York, NY, USA, Article 2, 4 pages. DOI= http://dx.doi.org/10.1145/2071423.2071449 Google ScholarDigital Library
Kadomura, A., Tsukada, K. and Siio. I. 2013. EducaTableware: computer-augmented tableware to enhance the eating experiences. In CHI '13 Extended Abstracts on Human Factors in Computing Systems (CHI EA '13). ACM, New York, NY, USA, 3071--3074. http://dx.doi.org/10.1145/2468356.2479613 Google ScholarDigital Library
Haverkamp, M. 2015. Can synesthetic perception help to define attractive product design? In 5^th International Congress of Synaesthesia, Science & Arts. Alcalá la Real, Jaén, Spain.Google Scholar
Calvert, G., Spence, C. and Stein, B. E. 2004. The handbook of multisensory processes. MIT press, Cambridge, MA.Google Scholar
Velasco, C., Obrist, M., Petit, O., Karunanayaka, K., Cheok, A. D. and Spence, C. (2016). Cross-modal correspondences in the context of digital taste and flavor interfaces. Presented at CHI Conference on Human Factors in Computing Systems, San Jose, CA, USA.Google Scholar
Spence, C. 2011. Crossmodal correspondences: A tutorial review. Atten. Percept. Psychophys. 73(4), 971--995. http://dx.doi.org/10.3758/s13414-010-0073-7Google ScholarCross Ref
Chen, L. and Vroomen, J. (2013). Intersensory binding across space and time: a tutorial review. Atten. Percept. Psychophys., 75(5), 790--811. http://dx.doi.org/10.3758/s13414-013-0475-4.Google ScholarCross Ref
Iordanescu, L., Grabowecky, M. and Suzuki, S. 2011. Object-based auditory facilitation of visual search for pictures and words with frequent and rare targets. Acta Psychol. 137(2), 252--259. http://dx.doi.org/10.1016/j.actpsy.2010.07.017Google ScholarCross Ref
Milliman, R. E. 1986. The influence of background music on the behavior of restaurant patrons. J. Cons. Res. 13(2), 286--289. http://dx.doi.org/10.1086/209068Google ScholarCross Ref
Marks, L. E. 1978. The unity of the senses: Interrelations among the modalities. Academic Press, New York, NY.Google Scholar
Martino, G. and Marks, L. E. 2001. Synesthesia: Strong and weak. Curr. Dir. Psychol. Sci. 10(2), 61--65. http://dx.doi.org/10.1111/1467-8721.00116Google ScholarCross Ref
Reinoso Carvalho, F., Van Ee, R., Rychtarikova, M., Touhafi, A., Steenhaut, K., Persoone, D., Spence, C. and Leman, M. 2015. Does music influence the multisensory tasting experience?. J. Sense. Stud. 30(5), 404--412. http://dx.doi.org/10.1111/joss.12168Google ScholarCross Ref
Velasco, C., Woods, A. T., Petit, O., Cheok, A. D. and Spence, C. 2016. Crossmodal correspondences between taste and shape, and their implications for product packaging: A review. Food Qual. Prefer., 52, 17--26. http://dx.doi.org/10.1016/j.foodqual.2016.03.005Google ScholarCross Ref
Bahrick, L. E., Lickliter, R. and Flom, R. 2004. Intersensory redundancy guides infants' selective attention, perceptual and cognitive development. Curr. Dir. Psychol. Sci. 13(99), 102. http://dx.doi.org/10.1111/j.1467-7687.2006.00539.xGoogle Scholar
Parise, C. V., Knorre, K. and Ernst, M. O. 2014. Natural auditory scene statistics shapes human spatial hearing. Proc. Natl. Acad. Sci. U.S.A., 111(16), 6104--6108. http://dx.doi.org/10.1073/pnas.1322705111Google ScholarCross Ref
Deroy, O., Crisinel, A. S. and Spence, C. 2013. Crossmodal correspondences between odors and contingent features: odors, musical notes, and geometrical shapes. Psychon. Bull. Rev., 20(5), 878--896. http://dx.doi.org/10.3758/s13423-013-0397-0Google ScholarCross Ref
Palmer, S. E., Langlois, T. A. and Schloss, K. B. 2016. Music-to-color associations of single-line piano melodies in non-synesthetes. Multis. Res. 29(1-3), 157--193. http://dx.doi.org/10.1163/22134808-00002486Google ScholarCross Ref
Parise, C. and Spence, C. 2013. Audiovisual cross-modal correspondences in the general population. In J. Simner & E. Hubbard (Eds.), The Oxford handbook of synaesthesia (pp. 790--815). Oxford University Press, Oxford, UK.Google Scholar
Ludden, G. D., Kudrowitz, B. M., Schifferstein, H. N. and Hekkert, P. 2012. Surprise and humor in product design. Humor 25(3), 285--309. http://dx.doi.org/10.1515/humor-2012-0015Google ScholarCross Ref
Ludden, G. D. and Schifferstein, H. N. 2007. Effects of visual-auditory incongruity on product expression and surprise. Int. J. Des. 1(3), 29--39.Google Scholar
Ludden, G. D., Schifferstein, H. N. and Hekkert, P. 2008. Surprise as a design strategy. Des. Issues 24(2), 28--38. http://dx.doi.org/10.1162/desi.2008.24.2.28Google ScholarCross Ref
Mandler, G. P. 1982. The structure of value: Accounting for taste. In M. S. Clark & S. T. Fiske (Eds.), Affect and cognition. The 17th Annual Carnegie Symposium on Cognition (pp. 3--36). Lawrence Erlbaum, Hillsdale, NJ.Google Scholar
Marinetti, F. T. 1909. The founding and manifesto of futurism. Retrieved from http://web.itu.edu.tr/~inceogl4/thresholds/futuristmanifesto.docGoogle Scholar
Howells, T. 2014. Experimental eating. Black Dog Publishing, London, UK.Google Scholar
Fox, J., Bailenson, J. and Binney, J. 2009. Virtual experiences, physical behaviors: The effect of presence on imitation of an eating avatar. Presence-Teleop. Virt., 18(4), 294--303. http://dx.doi.org/10.1162/pres.18.4.294 Google ScholarDigital Library
Project Nourishing, retrieved from http://www.projectnourished.com/Google Scholar
Spence, C. and Wang, Q. J. 2015. Sensory expectations elicited by the sounds of opening the packaging and pouring a beverage. Flavour 4:35. http://dx.doi.org/10.1186/s13411-015-0044-yGoogle ScholarCross Ref
Stroebele, N., & de Castro, J. M. 2006. Listening to music while eating is related to increases in people's food intake and meal duration. Appetite 47(3), 285--289. http://dx.doi.org/10.1016/j.appet.2006.04.001Google ScholarCross Ref
Guéguen, N., Jacob, C., Le Guellec, H., Morineau, T., & Lourel, M. 2008. Sound level of environmental music and drinking behavior: A field experiment with beer drinkers. Alcohol Clin. Exp. Res. 32(10), 1795--1798. http://dx.doi.org/10.1111/j.1530-0277.2008.00764.xGoogle ScholarCross Ref
Elder, R. S., & Mohr, G. S. 2016. The crunch effect: Food sound salience as a consumption monitoring cue. Food Qual. Prefer. 51, 39--46. http://dx.doi.org/10.1016/j.foodqual.2016.02.015Google ScholarCross Ref

Index Terms

A multisensory approach for the design of food and drink enhancing sonic systems
1. Applied computing
  1. Law, social and behavioral sciences
    1. Psychology

Recommendations

The Future of Human-Food Interaction
CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems

There is an increasing interest in food within the HCI discipline, with many interactive prototypes emerging that augment, extend and challenge the various ways people engage with food, ranging from growing plants, cooking ingredients, serving dishes and ...
Read More
Design of Cyber Food:: Beginning to Understand Food as Computational Artifact
CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems

With a growing interest in HCI around food, there is a trend to combine computing technology and food to facilitate novel eating experiences. However, most current systems tend to superimpose the technology over food rather than consider food itself as a ...
Read More
Exploring the Design Space for Human-Food-Technology Interaction: An Approach from the Lens of Eating Experiences
Embedded in everyday practices, food can be a rich resource for interaction design. This article focuses on eating experiences to uncover how bodily, sensory, and socio-cultural aspects of eating can be better leveraged for the design of user experience. ...
Read More

Comments

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Published in
MHFI '16: Proceedings of the 1st Workshop on Multi-sensorial Approaches to Human-Food Interaction
November 2016
48 pages
ISBN:9781450345613
DOI:10.1145/3007577
Editors:
Anton Nijholt
HMI, University of Twente, Netherlands, and Imagineering Institute, Malaysia
,
Carlos Velasco
BI Norwegian Business School, Norway, and University of Oxford, UK
,
Gijs Huisman
HMI, University of Twente, Netherlands
,
Kasun Karunanayaka
Imagineering Institute, Malaysia
Copyright © 2016 ACM
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
- Published: 16 November 2016
Permissions
Request permissions about this article.
Request Permissions

Check for updates
Author Tags
drink
eating
food
multisensory
sound
Qualifiers
- research-article
Conference

Acceptance Rates
MHFI '16 Paper Acceptance Rate6of8submissions,75%Overall Acceptance Rate6of8submissions,75%
More
Funding Sources
Other Metrics
View Article Metrics

Article Metrics
- 27
  Total Citations
  View Citations
- 672
  Total Downloads
- Downloads (Last 12 months)71
- Downloads (Last 6 weeks)6
Other Metrics
View Author Metrics
Cited By
View all

PDF Format

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

A multisensory approach for the design of food and drink enhancing sonic systems

MHFI '16: Proceedings of the 1st Workshop on Multi-sensorial Approaches to Human-Food Interaction

ABSTRACT

References

Cited By

Index Terms

Recommendations

The Future of Human-Food Interaction

Design of Cyber Food:: Beginning to Understand Food as Computational Artifact

Exploring the Design Space for Human-Food-Technology Interaction: An Approach from the Lens of Eating Experiences

Comments

Login options

Full Access

Published in

Sponsors

In-Cooperation

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Funding Sources

Other Metrics

Article Metrics

Other Metrics

Cited By

PDF Format

eReader

Digital Edition

Caption

A multisensory approach for the design of food and drink enhancing sonic systems

MHFI '16: Proceedings of the 1st Workshop on Multi-sensorial Approaches to Human-Food Interaction

ABSTRACT

References

Cited By

Index Terms

Recommendations

The Future of Human-Food Interaction

Design of Cyber Food:: Beginning to Understand Food as Computational Artifact

Exploring the Design Space for Human-Food-Technology Interaction: An Approach from the Lens of Eating Experiences

Comments

Login options

Full Access

Published in

Sponsors

In-Cooperation

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Funding Sources

Article Metrics

Other Metrics

PDF Format

eReader

Digital Edition

Share this Publication link

Share on Social Media